Read the article by Sally Holland at CNN.

Mitchell D. Chester, Ed.D.
Commissioner of Elementary and Secondary Education

Dear Commissioner Chester,

I work with a number of schools helping them review MCAS data and results. Thank you for shortening the turnaround time for MCAS results for schools, and for shortening the test periods.  Schools will greatly benefit from the increased analysis and planning window you have created.  However, the new policy regarding the release of only a partial set of questions greatly impacts the ability of teachers and schools to review those MCAS results in a meaningful way.

Having a report showing only strand information puts teachers at a big disadvantage. It is very difficult without the actual questions to see where students are having difficulty.  For example, the strand information does not indicate whether a math question is a whisker plot, a line graph, or a stem and leaf plot; students may have done well on one but not another.

ELA questions are even more problematic. The strand indicators are almost meaningless (fiction, non-fiction, poetry, etc.) because they don’t tie back to student skills, such as “finding evidence”, or “making inferences” about the passages.  When the questions and answers are released and provided to the teachers, these interpretations can be made easily; without the questions, it is almost not worth looking at the English MCAS results.  While it is not necessary to release all the reading passages, the questions and answers are fundamental to proper analysis.

Being able to see the question and answer content also allows teachers to see which answers the students indicated, showing the answer response patterns of their classes.  Many times students choose the same wrong answer, showing that they misunderstand a concept in the same way.  For example, in math when calculating an expression with absolute value operations and squaring, a number of students might calculate the square correctly, but not get the proper sign.  Again, without access to the questions, this type of inference is impossible.

Release of test items is critical for teachers to conduct a meaningful analysis, and to help their students make real progress.  While it may allow for shortening the test period, and perhaps save money in a time of state budget scarcity, it is a big step backward for MCAS.

I am sure you have considered other options, but I implore you to look more carefully.  I am confident there are a set schools that would allow future questions to be field tested, perhaps in an interim assessment in November.  Such an approach could provide a win-win situation.  Schools who participate in the interim assessment program would get better information about student progress before the spring exams, and the entire community would benefit from the release of the common test items from the March and May assessments.

I urge you to support schools and teachers by reversing the new policy that limits the release of test items.

Warm regards,

Mike Oates

mfoates@gmail.com

As we started working on one off requests, we decided to put the entire material in a simple to use spreadsheet so it can benefit as many teachers and tutors as possible. To use the spreadsheet and see this for yourself click on the MCAS Question and Content Analysis-v2.xls in your browser. In order for this to work, you will need Microsoft Excel (part of MS Office suite).

Save the spreadsheet on your local drive, open the spreadsheet and click on the “MCAS-TestAnalysis” Tab.

On this tab make the selection for various choices in the drop down boxes on the upper left:

• Grade level  (Row 3 )
• Year published (Row 4)
• Test Name (Row 5)

You can further refine the data on Row 8 by filtering

• Subject name
• Topics name
• Standard name

By default the report shows data for grade 6 but you can change that in Row 3 by making other selection.

The question distribution report will show as you make selection changes. I hope you will find this useful. Few teachers are using this data to create a series of formative assessments that focus on the highest frequency questions for each strand.

If you have any feedback or comments please let us know. Also, feel free to pass this along to any other teachers who you think will find it useful.

In the future we will try to hook this report into Socrato as a feature so you will have access without Excel. For more interesting content and learning data analysis checkout www.socrato.com or contact us to setup a demo.

Best Regards,
-Raju

UPDATE – uploaded the new version of excel file – this has the full description of each standard.

User Interface

• More content  – In addition to grade 3 – 5 MCAS material, more content was added for grades 6 and 7.
• Easier access to content – Question search by grade level.
• Improved reporting - The Question distribution report now shows students’ points-data for open response questions.
• More stable upload of students’ assessment data.
• Fixed a number of issues for students TIA and roster data upload process. (e.g., information for one question was captured incorrectly and the session ID was getting lost).
• Fixed a number of other bugs.

Under the Hood

• Faster performance for take test area – With a number of schools now using Scorato in their school computer labs, we needed to make improvements so the system does not slow down. We made a good number of changes to how the take test is managed in the system. This will allow our servers to support a larger number of concurrent users.
• More stable and faster reports generation – We have completely revamped how the PDF files are generated for SAT diagnostic reports. With this change, the time required for report generation has dropped significantly. The current load calculation shows that the first report is available in about 10 -15 seconds, and subsequent reports are created in less than 5 seconds. These are complex reports, so this improvement constitutes a significant improvement compared the previous time lapse required to generate the reports. Pretty cool!

If you encounter any issues with these changes and improvements, please let us know.

Looking Forward

New and improved key features which are currently works in progress include:

• Additional reports which will allow teachers and principals to slice and dice their school’s and students’ data. They will be able to leverage these reports for instruction planning. This will allow them to automatically create custom and personalized assignments from Scorato. We are very excited about this.
• Making the delivery (printing and emailing) of all reports easier.

A big thank you goes out to all who have sent suggestions, requests and issues to us. We welcome your feedback and try to act on it in a timely fashion.

-Raju

Teachers who are tired of “cut and paste” can now go directly to our MCAS Question Search page, select the grade, select the strand standard, and we’ll provide a shopping list of questions fom which you can choose. If there are passages included, you can look at the passage detail with one click. Select the questions you like, put them in your QuestionCart, and create your own test. Print it for class assignments or assign it to students to practice online.

To teachers and school administrators, if you want learning data analysis for prior year “Test Item Analysis” from MCAS, please feel free to contact us. We will be happy to give you custom demo or you can learn more on the Socrato website. We are working with a number of schools and tutoring centers and will be happy to share our work to see if we can be of help to you.

Here are a few of my observations:

• The educational data belong to students, but unfortunately students don’t have any way to carry it around easily. Dr. Mostafa makes a point that the data should be like health records. I will go a step further and suggest that education data should be more like a combination of health records and financial records. Think of the data in Quicken / Quickbooks. The data belong to students and should therefore be accessible to them everywhere. The data should also be capable of providing insight into educational decision making. For example, Quicken provides insight into sources of income and categories of expenses. In the same way, educational records, educational data should (and we believe will) provide insight into strengths and weaknesses of students, as well as strategies and recommendations to manage them.

• For k-12 students the information should be available to parents when they want it and in broken out in a detailed format. Given that people can receive weekly summaries on Facebook or Linkedin about what their connections are doing, or from Quicken about what bills are due in coming week, why should people be able to receive information about their kids’ education status, e.g., how they are doing academically on the weekly basis in class and where they need to focus to improve in a given area? I believe this real-time, detailed data access will improve transparency and participation from parents to help their kids.

• The convergence of Summative and Formative Assessment is happening. As I referenced in my previous article about this topic, it makes lot of sense that each formative assessment should into summative assessment.

There is certainly a “tipping point” building up in the education space. The question is what is: who will become the Napster in the education arena?

-Raju Gupta
Socrato

The key discussion of this post is about leveraging technology. This story, even though occurs in the past is still relevant today.  It raises interesting points about reusing good educational content, improving efficiency in test practice and preparation and the behind-the-scenes analysis. If technology were implemented and leveraged the right way it could have prevent a number of recent high profile “testing” scandals like: Armed Services Content Sharing and the Profession Exam Piracy issue.

When used correctly technology can really be a boon for both educators and students. Examples of good implementation of process and technology in education are some of the standardized tests like the GRE and GMAT being conducted leveraging computers, but done in the proctored centers. This has allowed students to schedule exam at their own convenience as well as receive results very quickly instead of waiting for months. This helps everyone involved.

While the use of technology in conducting educational tests has become mainstream, areas such as content distribution, test preparation and practice, and test analysis still need vast improvements in order to help students. Over the past 10 or so years we’ve seen a lot of focus and improvement in the process and distribution of entertainment related content such as digital music, video, mobile and social networking. As Generation X becomes entrenched with computers and the internet, they will begin to venture into other areas of internet. The shift is already happening where expensive text books and educational content is appearing on p2p networks in digital format. Amazon is also venturing into the e-textbook space via a student version of Kindle where custom books are becoming the norm.

I hope over the course of the next few years the education and publishing industry will see similar advances in content distribution and process management as we have seen in the music, video and social networking spaces.

Issue Resolved

Increased testing effort on Safari browser
We have increased testing efforts on the Safari browser. We are finding a few issues and are making fixes to improve Socrato on Safari. There are certain issues, specifically “frozen” screens or dropdown selectors which are not working,  which we are hearing from you but we are not able to reproduce in our testing.

• Please let us know if you are having these issues so we can learn more about how to reproduce the problems.

Improved performance for assessment result upload with progress bar
A couple of you mentioned having issues with machine freezing while uploading of student scores. And refreshing of page was resulting in double data entry of the scores.

• Now when you do result upload, a progress indicator will appear to show the system is processing the student scores. In addition we have number of hidden changes which will eliminate duplicate upload.

Ability to delete or update previously uploaded session results
People were having difficulty in deleting or modifying the uploaded user data that was loaded in error

• We made it easy to edit and delete the entire upload if you uploaded the wrong data. There is a ‘Delete All’ button on Edit sessions page, or you can delete results student by student.
• In session list drop down there is a unique identifier (timestamp) for all your uploads that indicates when it was uploaded

Updated scoring calculation for Open Response question
The system was treating any data entry in open response question as correct, including a score of zero.

• Now if you assign points (greater than 0) to a question it will be marked as correct, while zero will be marked incorrect (instead of correct). All empty score will be considered as ungraded.
• In future we are considering adding the concept of partially correct. Let us know you would find this helpful.

Question Distribution report now shows Open Response Questions

• In the Question Distribution report you will get to see the open response and short description questions as part of report. Currently we are not proving the score distribution (it’s on high priority list)

Temporary workaround for Printing large passages (more work in progress – see below)
Few users are having issues printing large passage. It shows correctly on the computer screen but prints only partial information from certain browsers. We are able to reproduce the issue and have a temporary workaround.

• For long ELA passages current workaround is to print on Safari. It’s not perfect but does the job.
• To make global fix we will have to cut the passage images in smaller chunks to fit page size. It will take time because there are 143 passages. Please let us know any passage you want expedited and we will fix those first. We apologize about this oversight.

Printing Icon does not do anything on IE
In Microsoft Internet Explorer (IE) when a user clicks on the print button, nothing happens and they get a javascript error.

• We are able to reproduce the issue on IE. We believe it is because the pop-up blocker is turned on. The fix for this is to either temporarily disable the pop-up blocker or add www.socrato.com to your “allow pop-up from this site” list.
• You can do that by going to Tools à pop-up blocker à turn off pop-up blocker

Please give us call if you want us to walk you through.

Some other changes have been done as well, including a lot more help has been added at places. Please let us know if there are particular places that are confusing and we will add more help in those areas as well.

Open Issues on Hot List

• Show some type of “Grouping Indicator” for question from same passage on Search Question page and Question List
• Large passage text not printing completely on paper, where as it is available online.
  • This will require few days of effort. If you have request for particular content set please let us know. We will fix those first.
• In certain circumstances the test heading and description are interchanging (showing incorrectly) on Test List page
  • We have seen this once, but never after that. If you are seeing this we would be very grateful if you contacted us so we can learn how to reproduce it.
• Printing issue – keeping questions and answers together on same page
  • We are looking onto this and it may take some time
• Content upload made easy – Questions upload for various types of questions from outside to Socrato. We have work in progress which will make this very easy. Please stay tuned.

Once again, thank you for your support and please feel free to reach out to me any time for any Socrato related question.
Best Regards,

Socrato Team

New release streamlines MCAS Summative Evaluation and Formative Evaluation.

Thank you all for your patience, support, and feedback. We are pleased to share a few more updates to Socrato which we believe will make life bit easier.

Uploading of MCAS Test Results / Assessment Test Results is a lot smoother along with the real time reports

The issue of similar Test Names was causing confusion.

• That issue has been fixed. Pretty much all Test List pages now show a unique Test identifier (the Test Id #)
• On Enter Results page while selecting the Test for uploading results, the Test Id # and the description are shown in the selection box
• As soon as your test results are uploaded on the summary page you can click on the link for the Question Distribution report. A very revealing and helpful graphical reports shows up (my favorite report, and a recommendation from Sharon – thank you!)

Search MCAS Questions – Getting better, plus no more clicking to see question images.

There were few issues like ‘click here to see image’, or look into your printout to understand what taxonomy 7.D.3 means? Those issues are resolved.

• On ‘Search Question’ page the taxonomy description is available right in the search dropdown
• If the question search result has questions with images, you have an option to see the image right there (an option box to Show image). This fix is also available on ‘Question Preview’ page as well when you are creating / editing any custom test

 Create Custom Tests and Edit them any time.

The fix was requested to be able to modify the Test title and description as they were causing confusion

• Test Editing is now available for every one. If you have created the Test, you can edit the Test any time. Change the title, description, tags etc. You can even add or remove questions any time

If you are the creator of the Test you will see an Edit button.

 Cancelled or Rejected members are eliminated from process

Earlier the canceled/rejected group members (students) where showing up in the Enter Results, Attendance and other group screens.

• The attendance sheet does not show any cancelled members
• The Enter Results page also eliminates them
• They have also been removed from all group reports
• You can see which members are canceled/rejected on the Member List

 Invite your Students to join Socrato

• If your students are using Socrato un-officially, give them the registration code for your group. Invite and encourage them to join the group. When they do the practice in Socrato on their own you can track their efforts and get the Formative Assessment and other reports for no effort. Try it out!!

 Other Updates

• Number of other issue were resolved, as well few content changes were made to eliminate errors
• We also improved the performance of the group “home” page

 We are aware of a number of outstanding requests and issues. A number of improvements are already underway. We will love to hear from you, and get your feedback so we can prioritize the issues which are most important to all of us.

For any questions, suggestions, or ideas feel free to reach out to us via email, phone, or comments on the Socrato blog (http://blog.socrato.com).

Best,

Socrato Team

With this release Socrato also welcomes all new teachers supporting summer MCAS tutoring.

It’s inspiring to see how much effort, energy, and dedication goes in to help at-risk kids. We feel proud to be part of program, and we hope our efforts are making a difference for the teachers and students.

The focus of this release was to start supporting tutoring centers, schools, and classroom instructors by providing real time analysis and recommendations about student learning to teachers and students. In addition, we added about 1,100 more questions to the “question bank” to support MCAS test preparation. Key highlights are as follows.

• Search our large question bank by curriculum strands and standards, and create your own custom tests to print for students or assign to them online 
  • Share your test with other teachers.
  • Have your students take the tests on-line on Socrato, or off-line with printed quizzes.
  • The questions are mapped to the state curriculum, so you can quickly report on student and class performance according to the DOE standards.
• Vastly enhanced Group area
  • In addition to creating a simple group to collaborate, large or complex organizations like tutoring center, schools, districts or training organizations are supported. Groups are built to support easy sharing of test materials, and also provide reporting and detailed analysis at different levels of the group structure.
    
• Timesheet and attendance for group members
  • This feature was added based on the request from our key users. It allows full tracking of timesheet for instructor and electronic attendance submission for students. The time and attendance sheets are supported via user-configurable workflow for approval and processing.
• Assignments and sharable content libraries
  • Any content can be associated with groups along with comment for assignment and communication. Content can also be shared across multiple groups by putting leveraging the hierarchical group structure.
• Ability to load past MCAS TIA’s (Test Item Analysis Report) & Assignment Results
  • Teachers can upload data and run analysis for a group of students
• Exhaustive analysis and reporting
  • Students and Teachers can get real time analysis of their students and class. Example report are question response analysis, trends by students, trends by subject areas over time, and more.
• Sizable question bank
  • In addition a significant amount of content has been added into the question bank to support the full usage of all features.

Thank you for all of your feedback. It has been instrumental in shaping the new functionality and changes in this release.   We also have long list of enhancement requests. We are working on those.  Keep letting us know how we’re doing and what we can do to improve things.

 To all our users, thank you for your feedback and bug submissions. If you have any comments or questions please feel free to reach out to us.

 Best,

-Team Socrato