I came across an article in The New York Times about how educational technology companies inflate success rates and withhold negative results of studies done to evaluate their software. It seems that in many cases, students don’t show significant improvement on standardized tests after using some software programs.
The article focused on one particular software program, Carnegie’s Cognitive Tutor. At a school district in Augusta Georgia, teachers noted that after using the program, students who had been performing below average made enough progress to catch up with students in mainstream classes. Classroom benefits, though, don’t always yield the same results on standardized state tests.
“On the other hand, when the new state math test was given in March, 27 percent of the district’s 11th graders did not pass. . .”
It might be that students don’t do so well on the state tests because they are used to the way the software program presents information. When they take the test, which in most cases is paper and pencil, the difference between reading and working out problems on a computer versus choosing the correct multiple choice answer and entering the answer on a Scantron bubble sheet might cause students to work slower to adjust to the change and consequently, answer fewer questions. The difference in presentation might also cause confusion, which might result in more wrong answers.
This doesn’t excuse educational software companies for cherry-picking data or omitting whole studies altogether. But it might explain why the results of some studies aren’t relevant.
