[Reading-hall-of-fame] Tom Sticht Negative Numbers

[Jay Samuels]

This is in response only to the first part of Tom's message in which he is
concerned about how does one interpret negative gain scores from a pretest
to a posttest. The use of gain scores in research is fairly common today.
Why? Increasingly we find ourselves doing research, not in a laboratory, but
in schools. Many schools today do not want to cooperate with academic
researchers because of the pressures put on the schools to do well under "No
Child Left Behind" legislation, so it is difficult even to get permission to
work in the schools. If we are lucky we can get permission to work in a
school,  but the idea of using random assignment to treatment is not
allowed. We have to work with intact classes. Therefore, we use gain scores
and we campare the gains between the experimental and the control
treatments. 

What I am responding to in this note is not the statistical manipulations
involved in these comparisons in which one encounters either no gain or
negative gain. Instead what I am addressing is an interpretation. What if
instead of thinking of these scores as measures of learning, we thought of
them as performance on tests at two different times. Now, for example, with
the widespread use of Curriculum Based Assessment (Speed of reading on a
test passage when the student is tested for only one minute)and its
offshoot, the Dibels test, teachers are being asked to use gain scores to
measure growth. Of course, the assumption is that the gain will always be
positive, but it need not be. If for example, the pretest materials are
easy, the student will do well but if the posttest materials are harder, the
student does poorly. So, the gain is negative and gives us an experimental
error. But this situation is possible, but unlikely. The more common
situation is that the student may actually do poorly on the posttest and the
problem resides in the student, not the materials or the testing situation. 

In the school system where a lot of my experimental work is done in some
schools more than half the class has the south east Asian Hmong. These kids
speak Hmong at home, they speak Hmong in school to each other, and only use
English in the classroom. Their decoding skills are far better than their
comprehension skills because their vocabulary skills are so limited. Some of
the kids do not attend school regularly and they do not practice reading on
their own, so it is possible that for a few of the kids their posttest
scores will be lower than their pretest scores. It is even possible to get
negative gain from kids who attend school regularly and seem to be able and
eager participants in class. What does one do with these scores? If my
memory is corresct, I believe we kept them in, but I am going to talk to my
former grad student, a whiz in stats who is now with the Minnesota State
Department of Education doing data analysis for the State on how kids do on
the Minnesota Basic Skills Test, a mandated test in math and reading the
students must pass in order to get a high school graduation diploma. I will
find out what she did with the negative scores if they were encountered. Jay
samuels  


 Dr. (S)tanley Jay Samuels
 Department of Educational Psychology
 College of Education
 University of Minnesota
 Minneapolis, MN 55455

 Universitiy Phone 612 625 5586
 Fax  612624 8241