[Reading-hall-of-fame] negative 'gain'

[Colin Harrison]

Hi everyone

I wish to make two points: first is to remind ourselves that there is =
simply loads of 'error' in reading tests, and this alone means that =
negative 'gain' in post-tests will be common; and second, that we should =
not get hung up on it, since the stats we commonly use will tell us =
whether or not we have a stable picture emerging from our data.

I led an evaluation a few years back in which we tested using a standardise=
d two-form reading test the reading skills of 700 young people aged 11-14 =
who were in 7 inner-city schools. The participants received one of two =
parallel  versions of the test in the October and took the other version =
as  the post-test eight months later, in the June, following 8 months of =
exposure to a mixed programme of reading development activities.

After we scored  the post-test, I was astonished at how many of these =
inner-city kids had post-test scores that were lower than their pre-interve=
ntion scores. On visual inspection of the results, the data looked a =
nightmare to interpret. The scores were all over the place.

But guess what?  Sure enough, when we did the stats, in the two schools =
that had teachers who had really got into their staff-development program =
(and crucially, where reading development at system level was not in its =
first year), there were statistically significant differences favouring =
the kids from those two schools. Yep, although the scores in all seven =
schools included plenty of 'negative gains', the pooled results were =
indeed stable enough for significant differences to show through =
clearly.=20

Moral- don't expect statistical tests that are meant to be based on =
grouped data to do the job at the individual level; poor readers do not =
all tend to make massive leaps of progress. Contextual factors, local =
testing factors, test administration factors, text factors that reflect =
the effects of motivation, background knowledge ect, will all impact the =
data, and produce error at the individual level, which means that an =
individual's 'true' level is within an error band whose width may be =
greater than the grade-level improvement delivered by the programme.

We're painting by numbers here, it ain't a Durer oil.

Colin


Best wishes

Colin

---------------------------------------------------

Colin Harrison                    * Colin's office:        (+44)-115-951-44=
41
School of Education          * Research Office:  (+44)-115-951-4990
Jubilee Campus                 * Colin's assistant    (+44) 115-951-4518
University of Nottingham    * Colin's mobile:      (+44)-788-777-5154
Nottingham  NG8 1BB, UK * Research  Fax    (+44) 115 846 6188