[Reading-hall-of-fame] negative 'gain'

[Jay Samuels]

 Colin and David,

Thanks for your comments. Yes, at times I think that doing research is like
painting by numbers.

 This morning I read Ken Goodman's idea that we as a group endorse or
criticize some aspect of education. What I found interesting about ken's
idea was that at that moment I was thinking about writing to the Dean of
Education at Minnesota about the No Child Left Behind legislation. The
legislation seeks to bring almost every student to some level of academic
achievement. Now that Bush believes he has a mandate, he is pushing to have
the law extended to include high school testing.  My purpose in wanting to
contact the Dean at the College of Education at the University of Minnesota
was to ask what he thought about the idea of having a forum on the
legislation. Given what we know about factors that affect academic
achievement, is the legislation psychologically sound? Can we raise
achievement for all groups, and, if so, what will it take? Philosophical and
political goals are not always psychologically sound or feasible.  Perhaps,
the Reading Hall of Fame sponsor such a forum. Then, after we have come to
and arrived at an educated opinion, if our position runs counter to what the
No Child Left Behind desires, we can voice our opinion as a group. My time
table is about a year from now. By then we will have heard a variety of
opinions on the Bush education mandate. There are think tanks such as the
Manhattan Institute for Policy Research that have published papers by highly
educated researchers who are highly critical of our schools. So, not every
one is against the Bush legislation. I am reasonably certain that if we want
to conduct such a forum the topic is important enough that IRA could work
with us to find a prominent spot at the convention in San Antonio, TX. TX,
what a suitable place for such a forum. Jay samuels
-----Original Message-----
From: reading-hall-of-fame-admin@nottingham.ac.uk
[mail:reading-hall-of-fame-admin@nottingham.ac.uk] On Behalf Of Colin
Harrison
Sent: Sunday, November 07, 2004 6:00 PM
To: reading-hall-of-fame@nottingham.ac.uk
Subject: [Reading-hall-of-fame] negative 'gain'

Hi everyone

I wish to make two points: first is to remind ourselves that there is simply
loads of 'error' in reading tests, and this alone means that negative 'gain'
in post-tests will be common; and second, that we should not get hung up on
it, since the stats we commonly use will tell us whether or not we have a
stable picture emerging from our data.

I led an evaluation a few years back in which we tested using a standardised
two-form reading test the reading skills of 700 young people aged 11-14 who
were in 7 inner-city schools. The participants received one of two parallel
versions of the test in the October and took the other version as  the
post-test eight months later, in the June, following 8 months of exposure to
a mixed programme of reading development activities.

After we scored  the post-test, I was astonished at how many of these
inner-city kids had post-test scores that were lower than their
pre-intervention scores. On visual inspection of the results, the data
looked a nightmare to interpret. The scores were all over the place.

But guess what?  Sure enough, when we did the stats, in the two schools that
had teachers who had really got into their staff-development program (and
crucially, where reading development at system level was not in its first
year), there were statistically significant differences favouring the kids
from those two schools. Yep, although the scores in all seven schools
included plenty of 'negative gains', the pooled results were indeed stable
enough for significant differences to show through clearly. 

Moral- don't expect statistical tests that are meant to be based on grouped
data to do the job at the individual level; poor readers do not all tend to
make massive leaps of progress. Contextual factors, local testing factors,
test administration factors, text factors that reflect the effects of
motivation, background knowledge ect, will all impact the data, and produce
error at the individual level, which means that an individual's 'true' level
is within an error band whose width may be greater than the grade-level
improvement delivered by the programme.

We're painting by numbers here, it ain't a Durer oil.

Colin


Best wishes

Colin

---------------------------------------------------

Colin Harrison                    * Colin's office:
(+44)-115-951-4441
School of Education          * Research Office:  (+44)-115-951-4990
Jubilee Campus                 * Colin's assistant    (+44) 115-951-4518
University of Nottingham    * Colin's mobile:      (+44)-788-777-5154
Nottingham  NG8 1BB, UK * Research  Fax    (+44) 115 846 6188



_______________________________________________
Reading-hall-of-fame mailing list
Reading-hall-of-fame@nottingham.ac.uk
http://lists.nottingham.ac.uk/mailman/listinfo/reading-hall-of-fame