[Reading-hall-of-fame] Rejoinder to ALL Wrong

[tsticht at znet.com]

The following is from: www.nald.ca/WHATNEW/hnews/2005/murray.htm

It's ok to cry wolf if the wolf is at the door: A rejoinder to Tom Sticht

T. Scott Murray
International Study Director
The Adult Literacy and Life Skills Survey (ALL)

I am writing to respond to opinions voiced by Tom Sticht in "ALL Wrong 
Again! Can Adult Literacy Assessments Be Fixed?" published on May 17, 2005 
on the NALD website.

I do so somewhat reluctantly because I have considerable respect for Tom's 
contribution to the literacy field. In this case, however, he does 
injustice to the governments that have financed the ALL study, the national 
teams that devoted their lives to implementing it, to science and, worst of 
all, to the millions of adults with limited reading skills who could 
benefit from government attention to their plight.

I would like to begin by pointing out that the study is a collaborative 
effort that has involved a large team of international experts drawn from a 
very wide range of countries. The procedures employed in the study, and the 
inferences drawn from them, were subjected to intense scrutiny by some of 
the world's best technicians and policy makers in a range of related 
domains: psychometrics, assessment design, quality assurance and 
statistical analysis. 30 countries made an informed and studied decision to 
field the IALS and ALL studies. Tom seems to imply that they all fell off 
the proverbial turnip truck, that they spent millions of dollars each on 
something deeply flawed. I think not. Interested readers are referred to 
The Adult Literacy and Life Skills Survey: New Frameworks for Assessment 
(available at www.nald.ca or www.statcan.ca) for a sense of the theory 
underlying the ALL study and how the instruments were validated.

Tom also advances an argument that the performance levels employed in the 
IALS and ALL are inappropriate, citing the work of the US National Academy 
of Sciences as support for his position. Tom is right that none of the 
surveys (NALS, IALS, or ALL) ever underwent a standard setting process to 
identify a "suitable minimum needed for coping in today's societies". No 
such standard setting process was ever undertaken. What both Tom and NAS 
report fail to acknowledge is the DATA which lead us to say that adults who 
are in these two levels are at increased risk. In the US, for example, 
adult education is targeted at those who are over the age of 16 and who 
have not earned a high school diploma or GED. This idea was supported by 
the Committee on Reading of the National Academy of Education when they 
said, "we take the position that the reading problem in the United States 
should not be stated as one of teaching people to read at the minimum level 
of literacy, but rather as one of ensuring that every person arriving at 
adulthood will be able to read and understand the whole spectrum of printed 
materials that one is likely to encounter in daily life." The Committee 
went on to say that, "our national educational policy is that every child 
is expected to complete at least the twelfth grade; we ought then to expect 
every child to attain twelfth-grade literacy."

Using this as a stated goal, the NALS data reveals that adults Level 1 (0-
225) performed at a level that was below the average of those adults who 
had dropped out of high school (231), and far below those adults who 
terminated their education with a GED (268) or diploma (268). 
Interestingly, these latter mean scores are close to the top of Level 2 
(275). These data have been confirmed by both the IALS and ALL survey.

Recent reports on how American 15 year old students perform in 
international comparative assessments of math and reading confirm the 
patterns seen in NALS, IALS and ALL. Only 68 percent of US students who 
start high school earn a diploma and among those who do many are deemed by 
governors and business leaders as not having adequate skills to earn a 
middle class wage or successfully enter and complete post-secondary 
education programs.

Differences in proficiency matter are not trivial - about 53% of the poor 
and near poor in the US are in Level 1, 34% are in Level 2. 52% of 
individuals classified at Level 1 are out of the labour force. 3/4 of Level 
1 adults do not have savings upon which they could draw if needed. In sharp 
contrast 3/4 of Level 4 have savings. Differences in the level and 
distribution of literacy as defined in NALS, IALS and ALL explain up to 33% 
of wage differences, and over half of differences in the long term growth 
of GDP per capita, in OECD countries. It is reasonable, therefore, to 
assume that these things have a material impact on the quality of life of 
individuals with low skills. Tom seems not to care.

Tom then goes on to bring up the RP issue, yet again. What he fails to 
acknowledge is that a response probability has no impact on the underlying 
ability distribution. Even Andy Kolstad, one of the most ardent proponents 
of lower RP values, was quoted in the Washington Post as stating that the 
"true literacy proficiencies of the population remain as reported...the 
response probability convention influences how the results are interpreted 
not how well adults perform on the assessment."

Tom is correct when he says that an RP 50 is the point where one has equal 
errors about whether a person can or cannot perform a task. Put another 
way, however, RP 50 is also the point at which the least is known about 
whether an individual can or cannot perform a task correctly. If the goal 
is to know whether or not someone is likely to answer a question correctly 
one needs to set the RP criterion higher. Thus setting an RP is a judgment 
call that depends on the inferences one wants to make. Few employers or 
patients would set a low criterion.

Thus, response proficiencies are something that one applies after the fact 
to add interpretative value to the underlying proficiency scales. Setting 
an RP is a judgment call that depends on the inferences one wants to make. 
Adopting a less demanding response proficiency standard would reduce the 
numbers of people classified in the lowest skill levels. Unfortunately, 
however, it would also serve to blur the relationship of literacy to the 
educational, social, health and labour market outcomes that are of central 
concern to policy makers. I might forgive my tom cat if he only managed to 
find his way home 50% of the time but the empirical evidence presented in 
the NALS, IALS and ALL reports confirms that employers notions of mastery 
are much more demanding – anything lower than 80% attenuates the 
relationships with critical co-variates and hence the policy utility of the 
findings.

Recent data from a US study designed to explore the relationship of 
component reading skills to the emergence of fluency also provides more 
empirical support for applying an RP80 (see Strucker, J., Yamamoto, K. and 
Kirsch,(2004), Component Skills of Reading: Tipping Points and Five Classes 
of Adult Learners, ANCLI, Lyon)

Change the RP criterion means changing the interpretation of what it means 
to be in a proficiency level. In NALS, IALS and ALL someone who is in the 
middle of Level 2 would be expected to get a score of 80 percent correct on 
a hypothetical test drawn from Level 2 items or a grade of "B" in most 
American schools. Using an RP 50 criterion, an individual in the middle of 
Level 2 would be expected to get a grade of 50 on a hypothetical test made 
up of items that fell in Level 2 – equivalent to an "F" on that test, 
whereas an RP 67 would yield a "D". Not many parents would be pleased with 
such performance.

Finally, Tom seems to have missed the fact that the notion of literacy 
embodied in NALS, IALS and ALL is a dynamic one in which functionality can 
only be judged against the reading demands that individuals are likely to 
confront in their daily lives. It is a given that individuals are far more 
proficient with familiar tasks than implied in the NALS, IALS and ALL 
results. This, however, misses the point.

It is of critical importance to acknowledge that reading demands change, 
either in response to changes in society and technology, to natural changes 
that occur as adults negotiate the life course, or because adults develop 
goals or aspirations that involve a need to master new, unfamiliar reading 
tasks.

In this dynamic view, fluent reading, as defined by Level 3, can be thought 
of as a tool that helps individuals cope efficiently with such change. The 
analyses presented in the ALL comparative report provide compelling 
evidence that the transferable skills measured in the study are what 
matter. Individuals, and nations, that fail to attain Level 3 proficiency 
pay a serious price – wage inequality is growing, not shrinking and it is 
low skilled that suffer.

It is expected that the rapidly increasing global supply of literacy skill, 
and literacy's link to information and communication technologies and 
productivity, will amplify these effects. This evidence has played a 
critical role in convincing several countries to invest heavily in adult 
education and training. Swiss data presented in he ALL report reveals that 
policy can improve the level and social distribution of literacy skill over 
the course of a decade.

Crying wolf is ok if the wolf is indeed at the door. Perhaps it is Tom who 
is "ALL wrong."