[Reading-hall-of-fame] Rejoinder to ALL Wrong

[ken Goodman]

I find it quaint that governors and employers views of literay are cited 
as scientific evidence of lack of literacy.
As Shakespeare said "Me thinks the lady doth protest overly much." I 
find Tom's arguments persuasive and little in this long rambling defense 
of bad studies that refutes it.

Ken Goodman

tsticht at znet.com wrote:

>The following is from: www.nald.ca/WHATNEW/hnews/2005/murray.htm
>
>It's ok to cry wolf if the wolf is at the door: A rejoinder to Tom Sticht
>
>T. Scott Murray
>International Study Director
>The Adult Literacy and Life Skills Survey (ALL)
>
>I am writing to respond to opinions voiced by Tom Sticht in "ALL Wrong 
>Again! Can Adult Literacy Assessments Be Fixed?" published on May 17, 2005 
>on the NALD website.
>
>I do so somewhat reluctantly because I have considerable respect for Tom's 
>contribution to the literacy field. In this case, however, he does 
>injustice to the governments that have financed the ALL study, the national 
>teams that devoted their lives to implementing it, to science and, worst of 
>all, to the millions of adults with limited reading skills who could 
>benefit from government attention to their plight.
>
>I would like to begin by pointing out that the study is a collaborative 
>effort that has involved a large team of international experts drawn from a 
>very wide range of countries. The procedures employed in the study, and the 
>inferences drawn from them, were subjected to intense scrutiny by some of 
>the world's best technicians and policy makers in a range of related 
>domains: psychometrics, assessment design, quality assurance and 
>statistical analysis. 30 countries made an informed and studied decision to 
>field the IALS and ALL studies. Tom seems to imply that they all fell off 
>the proverbial turnip truck, that they spent millions of dollars each on 
>something deeply flawed. I think not. Interested readers are referred to 
>The Adult Literacy and Life Skills Survey: New Frameworks for Assessment 
>(available at www.nald.ca or www.statcan.ca) for a sense of the theory 
>underlying the ALL study and how the instruments were validated.
>
>Tom also advances an argument that the performance levels employed in the 
>IALS and ALL are inappropriate, citing the work of the US National Academy 
>of Sciences as support for his position. Tom is right that none of the 
>surveys (NALS, IALS, or ALL) ever underwent a standard setting process to 
>identify a "suitable minimum needed for coping in today's societies". No 
>such standard setting process was ever undertaken. What both Tom and NAS 
>report fail to acknowledge is the DATA which lead us to say that adults who 
>are in these two levels are at increased risk. In the US, for example, 
>adult education is targeted at those who are over the age of 16 and who 
>have not earned a high school diploma or GED. This idea was supported by 
>the Committee on Reading of the National Academy of Education when they 
>said, "we take the position that the reading problem in the United States 
>should not be stated as one of teaching people to read at the minimum level 
>of literacy, but rather as one of ensuring that every person arriving at 
>adulthood will be able to read and understand the whole spectrum of printed 
>materials that one is likely to encounter in daily life." The Committee 
>went on to say that, "our national educational policy is that every child 
>is expected to complete at least the twelfth grade; we ought then to expect 
>every child to attain twelfth-grade literacy."
>
>Using this as a stated goal, the NALS data reveals that adults Level 1 (0-
>225) performed at a level that was below the average of those adults who 
>had dropped out of high school (231), and far below those adults who 
>terminated their education with a GED (268) or diploma (268). 
>Interestingly, these latter mean scores are close to the top of Level 2 
>(275). These data have been confirmed by both the IALS and ALL survey.
>
>Recent reports on how American 15 year old students perform in 
>international comparative assessments of math and reading confirm the 
>patterns seen in NALS, IALS and ALL. Only 68 percent of US students who 
>start high school earn a diploma and among those who do many are deemed by 
>governors and business leaders as not having adequate skills to earn a 
>middle class wage or successfully enter and complete post-secondary 
>education programs.
>
>Differences in proficiency matter are not trivial - about 53% of the poor 
>and near poor in the US are in Level 1, 34% are in Level 2. 52% of 
>individuals classified at Level 1 are out of the labour force. 3/4 of Level 
>1 adults do not have savings upon which they could draw if needed. In sharp 
>contrast 3/4 of Level 4 have savings. Differences in the level and 
>distribution of literacy as defined in NALS, IALS and ALL explain up to 33% 
>of wage differences, and over half of differences in the long term growth 
>of GDP per capita, in OECD countries. It is reasonable, therefore, to 
>assume that these things have a material impact on the quality of life of 
>individuals with low skills. Tom seems not to care.
>
>Tom then goes on to bring up the RP issue, yet again. What he fails to 
>acknowledge is that a response probability has no impact on the underlying 
>ability distribution. Even Andy Kolstad, one of the most ardent proponents 
>of lower RP values, was quoted in the Washington Post as stating that the 
>"true literacy proficiencies of the population remain as reported...the 
>response probability convention influences how the results are interpreted 
>not how well adults perform on the assessment."
>
>Tom is correct when he says that an RP 50 is the point where one has equal 
>errors about whether a person can or cannot perform a task. Put another 
>way, however, RP 50 is also the point at which the least is known about 
>whether an individual can or cannot perform a task correctly. If the goal 
>is to know whether or not someone is likely to answer a question correctly 
>one needs to set the RP criterion higher. Thus setting an RP is a judgment 
>call that depends on the inferences one wants to make. Few employers or 
>patients would set a low criterion.
>
>Thus, response proficiencies are something that one applies after the fact 
>to add interpretative value to the underlying proficiency scales. Setting 
>an RP is a judgment call that depends on the inferences one wants to make. 
>Adopting a less demanding response proficiency standard would reduce the 
>numbers of people classified in the lowest skill levels. Unfortunately, 
>however, it would also serve to blur the relationship of literacy to the 
>educational, social, health and labour market outcomes that are of central 
>concern to policy makers. I might forgive my tom cat if he only managed to 
>find his way home 50% of the time but the empirical evidence presented in 
>the NALS, IALS and ALL reports confirms that employers notions of mastery 
>are much more demanding – anything lower than 80% attenuates the 
>relationships with critical co-variates and hence the policy utility of the 
>findings.
>
>Recent data from a US study designed to explore the relationship of 
>component reading skills to the emergence of fluency also provides more 
>empirical support for applying an RP80 (see Strucker, J., Yamamoto, K. and 
>Kirsch,(2004), Component Skills of Reading: Tipping Points and Five Classes 
>of Adult Learners, ANCLI, Lyon)
>
>Change the RP criterion means changing the interpretation of what it means 
>to be in a proficiency level. In NALS, IALS and ALL someone who is in the 
>middle of Level 2 would be expected to get a score of 80 percent correct on 
>a hypothetical test drawn from Level 2 items or a grade of "B" in most 
>American schools. Using an RP 50 criterion, an individual in the middle of 
>Level 2 would be expected to get a grade of 50 on a hypothetical test made 
>up of items that fell in Level 2 – equivalent to an "F" on that test, 
>whereas an RP 67 would yield a "D". Not many parents would be pleased with 
>such performance.
>
>Finally, Tom seems to have missed the fact that the notion of literacy 
>embodied in NALS, IALS and ALL is a dynamic one in which functionality can 
>only be judged against the reading demands that individuals are likely to 
>confront in their daily lives. It is a given that individuals are far more 
>proficient with familiar tasks than implied in the NALS, IALS and ALL 
>results. This, however, misses the point.
>
>It is of critical importance to acknowledge that reading demands change, 
>either in response to changes in society and technology, to natural changes 
>that occur as adults negotiate the life course, or because adults develop 
>goals or aspirations that involve a need to master new, unfamiliar reading 
>tasks.
>
>In this dynamic view, fluent reading, as defined by Level 3, can be thought 
>of as a tool that helps individuals cope efficiently with such change. The 
>analyses presented in the ALL comparative report provide compelling 
>evidence that the transferable skills measured in the study are what 
>matter. Individuals, and nations, that fail to attain Level 3 proficiency 
>pay a serious price – wage inequality is growing, not shrinking and it is 
>low skilled that suffer.
>
>It is expected that the rapidly increasing global supply of literacy skill, 
>and literacy's link to information and communication technologies and 
>productivity, will amplify these effects. This evidence has played a 
>critical role in convincing several countries to invest heavily in adult 
>education and training. Swiss data presented in he ALL report reveals that 
>policy can improve the level and social distribution of literacy skill over 
>the course of a decade.
>
>Crying wolf is ok if the wolf is indeed at the door. Perhaps it is Tom who 
>is "ALL wrong."
>
>
>
>_______________________________________________
>Reading-hall-of-fame mailing list
>Reading-hall-of-fame at nottingham.ac.uk
>http://lists.nottingham.ac.uk/mailman/listinfo/reading-hall-of-fame
>
>
>  
>