[Reading-hall-of-fame] Adult Literacy Assessment at Issue

[tsticht at znet.com]

Colleagues: I understand RHF member P. David Pearson oversaw the external 
review of the NAS/NRC report on adult literacy assessment that I discuss in 
relation to the new Adult Literacy and Lifeskills (ALL) report from 
Statistics Canada, OECD, and NCES. I think this illustrates what happens 
when government agencies invest lots of money,publicity, and egos into 
activities that are very socially, politically, and technically complex. It 
will be difficult, if not imppossible, for the ALL folks to adapt the 
recommendations of the NAS/NRC report because of their past commitment to 
the International Adult Literacy Survey with all its faults. I can't 
remember where I read it, but I recall something about "Oh what a tangled 
web we weave when first we practice to deceive!" I think there may be at 
least some self-deception at work here - and perhaps a bit of public 
deception too....? Tom Sticht






May 17, 2005

ALL Wrong – Again! Can Adult Literacy Assessments Be Fixed?

Tom Sticht
International Consultant in Adult Education

A new report on adult literacy jointly produced by Statistics Canada, the Organization for Economic Co-operation and Development (OECD), and the 
United State’s National Center for Education Statistics is once again 
stimulating calls for action, just as the International Adult Literacy 
Survey (IALS) did when it was released in the mid-1990s. The new report 
presents data from the new international Adult Literacy and Lifeskills 
(ALL) survey. 

Headlines from a press release dated May 11, 2005 from a national advocacy 
group for adult literacy in Canada says, "ABC CANADA calls for national 
strategy following latest adult literacy stats : 42 per cent of Canadian 
adults do not meet minimum literacy levels for coping."  The Press Release 
goes on to state, "After nine years, the same per cent of adults have low 
literacy.   Regarding those Canadians, aged 16 to 65, scoring literacy 
Levels 1 and 2, deemed below the minimum of what is suitable for coping 
with the demands of everyday life and work, the percentage reported in ALL 
is the same as was recorded in 1994 (42 per cent). Level 1 literacy skill 
means a person is unable to read, for example, information on a medicine 
bottle. Level 2 means the person can only deal with simple printed 
material, and has difficulty facing new demands or tasks at work or in the 
community."

The same day, another press release, this time from the Movement for 
Canadian Literacy (MCL), another group that advocates for adult literacy 
education, cited the ALL report and stated,  "Too many Canadian adults (4 
out of 10) score below the skill level necessary to meet the everyday 
demands of our information society."

The statements by the adult literacy advocacy organizations about the 
percentages of adults with literacy below the minimum for coping with the 
demands of everyday life and work are based on statements in the ALL report 
that "Depending on the country, between one-third and over two-thirds of 
adult populations do not attain skill Level 3, the level considered by 
experts as a suitable minimum level for coping with the increasing demands 
of the emerging knowledge society and information economy (OECD and 
Statistics Canada, 1995)."

However, as its justification for the idea that skill Level 3 (out of five 
possible skills levels) is considered by experts as a suitable minimum for 
coping in today’s societies the ALL report cites an earlier IALS report 
from 1995. But neither that report nor the present ALL report presents any 
names of experts who have reached that conclusion nor any studies that 
support the statement. 

Another new report, this one from the U. S. National Academy of Sciences, 
National Research Council (NAS/NRC)  makes the point that the NALS 
methodology used by the ALL can not reveal "the level considered by experts 
as a suitable minimum level for coping with the increasing demands of the 
emerging society."  The Executive Summary of the NAS/NRC report states that 
the methodology of the NALS did not provide information about the 
"mismatch" of skills of adults and demands of the economy, or anything else 
for that matter, but that some "unsupported inferences" along these lines 
were made by some. 

Quoting from the NAS/NRC report, "...the procedures used to develop the 
[NALS] assessment did not involve identifying the level of skills adults 
need in order to function adequately in society. When findings from the 
1992 survey were released, however, the performance levels were interpreted 
and discussed as if they represented standards for the level of literacy 
adults should have. The lowest two levels were referred to as inadequate, 
so low that adults with these skills would be unable to hold a well-paying 
job. The results of the assessment and these sorts of unsupported 
inferences about the results provoked widespread controversy in the media 
and among experts in adult literacy about the extent of literacy problems 
in the country."

Unfortunately, the ALL report continues this practice of drawing 
"unsupported inferences" about the levels of literacy needed by adults to 
meet the demands of contemporary societies in the industrialized world. 

The NAS/NRC report also points out the inaccuracy of statements such as in 
the ABC CANADA press release that "Level 1 literacy skill means a person is 
unable to read, for example, information on a medicine bottle. Level 2 
means the person can only deal with simple printed material, and has 
difficulty facing new demands or tasks at work or in the community." The 
NAS/NRC report indicates that just because a person is assigned to Level 1 
or Level 2 does not mean he or she cannot perform tasks at higher levels. 
In fact NALS data indicated that adults might be able to perform as many as 
25 percent, 50 percent  or even more tasks at higher levels, they simply 
could not perform them with an 80 percent probability of success. 

The ALL survey also continues the earlier NALS/IALS practice of using the 
80 percent probability of a correct response as the standard for 
proficiency. But the new NAS/NRC report states that  "The committee judged 
that a probability level of 80 percent was overly stringent given the uses 
of the assessment results." The NAS/NRC report presents data indicating 
that from a strictly statistical point of view, the most accurate, that is 
the most valid, response probability for setting proficiency levels is .50. 
The .50 probability level is the point at which the errors in making 
statements about whether adults can or cannot perform certain literacy 
tasks are equal. Using the .80 response standard, one is four times as 
likely to make the mistake of  saying that someone cannot do certain 
literacy tasks when in fact they can as they are to say a person can do 
literacy tasks when in fact they cannot. This greatly increases the 
percentage of adults who will be declared of low literacy.

In a paper for the Canadian Journal for the Study of Adult Education 
(Sticht, 2001) I presented a set of arguments against the use of the .80 
standard which are confirmed by the new NAS/NRC report. Using a .50 
standard for proficiency would reduce the percentages of adults placed in 
the lowest level of literacy in the ALL survey by about half. This means 
that the percentage of adults in Canada, for instance, in the lowest two 
levels of literacy, would drop from 42 percent to around 20 percent. 

The NAS/NRC report argues for a standard of .67 as a compromise between the 
.5 and .8 standards, while acknowledging that this is also judgmental and 
less accurate as was the use of the 80 percent probability standard in the 
ALL. 

Somewhat surprisingly,  citations and footnotes by the authors of the new 
ALL survey report indicate that the authors were aware of the many 
criticisms of validity of the NALS/IALS raised in earlier reports (e.g., 
Sticht, 2001; Kirsch et al, 2001) and by the new National Research Council 
report. But they chose to ignore the criticisms and to continue to 
propagate new data with the same faults as the earlier IALS data. In turn, 
this places adult literacy advocates and their organizations in the 
position of relying upon information of questionable validity about the 
scale of need for their services. By overstating the nature of the adult 
literacy "crisis" advocates run the risk of being seen as crying "wolf" 
when in most peoples view, they cannot see the wolf. 

Indeed, in the IALS survey, most adults, even those in the lowest level of 
literacy thought they could read, write and compute well enough to meet the 
demands of their jobs and daily lives. Given the many methodological 
problems of the IALS, it is not possible to determine whether these adults 
were correct or not in their self-perceived literacy and numeracy 
competence. I have not found any data on adult’s self-perception’s of 
literacy ability in the new ALL report. But it is likely that most adults 
still do not think that their literacy skills are standing in the way of 
their ability to cope with most of the demands for literacy and numeracy of 
the new "knowledge" or "information" societies. If this is the case, it 
will take a considerable effort to attract adults into programs to improve 
what they do not think needs much improvement. The present relatively low 
rates of enrollment in adult literacy programs in industrialized nations 
seem consistent with this point of view.

 For most adults, the old maxim seems to apply, "If it ain’t broke why fix 
it?"

This doesn’t apply to adult literacy assessments. They do need to be fixed. 

References

Committee on Performance Levels for Adult Literacy, et al. (Eds.). (2005). 
Measuring Literacy: Performance Levels for Adults, Interim Report. 
Washington, DC: National Academy Press. Available online at www.nap.edu/
catalog/11267.html

Kirsch, I. et al (2001). Technical Report and Data File User’s Manual for 
the 1992 National Adult Literacy Survey. NCES 2001-457. Washington, DC: U. 
S. Department of Education, National Center for Education Statistics. 

Statistics Canada (2005). Learning a Living: First Results of the Adult 
Literacy and Life Skills Survey. Available online at www.statcan.ca

Sticht, T. G. (2001). The International Adult Literacy Survey: How Well 
Does It Represent the Literacy of Adults?. The Canadian Journal for the 
Study of Adult Education, 15, 19-36.  

Thomas G. Sticht
International Consultant in Adult Education
2062 Valley View Blvd.
El Cajon, CA 92019-2059
Tel/fax: (619) 444-9133
Email: tsticht at aznet.net