[Reading-hall-of-fame] Adult Literacy Assessment at Issue

[P. David Pearson]

Tom and RHF members,

A couple of comments.  As I read your comments in 
the missive below, it looks to me that we are in 
full agreement about the arbitrary and dangerous 
character of "achievement levels" in the NALS 
and, I would argue, in ANY assessment.  I thought 
that the NAS/NRC report (by the way, my job as 
review coordinator was more to adjudicate the 
fairness and validity of the reviews than the 
report itself).  Achievement levels have the same 
virtues and vices as any procedure that 
"pretends" that all of the items assigned to a 
band (e.g., below basic, basic, proficient, 
advanced) are alike in some conceptually 
distinguishable way.  The problem in the NALS is 
not quite as bad as the problem with NAEP because 
NALS has a more robust distribution of items 
across all levels than does NAEP.  NAEP has very 
few items to measure the "floor" at 4th grade. 
Basically, what we know about the lowest 
performing tier of 4th graders is that they don't 
do well responding to items on "grade level" 
passages.  There are so few easy passages in the 
4th grade NAEP that we cannot accurately estimate 
what they CAN do.

In my view, the achievement levels approach, as a 
mode of score reporting, is fundamentally 
misleading.   Achievement levels claim a real 
world connection (can do X or Y or Z) that is 
unwarranted.  When all is said and done, these 
assessments tell us little more than who did 
better than whom. We would be better off with 
some sort of standard score or even percentiles. 
At least then we would be making claims warranted 
by the test, and we would not permit politicians 
and policy makers to go around making 
preposterous claims, such as 40% of America's 4th 
graders read below basic or can't read 4th grade 
material.  Alas, the popularity of achievement 
levels suggests to me that we will have them 
around for a long time.

Given the inevitability of achievement levels, 
then the questions that Tom raises are critical. 
What sort of standard do we set at the item level 
for proficiency?  the higher the level of 
proficiency, the more alarming will be the claims 
about performance made from the data.  Our stance 
in the report was to discourage

David

>Colleagues: I understand RHF member P. David Pearson oversaw the external
>review of the NAS/NRC report on adult literacy assessment that I discuss in
>relation to the new Adult Literacy and Lifeskills (ALL) report from
>Statistics Canada, OECD, and NCES. I think this illustrates what happens
>when government agencies invest lots of money,publicity, and egos into
>activities that are very socially, politically, and technically complex. It
>will be difficult, if not imppossible, for the ALL folks to adapt the
>recommendations of the NAS/NRC report because of their past commitment to
>the International Adult Literacy Survey with all its faults. I can't
>remember where I read it, but I recall something about "Oh what a tangled
>web we weave when first we practice to deceive!" I think there may be at
>least some self-deception at work here - and perhaps a bit of public
>deception too....? Tom Sticht
>
>
>
>
>
>
>May 17, 2005
>
>ALL Wrong – Again! Can Adult Literacy Assessments Be Fixed?
>
>Tom Sticht
>International Consultant in Adult Education
>
>A new report on adult literacy jointly produced 
>by Statistics Canada, the Organization for 
>Economic Co-operation and Development (OECD), 
>and the
>United State’s National Center for Education Statistics is once again
>stimulating calls for action, just as the International Adult Literacy
>Survey (IALS) did when it was released in the mid-1990s. The new report
>presents data from the new international Adult Literacy and Lifeskills
>(ALL) survey.
>
>Headlines from a press release dated May 11, 2005 from a national advocacy
>group for adult literacy in Canada says, "ABC CANADA calls for national
>strategy following latest adult literacy stats : 42 per cent of Canadian
>adults do not meet minimum literacy levels for coping."  The Press Release
>goes on to state, "After nine years, the same per cent of adults have low
>literacy.   Regarding those Canadians, aged 16 to 65, scoring literacy
>Levels 1 and 2, deemed below the minimum of what is suitable for coping
>with the demands of everyday life and work, the percentage reported in ALL
>is the same as was recorded in 1994 (42 per cent). Level 1 literacy skill
>means a person is unable to read, for example, information on a medicine
>bottle. Level 2 means the person can only deal with simple printed
>material, and has difficulty facing new demands or tasks at work or in the
>community."
>
>The same day, another press release, this time from the Movement for
>Canadian Literacy (MCL), another group that advocates for adult literacy
>education, cited the ALL report and stated,  "Too many Canadian adults (4
>out of 10) score below the skill level necessary to meet the everyday
>demands of our information society."
>
>The statements by the adult literacy advocacy organizations about the
>percentages of adults with literacy below the minimum for coping with the
>demands of everyday life and work are based on statements in the ALL report
>that "Depending on the country, between one-third and over two-thirds of
>adult populations do not attain skill Level 3, the level considered by
>experts as a suitable minimum level for coping with the increasing demands
>of the emerging knowledge society and information economy (OECD and
>Statistics Canada, 1995)."
>
>However, as its justification for the idea that skill Level 3 (out of five
>possible skills levels) is considered by experts as a suitable minimum for
>coping in today’s societies the ALL report cites an earlier IALS report
>from 1995. But neither that report nor the present ALL report presents any
>names of experts who have reached that conclusion nor any studies that
>support the statement.
>
>Another new report, this one from the U. S. National Academy of Sciences,
>National Research Council (NAS/NRC)  makes the point that the NALS
>methodology used by the ALL can not reveal "the level considered by experts
>as a suitable minimum level for coping with the increasing demands of the
>emerging society."  The Executive Summary of the NAS/NRC report states that
>the methodology of the NALS did not provide information about the
>"mismatch" of skills of adults and demands of the economy, or anything else
>for that matter, but that some "unsupported inferences" along these lines
>were made by some.
>
>Quoting from the NAS/NRC report, "...the procedures used to develop the
>[NALS] assessment did not involve identifying the level of skills adults
>need in order to function adequately in society. When findings from the
>1992 survey were released, however, the performance levels were interpreted
>and discussed as if they represented standards for the level of literacy
>adults should have. The lowest two levels were referred to as inadequate,
>so low that adults with these skills would be unable to hold a well-paying
>job. The results of the assessment and these sorts of unsupported
>inferences about the results provoked widespread controversy in the media
>and among experts in adult literacy about the extent of literacy problems
>in the country."
>
>Unfortunately, the ALL report continues this practice of drawing
>"unsupported inferences" about the levels of literacy needed by adults to
>meet the demands of contemporary societies in the industrialized world.
>
>The NAS/NRC report also points out the inaccuracy of statements such as in
>the ABC CANADA press release that "Level 1 literacy skill means a person is
>unable to read, for example, information on a medicine bottle. Level 2
>means the person can only deal with simple printed material, and has
>difficulty facing new demands or tasks at work or in the community." The
>NAS/NRC report indicates that just because a person is assigned to Level 1
>or Level 2 does not mean he or she cannot perform tasks at higher levels.
>In fact NALS data indicated that adults might be able to perform as many as
>25 percent, 50 percent  or even more tasks at higher levels, they simply
>could not perform them with an 80 percent probability of success.
>
>The ALL survey also continues the earlier NALS/IALS practice of using the
>80 percent probability of a correct response as the standard for
>proficiency. But the new NAS/NRC report states that  "The committee judged
>that a probability level of 80 percent was overly stringent given the uses
>of the assessment results." The NAS/NRC report presents data indicating
>that from a strictly statistical point of view, the most accurate, that is
>the most valid, response probability for setting proficiency levels is .50.
>The .50 probability level is the point at which the errors in making
>statements about whether adults can or cannot perform certain literacy
>tasks are equal. Using the .80 response standard, one is four times as
>likely to make the mistake of  saying that someone cannot do certain
>literacy tasks when in fact they can as they are to say a person can do
>literacy tasks when in fact they cannot. This greatly increases the
>percentage of adults who will be declared of low literacy.
>
>In a paper for the Canadian Journal for the Study of Adult Education
>(Sticht, 2001) I presented a set of arguments against the use of the .80
>standard which are confirmed by the new NAS/NRC report. Using a .50
>standard for proficiency would reduce the percentages of adults placed in
>the lowest level of literacy in the ALL survey by about half. This means
>that the percentage of adults in Canada, for instance, in the lowest two
>levels of literacy, would drop from 42 percent to around 20 percent.
>
>The NAS/NRC report argues for a standard of .67 as a compromise between the
>.5 and .8 standards, while acknowledging that this is also judgmental and
>less accurate as was the use of the 80 percent probability standard in the
>ALL.
>
>Somewhat surprisingly,  citations and footnotes by the authors of the new
>ALL survey report indicate that the authors were aware of the many
>criticisms of validity of the NALS/IALS raised in earlier reports (e.g.,
>Sticht, 2001; Kirsch et al, 2001) and by the new National Research Council
>report. But they chose to ignore the criticisms and to continue to
>propagate new data with the same faults as the earlier IALS data. In turn,
>this places adult literacy advocates and their organizations in the
>position of relying upon information of questionable validity about the
>scale of need for their services. By overstating the nature of the adult
>literacy "crisis" advocates run the risk of being seen as crying "wolf"
>when in most peoples view, they cannot see the wolf.
>
>Indeed, in the IALS survey, most adults, even those in the lowest level of
>literacy thought they could read, write and compute well enough to meet the
>demands of their jobs and daily lives. Given the many methodological
>problems of the IALS, it is not possible to determine whether these adults
>were correct or not in their self-perceived literacy and numeracy
>competence. I have not found any data on adult’s self-perception’s of
>literacy ability in the new ALL report. But it is likely that most adults
>still do not think that their literacy skills are standing in the way of
>their ability to cope with most of the demands for literacy and numeracy of
>the new "knowledge" or "information" societies. If this is the case, it
>will take a considerable effort to attract adults into programs to improve
>what they do not think needs much improvement. The present relatively low
>rates of enrollment in adult literacy programs in industrialized nations
>seem consistent with this point of view.
>
>  For most adults, the old maxim seems to apply, "If it ain’t broke why fix
>it?"
>
>This doesn’t apply to adult literacy assessments. They do need to be fixed.
>
>References
>
>Committee on Performance Levels for Adult Literacy, et al. (Eds.). (2005).
>Measuring Literacy: Performance Levels for Adults, Interim Report.
>Washington, DC: National Academy Press. Available online at www.nap.edu/
>catalog/11267.html
>
>Kirsch, I. et al (2001). Technical Report and Data File User’s Manual for
>the 1992 National Adult Literacy Survey. NCES 2001-457. Washington, DC: U.
>S. Department of Education, National Center for Education Statistics.
>
>Statistics Canada (2005). Learning a Living: First Results of the Adult
>Literacy and Life Skills Survey. Available online at www.statcan.ca
>
>Sticht, T. G. (2001). The International Adult Literacy Survey: How Well
>Does It Represent the Literacy of Adults?. The Canadian Journal for the
>Study of Adult Education, 15, 19-36. 
>
>Thomas G. Sticht
>International Consultant in Adult Education
>2062 Valley View Blvd.
>El Cajon, CA 92019-2059
>Tel/fax: (619) 444-9133
>Email: tsticht at aznet.net
>
>
>
>
>_______________________________________________
>Reading-hall-of-fame mailing list
>Reading-hall-of-fame at nottingham.ac.uk
>http://lists.nottingham.ac.uk/mailman/listinfo/reading-hall-of-fame


-- 
+++++++++++++++++++++++++++++++++
PLEASE NOTE:  SOCRATES IS DEAD, at least as an e-mail server
  at Berkeley!!!!
PLEASE DROP IT FROM MY ADDRESS!
NEW ADDRESS: ppearson at berkeley.edu
+++++++++++++++++++++++++++++++++
***********************************
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
P. David Pearson
Professor and Dean
Graduate School of Education
1501 Tolman Hall #1670
University of California, Berkeley
Berkeley CA 94720-1670
GSE Office: 510 643-6644
LHS Office: 510 642-7753
5519 Research Office:  510 643-5004
fax 510 643-8904
speakerphone 510 643-6739
ppearson at berkeley.edu
website: 
*******************
Home:  851 Euclid Ave
Berkeley, CA  94708 -1305
510 526 6986
Cell:  510 543 6508
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
***********************************