[Maths-Education] Re: Validity of muliple choice questions

[dylanwiliam at mac.com]

Dear Tandi

I would be suspicious of anyone who makes such claims, since they seem to me to be essentially meaningless.

I would agree that it is generally easier to write constructed-response items that tap higher order skills than it is to do the same with lower order skills, but the overlap between the two types is huge. Multiple choice items can be used to assess higher-order skills and constructed-response items can be used to test lower-order skills. Too often people making such claims compare good constructed-response tests with poor multiple-choice ones.

More importantly, it is now widely agreed that validity is a property of inferences based on evidence elicited by assessments, rather than of the assessments themselves. So it doesn't make sense that the validity of one test is less than that of another. Typically, multiple-choice tests cover more content, and so can better support inferences related to the breadth of knowledge, whereas inferences relating to the ability to construct proofs, or to communicate mathematically would probably be better supported by constructed response items.

Differences in perceptions of validity are often differences in construct definition—the dispute arises in the context of the assessment, but are really dispute about what mathematics is.

I have written about this in the following chapter in AERA's "Review of Research in Education" last year

Wiliam, D. (2010). What counts as evidence of educational achievement? The role of constructs in the pursuit of equity in assessment. In A. Luke, J. Green & G. Kelly (Eds.), What counts as evidence in educational settings? Rethinking equity, diversity and reform in the 21st century (Vol. 34, pp. 254-284). Washington, DC: American Educational Research Association.

Dylan


On 11 Aug 2011, at 10:39, Tandi Clausen-May wrote:

> ***********************************************************************************************************
> This message has been generated through the Mathematics Education email discussion list.
> Hitting the REPLY key sends a message to all list members.
> ***********************************************************************************************************
> I urgently need references to research evidence that 'suggests that open
> response questions tap higher order skills than multiple choice questions',
> or that 'the validity of multiple choice questions is less than that of open
> response questions' in mathematics assessment at school level (Key Stage 2,
> 3 or 4).   Can anybody help?
> 
> Thanks!
> 
> Tandi
> 
> Tandi Clausen-May
> Independent Consultant
> Mathematics curriculum and assessment
> Special education
> 
> Tel: 01628 776 204
> 
> 
> 
> 
> This message and any attachment are intended solely for the addressee and may contain confidential information. If you have received this message in error, please send it back to me, and immediately delete it.   Please do not use, copy or disclose the information contained in this message or in any attachment.  Any views or opinions expressed by the author of this email do not necessarily reflect the views of the University of Nottingham.
> 
> This message has been checked for viruses but the contents of an attachment
> may still contain software viruses which could damage your computer system:
> you are advised to perform your own checks. Email communications with the
> University of Nottingham may be monitored as permitted by UK legislation.
>