[Metal] Automatic generation of random formative and summative multiple choice tests

[Joel Feinstein]

Hi everyone,
I'm looking at the possibility of getting funding from the latest University HERMES call for the following project. However initial feedback from them says that funding is unlikely unless I have strong evidence of interest from other Schools in the University and beyond.
This appears unlikely for this round (deadline May 21st), so I am probably looking at the next round (October 2012). On the other hand, to quote our research facilitator David Hawker:

There will be a further round of Hermes in October, so if we can make some phone calls and enquiries by then and garner support for your idea, the University sounds as if it would be behind it. Of course if you are able to whip up a frenzy of interest in the next few days we can still go for the current call!

So, is there a frenzy of interest?
Best wishes,
   Joel

Automatic generation of random formative and summative multiple choice tests

What forms of assessment are both efficient and effective in encouraging student engagement?

Assessed coursework is a possibility, but has high overheads in terms of marking time, and there is also the issue of plagiarism.

Oral presentations and examinations can be an excellent way to measure understanding, but are usually impractical for large classes.

Multiple choice tests are very efficient on the marking side (machine marking), but the design of good questions is time-consuming, and there are currently limitations on the variations which can be randomly generated from a particular question type. Some randomization of numbers in questions and of the order of the questions is possible in some systems, but there is currently little scope for randomization of the variable names, function names, language expression, etc.

The proposed project involves investigation of the best way to generate large numbers (tens of thousands) of questions from any given prototype, by means of suitable randomization of numbers, variable names, function names, and language used (e.g., for quantifiers in mathematics, symbols can be used, or one can use "for all", "for each", "for every", "there exist(s)", "there are/is", "there corresponds", etc.). Ideally the questions could be output from the software in a form suitable for input into a variety of online systems (e.g. rogo, moodle, ...)

I already have a few sample prototype questions which can be used to test students' understanding of the use of mathematical quantifiers. I would need a substantial amount of dedicated time in order to develop software tools of my own and futher question prototypes in order to generate substantial question banks.

Although my initial proposal is for mathematics tests, the project could be extended to many other areas.

I would envisage the questions generated being used in the following way: having chosen which type of question will appear in a paper-based multiple-choice class test, the students are allowed to practice (formatively) this type of question as much as they like online, with instant feedback. Even if the class test is worth very little credit, it is likely that the students will put in a lot of work on the formative questions to prepare for the test.

I believe that a similar formative/summative system has been tried in Engineering with considerable success.

I believe that it would be a rather efficient use of staff time to generate prototype questions which can then be used again and again. There would be no need to generate new prototypes every year (though it may be that new prototypes would be introduced from time to time).


The benefits:

Increased student engagement.
Better levels of student understanding.
Essentially unlimited, ready-made assessments of students, with minimal staff input required.

These benefits are for us directly, and for anyone else who uses the system. Obviously, the fact that the system originates in Nottingham will raise the profile of Nottingham's teaching.

--
 Dr J. Feinstein       Tel:  +44-115-9514968
Pure Mathematics      FAX:  +44-115-9514951
Nottingham NG7 2RD    Web:  http://www.maths.nott.ac.uk/~jff
United Kingdom        Blog: http://explainingmaths.wordpress.com

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.nottingham.ac.uk/pipermail/metal/attachments/20120514/ea705c78/attachment.html>