Writing Effective Multiple Choice Questions

1. To get at application and analysis, use vignette-style questions.

• Describe an event, happening, circumstance, or other scenario
• incorporate material being tested in a new way

2. Have students interpret novel material that is referenced over several questions.

• Present map, reading, or data set that students haven’t seen before that requires interpretation

3. Consider principles of Universal Design for Learning (UDL) to write questions that are  accessible to all learners.

• Avoid idioms and low frequency words ex. I’ve been asked to define ‘ambiguous’, ‘hooves’, and ‘kettle’ on exams – words unrelated to course content
• Use plain language and simple, straightforward grammar
• Tell your students to ask if they’re confused about question wording or don’t know the definition of something (but if it’s course related you may decline to answer)

1. Write statements that are true but don’t answer the question.

• “Gravity is the force of attraction between two masses” is true, but it doesn’t answer the question, “Which Law of Thermodynamics explains [the scenario]?”

2. Write statements that might seem right to the student but are incorrect.

• Each incorrect distractor should be plausible

3. If you use a pair of concepts in the options, make it two pairs

• When you pair answers like covert and overt, or meiosis and mitosis it’s usually one of the pairs, so try to ‘pair pairs’
• eg. (a) Overt (b) Covert (c) Endogenous (d) Exogenous

4. Try to make all options the same length.

• The correct answer is often longer because it requires more explanation

5. Avoid using “All of the above”, “None of the above”

• Students are more likely to get these questions correct because these options are disproportionately the right answer
• if you insist on using these options, make sure they are only correct 1/4 times that you use them

6. Avoid using “A, B and D” style options

• These options are often targeting a different construct, such as working memory and executive functioning
• These options require students to compare 3-5 statements simultaneous, but people tend to max-out at keeping track of 4 pieces of information at once

1. Align exam questions with course learning outcomes

• Be cognizant of how the questions align with the course learning outcomes to ensure that you’re adequately assessing different outcomes in your exam
• Some learning outcomes will not be assessable with MCQ!
• And Some learning outcomes are not  appropriate for exams at all e.x. “Find and summary research from the primary literature” is probably better assessed with a research assignment

2. Make the distribution of answers even among As, Bs, Cs, and Ds

• Instructors tend to want to “hide” their answers in Bs and Cs
• students can score above chance, even if guessing

3. Use 3 – 4 options total

• More options do not improve discrimination for questions
• if you have 5 options, there is almost definitely a distractor that few students select – just get rid of it, it will not change your typical average for the exam¹

LOW DIFFICULTY

p(correct) > .8

MEDIUM DIFFICULTY

p(correct) from 0.5 – 0.8

HIGH DIFFICULTY

p(correct) < 0.4

LOW DI

0 – 0.15

• Consider eliminating question on the next iteration – it’s probably inflating the grades

• Consider question carefully – is it too focussed on recall?

• Consider eliminating question
• Indicates a poorly-written question or unclear teaching of the concept

HIGH DI

0.15 – 0.5+

• Keep, but consider ways of revising to increase cognitive complexity

• Keep! The ideal question will have a p(correct) = 0.7 & DI > .25+

• Consider revising
• If the DI is high enough, it’s probably worth keeping but try to lower the difficulty by revising wording or distractors