Writing great questions
Definitions & Key takeaways
When you’re learning something new, it’s sometimes tricky to know whether you really understand the material, and one of the best ways to do that is by taking a test that includes well written questions that assess whether you really do understand the material.
In fact, great questions - ones that are really well written - make a big difference in assessing not only the facts that you know, but how you apply those facts to solidify and retain that knowledge.
So how are great questions written? Well, when it comes to multiple choice questions – there are some ways to avoid making the answer obvious to a person who is test-wise, in other words, someone who can pick up on subtle unintended clues, and narrow down the choices or zero in on the answer without actually knowing much about the question topic.
The first trick is to create good distractors that can’t be easily eliminated. First – make sure that all of the options follow grammatically from the stem.
For example, look at this set of options - only B, C, and E follow grammatically from the lead-in. For testwise students - and by students I mean learners of any level that are answering the question - this sticks out like a weed on the lawn: A and D would be eliminated right away because the vowel at the beginning of these options calls for “an” at the end of the lead-in.
This is a typical error because question-writers generally make sure that the correct answer is grammatically consistent, but may be less careful with the distractors.
One easy way to avoid this mistake entirely is to have the lead-in be a complete sentence that ends with a question mark.
Second, when response options are part of a logical set, don’t include options that are not. For example, look at these options, three of them cover all of the possible options for morphine - less morphine, more morphine, or the same amount of morphine, and then the last option seems like an afterthought that has no real relationship to the others.
Students would eliminate D right away. If a set of options is mutually exclusive and exhaustive, then one of them has to be correct, and there is no point in including additional options.
Third, you want to avoid absolute terms like "always" or "never" because that’s a red flag since extreme statements are unlikely to be true.
Here’s an example where option A and B both violate this rule. It’s also a good idea, though, to avoid wishy-washy words like often, typically, and sometimes, since they’re more likely to be correct.
Now, let’s focus on tips for creating a good correct answer, that is - one that will blend in with the distractors. First, you want to make sure that the correct answer avoids convergence, which is where the correct answer has the most elements in common with other options.
Take a look at this example. Oftentimes, a question writer prepares a question stem and a correct answer, and distractors are generated by writing variations on the correct answer.
Students can easily work backwards - by looking for word repeats and picking the option that has the largest number of elements in common with other options - in this case option A has the most repeats with Tobramycin coming up three times and Ticarcillin coming up twice.
Second, you want to make sure that the correct answer has similar length and complexity as the distractors. Take a look at this example, where option D is so much longer and full of nuance compared with the distractors.
Students would probably choose it just because it looks like much more thought went into writing that option versus the others.
And it’s probably correct because item writers often take more time writing the correct answer - adding extra details to make sure it is clearly the most correct option and doing a little teaching along the way to make it a good learning experience.
Similarly, if only one option is a “double option,” it is likely to be correct. Okay - so those tips will help with any multiple choice question, but in the health sciences there are some specific ways to make questions really awesome.
First and most important of all, the question should be about the application of knowledge rather than the memorization of isolated facts.
Questions should focus on common and challenging issues in topics that affect patient care, with a particular emphasis on common mistakes and misunderstandings.
To do this well, it’s best to avoid cliche buzzwords like blue sclerae and cogwheel rigidity, and instead use less-interpreted clinical descriptions of findings.
This side-by-side example shows what the difference looks like, with the recall version using the term “acute ischemic stroke”, whereas the application version uses the clinical description “sudden onset of weakness of the left arm and leg and drooping of the mouth”.
The options are the same, but the application version is more challenging because it requires interpreting information, rather than simply giving away the diagnosis.
Next, try to write the question stem in the form of a story - a vignette, it could be about a patient, a clinical practice issue, or even a research study.
Since the patient vignette is most common let’s run through that one in a bit of detail. For starters, it’s good to make sure that each relevant part of the patient vignette is included in the following order: patient age and gender, site of care, chief complaint and duration of symptoms, pertinent history (HPI, personal and family history, medications), and physical exam findings; and in some cases, results of diagnostic studies and response to initial treatment.
It’s always good if you can use real clinical photos, ECGs, and imaging studies because that makes the question more realistic, and challenges students to interpret information.
After the question is framed up, there is a “lead-in” which is the final part of the question stem that leads into the options.
A good “lead-in” clearly defines the relationship between the stem and option set. In fact, it should be possible to answer a question (or come pretty close) with the options covered up.
When that’s not the case, it generally means that the lead-in is too vague. For example, a lead-in like “which of the following statements is true” can’t possibly be answered if the options are covered.
For the same reason, it’s best to avoid negating words like “least”, “except”, or “not”, because there may be multiple correct answers, which means that there’s not enough specificity.
Now, focusing on the options, it’s important to make them as homogenous as possible so they can be rank-ordered along a single dimension - for example, the likelihood of a diagnosis, going from most likely to least likely.
This also allows you to assess clinical judgment, challenging students to make more subtle distinctions among the options.
In contrast, there could be a correct answer with three random options as distractors that don’t have any relationship with each other or with the question stem.
A question like that makes the question easier to answer, tends to test knowledge of isolated facts, and offers less learning value.
As a final point, once the question and options are written out, it’s good to take a step back and make sure that it has a long stem and short options that don’t have any redundant text.
That way, the students are spending most of their time reading and understanding the clinical situation and then answering a relatively focused question about it.
Okay - quick recap - writing good questions in general can be done by making sure that you have a good story - vignette - and that the option set is well written so that the correct answer blends in well with the distractors.
For health science questions - it’s important to make sure that the question tests a specific clinical application of knowledge with options that can be rank-ordered along a single dimension, rather than testing recall of isolated facts.
No notes for this video yet
Try adding a note below