There is more to MCQs than you think

BELONG - RESPECT - ACHIEVE - ASPIRE

There is more to MCQs than you think

24 Oct ’14 Teaching 3

I have been growing more interested in Multiple-choice Questions or (MCQs). It started off as a QuickKey interest but it has become much more than that. If I thought I might be a edugeek, this pretty much confirms it. An unhealthy interest in assessment and MCQs is unlikely to make me a highly sought after dinner guest.

Reassuringly, I am not the only MCQ enthusiast. Exams watchdog Ofqual suggested that A levels could move away from their “exclusive focus on reading and interpreting traditional forms of text” and introduce more “multiple choice questions.” Shock horror. What is more there are over eight million hits on “how to write effective multiple choice questions,” and just shy of seven million hits focusing on MCQs purely on “higher order skills.” Unsurprisingly in this hyper-connected edu-landscape we work in, MCQs gets a lot of attention from online educationalists and assessment gurus.

Right, Multiple-choice Questions (MCQs) are a subset of “objective questions.” I didn’t even know that there were “sets” of question types at this point, let alone “subsets.” Objective questions are basically those questions which have a correct answer (usually, though not always, one correct answer.)

The term “objective” in this set of questions means there is complete objectivity in marking the test. Wrong is wrong. Correct is correct. However, that does not make MCQs infallible, of course, as the construction of question itself is fallible (hence the five million hits on how to write effective MCQs) even if the answer should not be.

So, to press on, this is what I have learnt so far…

Advantages

    • MCQs are predominantly used to test factual material and the understanding of concepts (discounting ARQs the new MCQ on the block – more on those later).
    • MCQs are rapidly and automatically marked (especially with QuickKey) – therefore there is no prior knowledge required to be an assessor.
    • Scoring is objective and reliable – no examiner bias or halo effect.
    • MCQs assessment highlights students knowledge and Question Level Analysis can also be used to highlight misunderstanding and common misunderstanding (ineffective teaching). QLA in principle is straightforward, however there is debate on how to most effectively apply QLA.
    • The validity, internal reliability of MCQ quizzes can be statistically assessed; Kuder-Richardson formula, Cronbach’s (alpha), a measure of internal consistency, distribution scores, scaled scores (z and T-scores). 
    • MCQ quizzes can be given a difficulty rating or (p-value). The higher the value the easier the question, or quiz.* Having discussed this with the staff at Boundary Oak School – I have posted an answer.
    • Question Level Analysis also provides information on how an exam itself can be improved (more of that in Part II).

Disadvantages

  • MCQs take time to set.
  • MCQs provide no credit for partial information or correct processes.
  • MCQs accept that some correct answers can be guessed.
  • High quality objective test require skilled question writers.

While the following notes refer specifically to the most common form of objective tests (MCQs), many of the comments are relate to all objective questions.

Characteristics of multiple-choice questions (MCQs)

MCQs are usually composed of four parts: the stem (the question or incomplete statement), the options (the suggested answers or completions), the incorrect responses, sometimes distractors in more subtle questions, and the key or correct response.

Up until recently, most MCQ stems I had seen were written questions, however working the Science department has shown me that graphs, diagrams or sets of results are common. Using these MCQ props leads to abilities other than recall being tested. Who said MCQs were easy? Not me, that is for sure.

As for the use of distractors, MCQs with carefully selected distractors, or closely matched keys and distractors certainly calls for more sophisticated thinking.

If you are debating four or five options. It is hardly surprising that five options provides the more reliable test results. The drawback, you have to think of another plausible distractor.

Five Basic MCQ types (plus one late entrant)

There are a wide range of MCQs that can be created both on and offline. Given that QuickKey and most teaching is offline, I have focused on the five basic offline MCQ types, setting aside rank ordering, drag and drop and free response questions.

Best answer

Composed of a “stem’ and generally 5 “distractors”, 1 of which is the best response.

Right answer

Composed of a “stem’ and generally 4 “distractors”, and 1 key response.

Multiple right answer

Composed of a “stem’ and generally 2 or 3 “distractors”, 3 or 2 key responses. Sometimes the answers are predefined. For example;

  1. 1 and 3 are correct
  2. 1 and 4 are correct
  3. 1, 2 3 are correct
  4. All are correct
  5. Only 3 is correct

Or there is this option taken from a Dylan Wiliams seminar back in 2011.

MCQAsk MCQs with more than one correct answer. Do your students really know the answer? In this example, if there are two correct answers that probability of getting it both correct decreases from 1 in 6, to 1 in 15 (2 in 6 x 1 in 5) chance. I later learn from Dylan Wiliam’s, not knowing how many correct answers there are decreases the chances of guessing to 1 in 64 (2^6).  (Lets not forget the ACE up your sleeve. If you really want to know, if your students fully understand the topic, don’t inform them that there are two or more correct responses and simply wait for the insightful student with a secure understanding to ask you…

Err, Sir… aren’t both A AND E right?!

Reason-assertion or ARQ

Each item in an assertion/reason questions consists of two items an assertion and a reason that are linked together. The student is required to first decide whether assertion is valid (true / false), and then decide whether the reason is valid (true/false). If one or both of the statements is false then the answer will be either c, d, or e. If both statements are true, a third step is required whereby the students must determine whether the second statement provides an accurate explanation for the first.

  • True True – Reason is correct explanation
  • 1 and 4 are correct – Reason is NOT correct explanation
  • True False
  • False True
  • False False

Here is an example using the Hulstrom curve that shows the relationship between particle diameters and entrainment velocities for water taken from GCE Geography.

Hjulstrom curve

Assertion Reason
Pair 1 Sand has a lower critical entrainment velocity than clay BECAUSE Sand particles are lighter than clay particles

I have not personally used ARQs but these seem tough. In researching ARQs most exemplar appear to be scientific, medical or business / economics and linked to higher education courses. Which makes sense, as doctors, pharmacists and economists make reasoned assertions.

According to Williams (2006) “traditional MCQs usually test only one issue/concept. ARQs, on the other hand, test two per question (the assertion and the reason statements) plus the validity of the ‘because’ statement in the event assertion and reason are both correct statements. On the basis that judging the correctness of two statements must be harder than judging the correctness of one, it would follow that ARQs present more of an intellectual challenge than traditional MCQs.” Simple. these questions are good for testing whether students can see or reason out the relationship between different concepts or events. However, they are hard to write because they depend on natural connections between events.

Analogy

Here the student is expect to deduce a relationship between two parts. I first came across these when I sat an intelligence test for graduate school in the US. (MAT)

FRYING PAN : (a. iron, b. ice, c. fire, d. pot) :: ROCK : HARD PLACE

The familiar phrase “between a rock and a hard place” describes a dilemma in which a person must decide between two equally unattractive choices. The familiar phrase “out of the frying pan and into the fire” is another metaphor involving two unappealing situations. So frying pan is analogous to rock, while fire is analogous to hard place. (Analogy type: association). The correct answer is (c)

Have I moved your opinion about MCQs yet?

Late entrant – Matching

Matching questions might be a fast assessment tool, even using the same possible five answers for each questions for example. 5 questions using the same 5 responses.

A Knowledge    B Comprehension    C   Application   D Analysis    E Evaluation

  1. Recall of information. Remembering facts, dates, names, events and places
  2. Put ideas, theories and concept into practice in new situations
  3. Interpreting information in own words as in making a summary or grasping meaning as in reading an article and being able to understand the content
  4. Assessing values of ideas and theory and make judgement or compare and discriminate between ideas
  5. Recognition of components and patterns and understanding their relationship

With this post, I have one extra reference to support the 5 millions already available. There is a relief.

MCQs_Google

qr code

Recall of information. Remembering facts, dates, names, events and places = Knowledge

Interpreting information in own words as in making a summary or grasping meaning as in reading an article and being able to understand the content = Comprehension

Put ideas, theories and concept into practice in new situations = Application

Recognition of components and patterns and understanding their relationship = Analysis

Assessing values of ideas and theory and make judgement or compare and discriminate between ideas = Evaluation

 

3 Responses

  1. […] outcomes in summative tests. I have used QuickKey for short progress checks but after reading this blog by @KristianStill I can see the value in multiple choice end of unit tests and using QuickKey […]

  2. Dylan Wiliam says:

    Actually, the probability of getting the question on Pythagoras’ theorem right by guesswork is only 1 in 15 (6×5/2) if you know there are exactly two correct answers. If you do not, then it’s 1 in 64 (2^6). And while I’m being picky, the only difference between MCQ-based tests and those that use constructed-response questions is the process that leads from the student’s response to the score for that item. Everything else including reliability and other aspects of validity are just as problematic for MCQ tests as they are for those that use constructed response questions.

  3. Agreed – incorrectly presented. Agreed, 1 in 15. If it is any reassurance it is presented correctly here. http://www.kristianstill.co.uk/wordpress/tried-and-tested/ and I will correct the paragraph above. I didn’t pick up on the fact that ‘not knowing’ how many correct answers are available decreased the chances of guessing correctly to 1 in 64.
    As for “being picky” – feel free. Until your comment I had not encountered the term “constructed-response questions” though there are significant scholarly references. Another line of investigation.

Leave a Reply

Your email address will not be published. Required fields are marked *