Question order effects: what changes when the sequence changes

Moving a question from position 30 to position 3 feels like it changes nothing. The wording is the same and the marks are the same. Only the number beside it is different.

That is probably close to true. But it helps to know what has actually been tested, what is only a reasonable guess, and what you can check in your own exam. This guide covers all three.

What has been tested

A few studies have compared exams given in their original order with the same exams given in a scrambled order. Our evidence guide covers them in full. In short:

• Sue (2009) gave students in two economics courses either an ordered or a scrambled version across three exams. Some scrambled versions averaged a little lower, but none of the differences were large enough to rule out chance.
• Bresnock, Graves and White (1989), as summarised by Sue, found that question order had no effect on test performance.
• Sue (2006), also as summarised in the 2009 paper, found that scrambling question order did not affect grades.

So: three studies, all of university economics students, all multiple-choice, all comparing total scores between two versions, and none found an effect.

That is useful to know. It is also narrower than “research shows question order doesn’t matter,” which is how it is often repeated.

What “no effect found” does and doesn’t mean

Small effects can be missed. These studies had fewer than a hundred students per exam. In Sue’s study, two versions of one exam differed by almost four marks on average, and that gap was still too small to separate from chance. A smaller effect would not show up at all.

Total scores can hide changes. The studies compared whole-paper totals. If moving questions made some students do better and others do worse, those changes could cancel out in the average. The studies can’t tell you whether that happened.

So the fair summary is: “No study we know of has found that question order affects average scores.” That is not the same as “Question order has been shown not to matter.”

Common explanations that haven’t been tested

You may hear these reasons why question order could matter. They sound sensible, but none of the studies above tested them. Treat them as ideas, not facts.

Tiredness. A question at the end of the exam is answered by a more tired student with less time left. Shuffling moves questions between the start and the end. But on a fixed set of questions, every question moved later pushes another one earlier, so the effect on the total score may mostly cancel out, even if individual questions are affected.

Warm-up questions. Starting with an easy question may help students settle in, and starting with a hard one may cost them time and confidence. Shuffled papers start with a random question, so some versions will start harder than others. This has not been tested in any of the research cited here.

Related questions next to each other. A question that follows a related one may be easier because the topic is fresh. Or it may be harder, because the earlier question put the student in the wrong frame of mind. Both are arguable, and neither has been measured here.

The order problem you can check for

One order effect is not a theory. You can check for it by reading your own paper.

Sometimes one question gives away the answer to another. For example:

• Question 12 mentions a term that question 30 asks students to define.
• Question 5 includes a diagram that helps answer question 18.
• One question’s stem contains the correct option for another question.

A student who meets the “helper” question first has an advantage over one who meets it later. Shuffling decides which student is which.

This is different from questions that refer to each other directly, like “Using your answer to question 4.” Those are covered in writing questions that survive shuffling. Here, nothing refers to anything. One question simply contains information that another one tests.

No software can reliably find these for you. Here is how to check:

1. Make one shuffled version.
2. Read it from start to finish, the way a student would.
3. For each question, ask: “Did an earlier question help me answer this?”
4. If so, rewrite one of the two questions, or put both in the same section so they always appear together.

This takes about ten minutes and doesn’t depend on any research being right.

Where the evidence is weakest

• Answer options. Shuffling the options inside a question is a separate thing from shuffling the questions. One study (as summarised by Sue) suggests it can change difficulty. ExamShuffle shuffles both by default, and each question has a Don’t shuffle these options switch if you want to keep its options in place. Are your exam versions equally difficult? covers this in more detail.
• Who was studied. University economics students only. Nothing here covers school pupils, essays or short-answer exams.
• Single questions. All the studies compare whole papers. None tells you what happens to one particular question when it moves.

These are the gaps in the sources cited on this site. There may be other research we haven’t covered. For a high-stakes decision, it is worth reading further.

What to take away

• Shuffling question order is reasonably safe. It has been tested a few times, and no study we know of found an effect on scores.
• That is not proof it never matters.
• The order problem most worth your time is one you can check yourself: questions that help answer other questions. Read one shuffled version through before the exam.