Are your exam versions equally difficult?

Sooner or later a student asks: “Was Set 2 harder?” “They’re the same questions” is true, but it doesn’t fully answer the question, because the papers were not in the same order.

This guide explains what can make shuffled versions differ in difficulty, why comparing your class averages won’t tell you much, and what you can check instead.

What stays the same, and what doesn’t

When versions are made by shuffling one exam, every student answers the same questions with the same wording and the same options. Nothing is rewritten or swapped. That is a much safer position than writing two different papers and hoping they are equally hard.

What changes is the arrangement:

• which question comes first, and which comes last
• the order of the options inside each question
• where the page breaks fall

So any difference in difficulty has to come from the arrangement.

The two kinds of shuffling are not equally safe

Shuffling the order of questions has been studied a few times, and no effect on scores was found. Shuffling the options inside a question has less support. One study, as summarised by Sue (2009), found that changing the order of the answer options could change how hard a test was. Our evidence guide covers the research in more detail.

ExamShuffle shuffles both by default. Shuffling options is what stops a student copying the answer letters from their neighbour, so it is useful. But on an exam where small differences matter, it is the part to think about.

Why this matters to individual students

Students don’t choose their version. They get whichever paper matches their seat. If one version were slightly harder, that difference would come from the seating plan, and the student would have no way to know.

A class average also hides individual differences. A student two marks from a grade boundary cares about their own score, not the average.

None of this means shuffling is unfair. It means that “the versions are equally hard” is something you assume and protect, not something you have proved.

Why you can’t measure it with one class

It is tempting to compare the averages of each version after marking. It won’t tell you much.

The groups are too small. In Sue’s study, the two versions of one exam were taken by 73 students and differed by almost four marks on average. That gap was still too small to tell apart from chance. A class of 30 split into two versions has only 15 students per version. Any difference you see is mostly down to which students got which paper.

Versions follow seating. Papers are handed out by seat, and students choose where to sit. If friends who didn’t revise sat together, one version gets them all. You can’t separate the effect of the paper from the effect of the seating.

So the answer is in how you build the exam, not in the results.

What to check instead

These checks don’t need any statistics, and they catch differences that really happen.

Every version has the same questions and options. Count them. A version that lost an option while you were editing really is easier.

Ordered option lists stay in order. Dates, number ranges and steps in a process should keep their original order. Shuffled, they make the student sort the options before answering. In ExamShuffle, turn on Don’t shuffle these options for those questions. Writing questions that survive shuffling explains how to spot them.

Some questions will shuffle more than others. ExamShuffle keeps “All of the above” and similar options in place, and keeps a question’s options fixed when one names others by letter. So some questions look the same in every version and others change completely. That is correct, but it is worth knowing.

Layout. This is the difference you are most likely to actually have. Look at the printed versions side by side and check for:

• a question split across two pages
• one version running to an extra page
• a diagram on a different page from its question

These are easy to fix and they genuinely make one paper harder to work through.

The versions are actually different. A short exam can only be arranged in so many ways. If you ask for more versions than it can produce, some will be identical. ExamShuffle warns you when this will happen. The fix is to make fewer versions.

High-stakes exams

For a regular class test, shuffle both questions and options. The risk is small and it makes copying harder.

For a final, a placement test, or anything where a grade boundary will be questioned, shuffle the questions but keep the options in their original order. That is the combination with the most research behind it.

In ExamShuffle this setting is per question, not for the whole paper. On the review screen, turn on Don’t shuffle these options for each multiple-choice question. On a 40-question exam that means 40 switches, so decide before you start whether you need it.

Also, if you compare this year’s results with last year’s, use the same option order both years. Otherwise the averages may not be directly comparable.

Answering the student

When a student asks whether Set 2 was harder, you can give a clear answer: the questions were identical, only the order changed, the order was random rather than chosen to disadvantage anyone, and you checked the versions for layout problems before the exam.