Multiple Choice vs. Free Response

Multiple Choice vs. Free Response

When we give an assessment, it is our mission to understand the student who takes it as best we can. Multiple choice tests usually look to understand student performance, but we want to know how students think.

We did an experiment where we had the same students take the same question 3 years in a row. The first two years were multiple choice, the third year was a free-form answer. This is what we learned!

The Question

In the first two years, the options were:

A. y = x – 2/3

B. y = (-2/3)x + 1

C. y = (-3/2)x + 1

D. y = x – 3/2

In 2019, over 7,000 students took it and 81% chose option B, the correct answer.

In 2020, over 6,000 students in the same schools took it, and 78% chose option B.

For the students who chose option B in 2019, just over 70% chose it again in 2020.

When we asked it for the third time, in 2021, we took away multiple choice and had students write an equation. What do you think the results were? How many students got the right answer?

I’ll give you a little time to think about it.

Over 7,000 students took it and just over 40% got it right. That is half of what we were seeing in the 2 years of multiple choice. For the students who chose Option B in 2020, ~57% got the right equation in 2021. What happened?

It’s actually quite fascinating. 14% didn’t even write an equation but something else.

slope(m)=rise/run=-2/3

Example Student Answer

-3, 1, 3

Example Student Answer

But then, we also realized just how much more information is included in a free form answer rather than multiple choice.

You can really start to understand the students better. Are they writing equations when prompted? What form is that equation in? What did they write for the slope? What did they write for the y-intercept? All of these questions give you a picture of what students understand and what are they still working through.

Let’s take a deeper look!

Below is a graph of the top 10 equations students wrote:

What do you notice?

We immediately noticed that students are writing equations with the correct intercept. They are all gravitating around that point. Let’s dive in.

Most Popular Intercepts# of StudentsOverallWith Equations
14,34161.18%82.94%
31662.34%3.17%
2821.16%1.57%
-1741.04%1.41%
-3480.68%0.92%
-2310.44%0.59%
1.5120.17%0.23%
5110.16%0.21%
4100.14%0.19%

Of the students who wrote an equation, ~83% of them got the right intercept. It is almost 90% for students who chose the right answer, Option B, in 2020. These students have a pretty good idea of how to find the intercept. If you go back to the multiple choice version, you’d see that students can get to a 50/50 chance, just from that.

What about the slopes?

Most Popular Slopes# of StudentsOverallWith Equations
-2/32,93341.33%56.04%
2/377210.88%14.75%
12553.59%4.87%
31892.66%3.61%
-3/21281.80%2.45%
-11251.76%2.39%
21111.56%2.12%
-3891.25%1.70%
1.5861.21%1.64%
-2540.76%1.03%
-0.5440.62%0.84%

Here we see the correct slope (-2/3), the absolute value of the correct slope (2/3), the inverse of the correct slope (-3/2), the absolute value of the inverse of the correct slope (3/2) and some other neat variations. 1 and -1 are just not considering slope at all.

The important takeaway here is that free-form response gives us insight into how students are thinking that is far beyond what multiple choice gives us. Depending on how students are thinking, we can actually re-engage with them and help instead of starting a lesson over from the beginning. We can engage with them as mathematical thinkers instead of struggling students. We can acknowledge their work and effort instead of just telling them they are wrong. We can change their experience of math education to be one of exploration and discovery instead of memorization and testing.

UPDATE: And now we have an update with students who took a free response 2 years in a row. I’ve been curious on how predictive free response would be over multiple choice. As mentioned at the top of the post, if a student got the multiple choice question correct, they had just above a 70% chance to get it right again. If a student got the multiple choice question correct, they had a 57% chance to get it right when answering without the multiple choice supports. And now, we finally got the predictability of students who got the free response right in 2021 and then again in 2022. The answer is 76%. So, not only does free response give us significantly more information than multiple choice, but is also more predictive of students being able to get the right answer again.

I’d love to hear your thoughts on asset-based approach, multiple choice assessments and how I can help. Contact me here or find us on twitter.

Or if you want to keep reading: Check out how we think about students writing functions.