Claire Longmoor didn’t expect her math problem to go viral.
“An orchestra of 120 players takes 40 minutes to play Beethoven’s 9th symphony,” the question read. “How long would it take for 60 players to play the symphony?”
As with so many puzzles that find their way to the internet, the responses were radically split and mostly wrong: One group of people, who were perhaps reading too fast, confidently declared the answer to be 20 minutes. The second camp reasoned that half as many musicians would have to work twice as hard, so the answer must be 80 minutes.
Yet a third group was stupefied, questioning the teacher’s ability to write good questions. “Think the person who came up with that question really doesn’t know how an orchestra works!” the Wexford Sinfonia Orchestra tweeted.
It’s a trick question, Longmoor admitted one designed to keep her students on their toes, echoing a common sentiment among test makers that such questions force students to read carefully, ensuring that they attend to substantive questions later on. But do trick questions actually work as intended?
Andrew Butler, a professor of psychological and brain sciences at Washington University, doesn’t think so. Trick questions are not “productive for learning” and can easily backfire, he says. The result: confused students, artificially reduced test performance, and a murkier picture of what students actually know.
Other research on test design suggests that all too often, we’re not just assessing what students know, but also getting a peek into the psychological and cognitive eddies that disrupt a student’s thinking—a high-stakes test that causes anxiety can become a barometer of a student’s poise, rather than their knowledge. A well-designed test is rigorous and keeps implicit bias in check while being mindful of the role that confidence, mindset, and anxiety play in test-taking. Here are eight tips to create effective tests, based on a review of more than a dozen recent studies.
1. HELP STUDENTS DEVELOP GOOD TEST PREP HABITS
Students often overestimate how prepared they are for an upcoming test, which can result in unexpected low performance, according to a 2017 study. Consider asking students to make and show you a study plan involving productive study strategies like self-quizzing, teaching the major concepts to peers, or spacing out their studying into multiple sessions instead of cramming the night before.
To help address test anxiety, researchers recommend setting aside a little time for simple writing or self-talk exercises before the test—they allow students to shore up their confidence, recall their test-taking strategies, and put the exam into perspective. In a 2019 study, for example, elementary students who spent a few minutes before a test and “silently spoke words of encouragement to themselves that were focused on effort” saw their math scores rise. And in a 2019 study of ninth graders, researchers found that a simple 10-minute expressive writing activity that reframed test anxiety as “a beneficial and energizing force” led to course failure rates being cut in half for vulnerable students.
2. FIND THE SWEET SPOT FOR RIGOR
Design tests so they’re at an appropriate level of difficulty for your students. Overly difficult tests sap students’ motivation and increase the likelihood that students will remember the wrong answers, according to a 2018 study.
In the end, “tests that are extremely easy or difficult are essentially useless for both assessment and learning,” the study concludes. Students who study moderately should get roughly 70 to 80 percent of the questions correct.
3. BUT START WITH EASY QUESTIONS
Don’t start a test with challenging questions; let students ease into a test. Asking difficult questions to probe for deep knowledge is important, but remember that confidence and mindset can dramatically affect outcomes—and therefore muddy the waters of your assessment.
A 2021 study found that students were more likely to do worse on a test if difficult questions were at the beginning instead of nearer to the middle or the end of the test. “Students might be disheartened by seeing a hard question early in the test, as a signal of the general difficulty of the rest of the test,” the researchers explain.
4. BE AWARE OF IMPLICIT BIAS
Question format matters. In a 2018 study, researchers analyzed test scores for 8 million students and discovered that boys tend to outperform girls on multiple-choice questions, accounting for roughly 25 percent of the gender achievement gap. Girls performed significantly better than boys on open-ended questions. Consider the mix of your testing formats: Combine traditional testing formats—multiple choice, short answer, and essay questions—with creative, open-ended assessments that can elicit different strengths and interests.
Be mindful, as well, of how cultural or racial bias and background knowledge can infiltrate the language and framing of test questions. In an infamous example, an SAT analogy question required students to select “oarsman:regatta” in response to the word pair “runner: marathon,” an expectation that was fraught with classist, racial, and geographic overtones.
Other studies reveal that without a threshold of background knowledge, students fail to grasp the intent of their reading—an incorrect answer on a test may signify the failure to determine the meaning of the question, rather than measure the student’s understanding of the material. Keep test questions free of unnecessary jargon, revise tests to simplify questions, and consider allowing students to ask for clarification before you start the test.
