Common Mistakes in Item Writing: When the Shoes Don’t Match the Outfit

Shoes make the outfit. But I think we can all agree that you’ve got to wear shoes that work with the clothes. You can’t wear saucy red patent leather espadrillesred shoes   red shoes with pants–with serviceable but baggy and, let’s be honest, nondescript sweatpants. (Unless you’re someone whose name is Not Me. I have to admit that Naomi–the model, my youngest-by-five-minutes daughter–doesn’t look half bad in the espadrilles-with-sweatpants combination, thus defeating my purpose in using this analogy. But that girl looks great in anything.)

Some things go together like pizza and Zinfandel. Some things don’t. (PSA: don’t drink red wine of any varietal with halibut, snapper, tilapia, sea bass, or any white fish. Trust me on that. Pinot Noir and sockeye salmon is nice.)

It’s the same dang thing in item writing. When you write multiple-choice test questions, the answer choices have to be compatible with the stem. This may seem like a self-evident proposition, but item writers often write stems with answer choices that confuse content with process or purpose. This means that the item writer might ask a question that relates to content, but then the item writer provides answer choices that relate to process or purpose. When you think about how all the parts of the item fit together, you’re less likely to make this mistake. That is, you can’t ask a “What?” question and offer only “How?” answers.

Here is an item that confuses purpose with content:

What is the main idea of the speech?
A     to discuss the history of voting
B     to persuade the audience to vote
C     to explain the requirements for voters
D     to entertain the audience with tales about voting

Do you see the problem? We’re asking the student to identify the main idea, but we’re offering answer choices that describe possible purposes of the speaker in giving the speech. This means that there is no correct response. It’s impossible that “to persuade the audience to vote” could be a main idea of a text.

The fix is to replace the answer choices with four answer choices that could conceivably relate to main ideas. We can use this formula:
A     [the actual main idea] correct response
B     [a wrong interpretation of the main idea] distractor
C     [a supporting detail from the text] distractor
D     [the main idea of a paragraph from the text] distractor

Other possible distractors (wrong answers) could be an overly general or overly specific statement of the main idea, another supporting detail from the text, or a general statement of the topic (which is not the same as the main idea). Our new item and improved item looks like this:

What is the main idea of the speech?
A     Only educated citizens should be allowed to vote.
B     Eligible citizens should exercise their right to vote.*
C     Many governments pass laws to make voting mandatory.
D     Some candidates are elected by only a minority of eligible voters.

In this version, the stem and the answer choices match; both relate to the content of the text–main idea–and not to its purpose–how the author of the text achieves an effect or produces a result. And the best part is that we don’t have to waste our earlier work. We can use the old answer choices in a perfectly acceptable new item that addresses purpose:

What is the president’s purpose in writing the speech?
A     to discuss the history of voting
B     to persuade the audience to vote*
C     to explain the requirements for voters
D     to entertain the audience with jokes about voting

Not only did we repair a flawed item, but we used the raw material to write another item. Yay, us. I hate waste. I encourage you to salvage what you can, when you can. I repeat: when you can. It’s not always possible nor desirable. You have to learn what’s worth keeping. Be ruthless about discarding what doesn’t work. Don’t hoard. In item writing; if in the privacy of your home you want to stack up columns of newspapers to the ceiling and fill thousands of baskets with beanie babies and put them in your driveway and cover them with a tarp, go right ahead.

Back to how to salvage our bad item. In this case, the mistake is simply a mismatch between stem and answer choices, so we can use the material from the mistake to build two solid items: one that addresses main idea and one that addresses author’s purpose.

Let’s look at what happens when the item writer writes an item that confuses process with content. By “process,” I mean how the author of a text achieves an effect or develops an idea or somehow uses specific techniques for an identifiable purpose. Here is an item that confuses process with content:

How does the president develop the idea that voting is a privilege?
A     Literacy should not be used as a requirement for voting.
B     In some countries, citizens are excluded from voting in elections.
C     Americans are not allowed to vote until they are eighteen years old.
D     In the United States, different groups of people have fought for the right to vote.

Again, we have an item with no correct response. We can’t answer a question that asks “How?” and then offer answer choices that are ideas. The question “How?” must be answered with words that show process or action. If we ask about process, we have to provide answers that describe processes. Again, there are two easy fixes: we can either change the stem to match the answer choices, or we can change the answer choices to match the stem.

Here we change the answer choices:

How does the president develop the idea that voting is a privilege?
A     by offering a personal anecdote about dedicated voters
B     by repeating the words from legislation that concerns voting
C     by describing the method used to determine voting eligibility
D     by providing examples of people who are not allowed to vote*

Be careful with this format. Use it judiciously. Be sure that you integrate the process of each answer choice (by offering, by repeating, by describing, by providing) with the content (the second half of the answer choices). If there’s not a good fit–if the choice of the verb isn’t logical with what follows–the item ends up looking like a misbegotten, cobbled-together Frankenstein. Don’t wear out the thesaurus on this one. If your answer choices are all synonyms, then put that verb in the stem, e.g., “What information does the president include to support the idea that voting is a privilege?” Then you eliminate the redundancy and your answer choices are all noun phrases.

Don’t do this:

How does the president develop the idea that voting is a privilege?
A     by showing beliefs held by dedicated voters
B     by demonstrating knowledge of legislation
C     by conveying passion for supporting government
D     by listing methods for voters to become informed

Do this:

What does the president demonstrate in order to develop the idea that voting is a privilege?
A     knowledge of legislation
B     beliefs held by dedicated voters
C     passion for supporting government
D     methods for voters to become informed

The rule to remember is that every word in the item should serve a specific purpose. If the words are just clutter—and particularly if they are interchangeable with any other words in the item—clean up. Get rid of any unnecessary words. Clarity is the goal to which we constantly aspire, and minimalism is our means of achieving that goal. Clarity will help us avoid doing harm to the test-taker.

The other solution to our problem above is to change the stem. Instead of asking how the president develops that particular idea, we can ask any of these questions:

  • What evidence the president use to develop the idea that voting is a privilege?
  • Which fact does the president use to support the idea that voting a privilege?
  •  According to the president, why should voters feel that voting is a privilege?

You can take that in various directions, customizing the stem depending on the standard you wish to target and how you would like to address that standard. Here’s one possibility:

Which detail does the president provide as evidence to support the assertion that voting is a privilege?
A     Literacy is not used as a requirement for voting.
B     In some countries, citizens are excluded from voting in elections.
C     Americans are not allowed to vote until they are eighteen years old.
D    In the United States, different groups of people have fought for the right to vote.

red shoes with dressIf every item writer read and followed these guidelines, we could eliminate one whole category of bad items. Here’s to making the world a better place, one outfit item at a time.

Email UpdatesWIMJ usually has updates on a bi-weekly schedule. If you like this post, sign up. I'll write more.

Speak Your Mind

*