Go Deep: How to Avoid Writing Superficial Reading Comprehension and Literary Analysis Items

Item writing is so complex–it is this interweaving of interpretation and analysis and extension of ideas beyond the text; it is the conjoining of skills and knowledge in order to measure students’ skills and knowledge.

interrelationshipsThere are so many ways to get it wrong. In writing items to measure reading comprehension, one of the most common roads to wrongness is writing superficial items.

In order to learn how to avoid writing superficial reading comprehension and literary analysis items, we have to consider why writers make this mistake. Superficial items are easier to write than items that require interpretation and analysis, but that doesn’t mean that item writers make this mistake because they’re lazy.

Writing superficial items is a symptom, not a disease: either the writer didn’t completely master the content of the text or that the writer lacks confidence in his or her interpretation. (There may be another root cause, but I don’t know what the heck it might be. Let me know if you do.) When the former is true, there will be a high proportion of superficial items along with a smattering of items that narrowly focus on one aspect or segment of the text–the one aspect or segment that the item writer did master. When the latter is true, the items will be more generally distributed across the passage, will stick to literal interpretations, and will probably rely on some formulaic writing.

Even highly experienced item writers may occasionally fall prey to one disease or the other. A passage may be particularly difficult. Passages that deal largely in abstraction are far more challenging for item writers. Literary passages that are more nuance than action are tough, too. This doesn’t mean these passages are unsuitable–on the contrary, these are exactly the passages that exhibit the text complexity required by the Common Core Standards.

Or we might be tired. Brains get overworked, too, and this work of item writing, if we do it right, is like being in grad school for a living. When we’re writing reading comprehension and literary analysis items, we’re interpreting and dissecting text all the livelong day, using needlenose tweezers to tease out fine distinctions in language usage, poring over the text with a magnifying glass: What did the author mean, what is the theme, what did this character feel, why did that character do that, how is this idea related to that, which words contribute to which tone, and every part of every item has to be grounded in or based on evidence from the text. What is the effect of using “harmony” in one paragraph and “disharmony” in the paragraph following? What is the effect of repeating the phrase “Yes, and” in the first two paragraphs of the passage? How does the author’s use of the first-person plural narrative point of view contribute to the reader’s understanding of the passage? Writing these items is no simple task.

How can we as item writers mine a text for all its rich density?

First, go for a walk.

first go for a walk

Then read and reread the text. Do this in a quiet place where you’re free of distractions. Don’t watch TV or try to carry on conversations as you’re reading. Focus. If your work allows, put the text away for a few hours or a day and do other tasks. You’ll probably think about the text–about the ideas or characters or theme–as you go on and do other things. Excellent. Your mind is doing the groundwork already. If you don’t have other work, proceed.

Read the text again. Five, six readings are not too many. It’s not onerous–we never encounter a text longer than 2000 words, and more commonly, a reading passage consists of fewer than a thousand words, words that the item writer should know backwards and forwards. Read like you’re going to take a high-stakes test yourself.

Next, take roll call by annotating the passage. This is how you find out what you’ve got to work with. Get out the pickax and dig, dig, dig. Document your finds. Mark up the theme, main idea, and/or the author’s argument. Number the claims or pieces of evidence for the author’s argument. Draw arrows to connect cause and effect. Circle above-grade-level vocabulary. Underline gorgeous figurative language or compelling rhetorical language. Note the intended effect of such language. Label appeals to ethos, logos, and pathos, along with examples of bandwagon, glittering generalities, flag-waving, and so on. Identify examples of irony, sarcasm, hyperbole, paradox. List metaphors, symbols, motifs, and other literary devices. Label flashbacks and foreshadowing.

If the text is nonfiction, answer these questions and more like them:

  • What is the author’s purpose?
  • What is the tone?
  • What does the reader learn from the diction?
  • What is the main idea or argument?
  • What facts, reasons, examples, etc. develop and support the main idea or argument?
  • How are ideas connected in the text?
  • What means of organization does the author use to develop the ideas in the text?

If the text is literary, write brief descriptions of the narrative elements. Answer these questions and more like them:

  • What is the topic?
  • What is the theme?
  • What is the setting?
  • What is the tone?
  • What is the mood?
  • What is the narrative point of view?
  • Who is the protagonist?
  • What are the stakes–what does the protagonist want?
  • What is the inner conflict?
  • What forces oppose the protagonist?
  • How is the protagonist transformed?

That latter may be the most important question–without transformation or insight or epiphany, without some kind of change or at least a stepping stone to change or even a refusal of an opportunity, there is no story. If you don’t get the transformation (or insight or epiphany or what have you), you can’t write meaningful items about the text. Conflict is big, and theme is even bigger. All three of these should be present and accounted for in a literary text.

These aren’t the only questions. Consider how the author uses language, syntax, punctuation, and even paragraphing. Think about how the characters are developed–does the author use dialogue or are characters revealed through interior monologue or their actions and reactions? And then how are all the narrative elements interrelated? How does the setting support the theme or contribute to the mood? How does the plot develop the theme? It’s endless, really.

For every answer, we need to identify specific text evidence.

Once you’ve thoroughly annotated the text AND have answered all of the questions AND have identified specific text evidence for each answer, then you know this text. You might know it better even than the writer does. It’s true that good readers may understand the text better than the writer–so much of writing happens in the subconscious depths, and writers aren’t always of aware of the effects of what they do (in fact, thinking too much about the effects of what you’re writing can be crippling to the writer). Now you know the text and you’re ready to write the items.

This is what real item writing looks like. This kind of preparation takes time–it may take three hours. Trust me, it’s worth it. The item writing will not only be easier and faster–you’ll make up the time, even if you’re paid by the item (if you’re paid a decent rate, which is not uniformly the case, unfortunately, but what you should do is count this cost in determining whether to accept or decline assignments)– but the items will be of a higher quality, will have some of the depth and richness of the passage, may even support and enhance classroom instruction, and may achieve what we all aspire to: the elegance of truth.

What we’re doing is deciphering the truth of the text, and then we’re turning around to ask the student to do likewise.

Eventually, if you do this enough, all of this becomes habit. By “eventually,” what I mean is “after years and years and years of practice.” Years, not weeks or months. Years and years and years, not one or two years. I annotate passages only occasionally now, when I hit a wall in item writing–but I have so much practice in annotating text that I can usually do mental annotations. Literature was my major field of study in college and writing in grad school; I’ve been writing all these years, and specifically writing passages and items for much of two decades; and a great deal of my work is evaluating text (determining the suitability of the passages for the purpose) and editing text. I’m in a writers’ group in which we closely examine each other’s pages down to the effect of a paragraph break or that of an intentionally placed sentence fragment. And I read. A lot.

read a lotLast week, I read Cormac McCarthy’s The Road, All the Pretty Horses (I’m reading all his novels because there is such value in reading all of an author’s work–you inhabit the author’s world and way of thinking about the world, it’s remarkable to follow a writer on this journey–Helen Oyeyemi was my last writer project), and Debbie Stier’s The Perfect Score Project (a single mother of teenagers spends a year studying for and making multiple attempts at the SAT in an effort to motivate her own child), and one other book, I forget what it is but it will come to me. A novel. Where is my brain, I ask you.

That latter moment of forgetfulness convey the impression that I read with inattention, but I don’t. When I read, I don’t skim. I burrow in. When I’m done, I think about what I read while I’m walking the dog or driving or cooking or otherwise engaged in the drudgery of daily life: about the themes, the characters and the choices they make and how they are influenced by events and other characters, about the writing and how it affects me.

I do what I can (we’re human, we have an innate desire to be the Great Decider of Everything) to avoid an evaluative perspective of good/not good, love it/hate it–evaluation shuts down thinking; the evaluator gets to be the judge and makes a ruling and then closes the case, which is antithetical to what we’re trying to do.  We want to unpack, to open everything up, we don’t want to slam the suitcase shut and walk away. Evaluation has its place, but its place is not in the literary analysis that we do. Better to use it in questions of which car to buy or what’s the best time to leave to avoid traffic or which are better, the white peaches or the yellow ones at the farmers market.

Instead, I think about what I read in terms of how I am affected as a reader: what do I find poignant or shocking or melancholy or uplifting or funny and why. What is transcendent. This kind of reading will definitely help anyone become a better item writer and a better writer of all manner of text; reading that is purely consuming the entertainment will not help at all and may even hamper you, because of habits. If you’re in the habit of reading leisurely through a text, finishing, and tossing it on the coffee table with a “Next!”, it will be so difficult to harness your mental faculties to do the focused literary analysis that is necessary for item writing. We have to build those muscles. (Which is not to say that I never engage in the consuming entertainment kind of reading–I do, and it’s fun, and it’s the only way to read books that may be fun to read for a particular reason–mysteries, for example–but this kind of reading is like beer and potato chips for dinner: a treat once in a while, not so beneficial when it’s a daily event.)

This all sounds like so much work, doesn’t it? But what great thing was ever achieved without effort? Our wits don’t just get sharper on their own when they’re sprawled on the couch all day watching cartoons.

All right, so here are some resources that will help us train our minds in this task and others–these are also useful for writing in general, whether the writing be the spontaneous outpourings of one’s soul, or mercenary passage writing, or what have you:

These lists of emotions allow us to be precise when discussing how characters feel–enough of the happy/sad/angry/worried, is what I’m saying:

Learning about human nature (about what humans want and need, what motivates us, what inhibits us) is an excellent  tool for improving how we analyze the motivations and actions of characters (and it allows for the most excellent parlor game of Diagnose Everyone You Know):

Item writers should also be sure to refer frequently to these kinds of references to stay on course with their interpretations and analyses:

And here are more:

If you’re an item writer, you will improve your life and your items by bookmarking this page or by copying these links and bookmarking each one and then calling upon them as the opportunities arise. No writer should dismiss these kinds of tools. When I encounter a writer who, when offered such tools, implies that he or she is in no need of them, I worry. I worry because I know from my long and intimate acquaintance with this work that no matter how much one knows, one may still find it so easy to unintentionally take one of many wrong courses. Why anyone would insist on throwing away the maps and instead bumble through the thorny brambles with no assistance is beyond me.

Really, the best way to approach this work is the best way to approach any worthwhile endeavor: with humility and respect, with the willingness to be wrong, with the ability to be open to discovering–either on one’s own or through training and coaching, if one be so lucky–what is right, and steadfastly adhering to the goal of writing excellent items that will do what we intend for them to do: provide students with an opportunity to show us what they know or can do.



Email UpdatesWIMJ usually has updates on a bi-weekly schedule. If you like this post, sign up. I'll write more.


  1. […] Consider this Part II of how to avoid writing superficial items. […]

  2. […] in testing is a nuanced concept—it can refer to testing systems, testing practices, or test questions. A thoughtful critique requires time, attention, and a fundamental understanding of […]

Speak Your Mind