What endeavor doesn’t benefit from planning and preparation? What endeavor succeeds without preparation?
These first guidelines of the CCSSO/TILSA Quality Control Checklist for Item Development and Test Form Construction should be considered in early stages of planning, long before item writing assignments are made:
A test blueprint identifies the skills and/or knowledge to be assessed, provides the item-to-skill distribution, and specifies item formats.
Let’s think about creating a blueprint to assess writing at grade 6. We’ll base the blueprint on the Common Core State Standards.
In the CCSS, English conventions are addressed in the language standards, and what we might call writing strategies and application are addressed in the writing standards. The language standards could be assessed with a variety of formats: standalone or passage-dependent multiple choice items, standalone or passage-dependent technology-enhanced items, or as one component of an extended-constructed-response item.
Here is a writing standard:
1. Write arguments to support claims in an analysis of substantive topics or texts,using valid reasoning and relevant and sufficient evidence.a. Introduce precise claim(s), distinguish the claim(s) from alternate oropposing claims, and create an organization that establishes clearrelationships among claim(s), counterclaims, reasons, and evidence.
b. Develop claim(s) and counterclaims fairly, supplying evidence for eachwhile pointing out the strengths and limitations of both in a manner thatanticipates the audience’s knowledge level and concerns.
c. Use words, phrases, and clauses to link the major sections of the text,create cohesion, and clarify the relationships between claim(s) and reasons,between reasons and evidence, and between claim(s) and counterclaims.
Generally the above standard would be assessed with an extended-constructed-response item, because multiple-choice items and short constructed-response items don’t allow students sufficient opportunity to demonstrate the ability to “write arguments to support claims….” However, the subskills may be (and frequently are) assessed with multiple-choice items; this is more common at the district or classroom level than at the state level. You might see a question that addresses W.1.a by asking the student to choose the best opposing claim for a given argument. Such multiple-choice items may help teachers isolate specific areas in which a student needs instruction and support.
Here are language standards:
In a typical statewide high-stakes assessment program, the decisions that inform the development of a test blueprint and item specifications are made by committees, which is as it should be, and committees should include classroom teachers. Committees often include other stakeholders, e.g., business leaders who may be asked to identify skills and knowledge necessary in the workplace.
Once all of that preparation is complete, item development begins.
Now let’s say we’ve received an assignment to write those multiple-choice language items and that ECR writing prompt. We’ve read all of the project documentation and support materials; we have the item specifications in front of us.
It is a truth universally acknowledged that a test item should target one and only one skill or bit of content knowledge. Each idea should have one big idea; every part of the item should support that focus.
If we were going to write a multiple-choice item for W.2.b, our big idea would be how to spell grade-level appropriate words. We might write an item that looks like this:
Which word is spelled correctly?
This item clearly targets one skill: correctly spell grade-level-appropriate words. The stem tells the student exactly what to do. The item is phrased simply and concisely. The content is neutral; there are no highly-charged words. All of the answer choices are grade 6 words (according to EDL Core Vocabularies); all are words likely to be known to grade 6 students and are words that are significant to academic content areas. There are no tricky or esoteric rare words. The answer choices appear in a logical order (here we use alpha order). All of the distractors address common spelling mistakes: using s instead of c, using e instead of a, and writing phonetically. None of the words are homonyms and so none are context-dependent; each of these words have one correct spelling.
Here is a poor item addressing the same skill:
Which word is written correctly?
This item has multiple flaws. First, the big idea is not specified in the stem; the student doesn’t know what s/he is expected to do until s/he reads the answer choices. The answer choices are not grade-level-appropriate; “machine” is a grade 2 word, while “anchor” is grade 3. The word “rabbi” may not be familiar to grade 6 students. Answer choice A (“musheenz”) is plural, while the other ACs are singular. Answer choice A also offers mistakes that are unlikely to be made by students at the targeted grade level. The answer choices do not appear in any logical order. Finally, the correct response is a type of weapon.
As bad as this item is, though, we could make it even worse by
- increasing the reading load by burying the spelling words in sentences and offering four sentences as the answer choices;
- obscuring the targeted skill by adding in other types of conventions errors, such as mistakes in capitalization and punctuation;
- using homonyms, or words that are spelled differently depending on the context;
- using above-grade-level vocabulary.