New York State Item Review Criteria for Grade 3-8 English Language Arts Tests

Children reading

With the transition to the New York P12 Common Core State Standards (CCSS) significant changes have been made to the New York State Testing Program (NYSTP).  The 2013 Grade 3-8 Common Core English Language Arts (ELA) Tests will measure the Grade 3-8 CCSS for ELA & Literacy, and as such the tests will approach reading, writing, and language differently from past assessments. To consistently measure the CCSS for ELA with appropriate rigor, the State has developed detailed resources that may also be useful in the classroom.

The CCSS for ELA clearly require an increased depth in analysis of text.  Most standards require that students move well beyond straightforward comprehension and move towards insightful analysis of various aspects of complex texts.  To guide item development and to help ensure that NYS tests are measuring the CCSS for ELA with fidelity, the New York State Education Department (NYSED) has developed an Item Review Criteria.

Item Review Criteria

The Item Review Criteria is a framework used by assessment developers and reviewers to help ensure that NYSTP ELA tests have high-quality questions that measure the CCSS for ELA.  The Item Review Criteria articulate the four major item characteristics required for quality questions.  NYSED uses the Item Review Criteria to help ensure that each item:

  1. is clear;
  2. is fair;
  3. measures a specific Common Core standard (or standards) with fidelity; and
  4. conforms to the specifications for the item type.

The Item Review Criteria is organized by these four categories. Each section of the criteria includes pertinent questions that reviewers use to determine whether or not an item is of sufficient quality so that it can move forward in the development process. The first two sections of the Item Review Criteria, Clarity and Fairness, identify the basic components of quality questions. The criteria for Clarity are used to help ensure that students understand what is asked in each question and that the language choice in the question does not negatively impact a student’s ability to perform the required task.  Likewise, the Fairness criteria are used to ensure questions are un-biased, non-offensive, and not disadvantageous to any given subgroup.

The next section of the Item Review Criteria addresses how each item measures a given standard or standards.  The CCSS are complex and multi-faceted statements that require both analysis and prioritization.  The Item Review Criteria articulates the aspects of each standard that the questions need to address.  Questions that are designed to successfully focus on these key aspects are deemed to measure the intended standard.  By specifying the parts of each standard on which questions should be focused, item developers and reviewers are provided a clear and shared direction for item development.  While specifications for all standards are provided, reviewers only evaluate a given item against the standard that it was intended to measure.

The final Item Review Criteria section addresses the specific demands for different item types (i.e., multiple-choice, short-response and extended-response questions).  Reviewers evaluate each question to ensure that it conforms to the given requirements.  For example, multiple-choice questions must have, among other characteristics, one unambiguous key and text-based, plausible distracters.

How the Item Review Criteria is Used

Using these criteria to review draft questions is one of many steps employed to help ensure a valid, fair, and quality assessment. Draft questions that meet these criteria are allowed to move forward in the development process. The next step is for the questions to be reviewed, and when necessary, edited, by a panel of certified New York State educators [1].  The educator panel evaluates each question for fairness, valid measurement of the intended standard(s), and language clarity.  Only questions that are approved by the educator panel are allowed to be field-tested.

[1] To volunteer to participate in NYSED Educator Item Reviews: http://www.p12.nysed.gov/assessment/teacher/home.html#teacher-op

How Educators Can Use the Item Review Criteria

While the Item Review Criteria was created specifically for evaluating state test questions, the criteria can also be used by educators in daily practice and long-term planning.  The Item Review Criteria can be used in two ways:

  1. as a resource for educator created/administered tests
  2. as a shared, consistent understanding of the standards between educators and the State.

The Item Review Criteria helps to guide development of good test questions.  Item review criteria can be used by teachers to guide curricular and formative assessment development.  The first two criteria (Clarity and Fairness) attend to concerns that all quality test questions need to address.

The most important aspect of the criteria for educators, however, is the section that begins on page 2 and ends on page 15.  These criteria contain valuable information regarding NYSED’s approach to measuring each grade-level standard as well as important non-negotiable elements to assess student mastery of the Common Core. Among other things, these criteria:

  • illuminate the depth of analysis called for by each standard
  • show the variety of ways students can demonstrate their ability to marshal textual evidence in their writing
  • give valuable insight and a protocol for educators creating their own CCSS-aligned assessment questions (or assignments).
  • build a framework for unpacking the standards with teachers within and across grades levels
  • signal to leaders the depth of analysis that administrators should look for in classrooms
  • offer guidance on the depth of analysis to look for in text books and other materials that claim alignment to the Common Core

A Use-Case Example:

A 4th grade ELA teacher wants to dig deep into the analysis required by R.L.5, he/she could use the Item Criteria alongside the Passage Protocol to ensure the questions asked in class move beyond surface-level identification of structural elements in a text and "determine a theme of a story, drama, or poem from details in the text, including how characters in a story or drama respond to challenges".   By studying the standard in this manner, this teacher may make important pacing decisions, such as choosing to do a close read of a section of a story, drama or poem in order to tease-out the analysis demanded by this standard.

Teachers can use the Item Review Criteria to design or modify their instruction and/or assessments to ensure calibration with the approach NYSED is taking to the measuring the CCSS.

Downloadable Resources

Resources may contain links to sites external to the EngageNY.org website. These sites may not be within the jurisdiction of NYSED and in such cases NYSED is not responsible for its content.