1. Introduction

1.1 Criterion-referenced testing

Glaser (1963) distinguished between two fundamentally different types of educational test results: norm-referenced scores and criterion-referenced scores, requiring that a criterion-referenced measure should provide “explicit information as to what the individual can or cannot do”, “which is independent of reference to the performance of others”. Six years later, Popham and Husek (1969) noticed that criterion-referenced measurement had “been the subject of many discussions over the years”, but also that “other than adding new terms to the technical lexicon, the two constructs have made little difference in measurement practice”. 

Correspondence should be made to Svend Kreiner, Section of Biostatistics, Department of Public Health, University of Copenhagen, Øster Farigmagsgade 5, B, 2099, Copenhagen K, Denmark. Email: svend.kreiner@mail.tele.dk

© 2026 The Authors .This is an open access article under the CC BY 4.0 license (https://creativecommons.org/licenses/by/4.0/)

The 1963 paper by Glaser and the 1969 paper by Popham and Husak were reprinted together with a collection of symposium papers edited by Popham (1971) in the hope that it would encourage colleagues to “tackle the tricky technical problems facing us all” (Popham 1971, preface). Despite these efforts and after developing tests for more than a decade[1], Popham (2014) refers to “half a century wasted” for several reasons, including “slapping the label criterion-referenced on tests rather than on test-based interpretations”. 

To Popham and Husek (1969), the fundamental difference between the two types of tests lies in the definition of items. Today, it is not relevant to distinguish between norm-referenced and criterion- referenced testing. Measurement by educational tests is quantitative measurement supported by IRT or Rasch models, and Wright (1980, p. 196), discussing probabilistic models for attainment tests, insisted that interpretation of test scores from Rasch models should do the same as Glaser required of criterion-referenced tests by providing information on students’ “strengths and weaknesses”.  In other words, interpretation should be criterion-referenced. 

1.2 Interpretability

De Vet et al. (2011) define interpretability as “the degree to which one can assign qualitative meaning – that is, clinical or commonly understood connotations – to an instrument’s quantitative score” (p. 228). The definition applies to measurement supported by both classical and contemporary test theory. Hambleton and Zenisky (2013) describe two ways to interpret test scores supported by conventional IRT models, both of which are important for our paper: interpretation by item mapping and interpretation by scale anchoring.

Interpretation by item mapping extends the interpretation by Wright maps and ICC curves illustrated by Wright and Masters (1982). Wright maps provide information on the locations of dichotomous items and Thurstonian thresholds of polytomous items, and interpretation focuses on whether events defined by responses to items are smaller or larger than 50 %. This information is useful for interpretation, but it is not enough to distinguish between what students can and cannot do. To address this issue, Hambleton and Zenisky (2013) replace the 50% probability criterion with larger probabilities, and assess response probabilities at a small set of so-called benchmarks on the measurement scale.

Interpretation by scale anchoring generates estimates of probabilities and expected frequencies of the responses to items at the selected benchmarks. Forsyth (1991) and Beaton and Allen (1992) describe two ways to calculate scale-anchored probabilities and frequencies. The direct method uses data counting the responses to items for persons close to the benchmark values, while the smoothing method uses the estimates of the item parameters of an IRT model to calculate probabilities given the person parameter associated with the benchmark. In large-sample studies, the two estimates will be close to equivalent, if the IRT model actually fits the data.

The benchmarks of Hambleton and Zenisky and of Beaton and Allen do not depend on subject matter theory and do not refer to the criteria against which to interpret test scores. They are either norm-referenced benchmarks or equidistant benchmarks of convenience.

1.3 Applications of educational tests.

Popham (2009) distinguishes between formative and accountability testing. Accountability testing is required when somebody “in a position of authority doubts whether a particular educational program is doing a good job” (Popham, 2009, p. 96). The focus in accountability studies is always on quantitative measurement and differences that statistical analyses can disclose. Test results collected during accountability studies are not meant to be useful to teachers in classroom applications, but interpretation of test results may be essential when those responsible for accountability studies need to share their results in meaningful terms with politicians, civil servants, and the public.

PIRLS (Progress in International Reading Literacy Study) is an international accountability study of functional literacy among fourth-grade students. PIRLS^[2] recognizes that policy and curriculum improvements require more information than what statistical analyses of quantitative test results are able to make available. For this reason, PIRLS partitions the range of test scores into a number of ordinal proficiency categories that they interpret through anchor scaling as defined by Beaton and Allen (1992). PIRLS does not claim that the proficiency categories are criterion-referenced, and Bruggink et al. (2022) propose applications of PIRLS in classrooms that do not involve PIRLS’s proficiency categories. We therefore take it that they do not consider the interpretation inherent in the classification to be useful to teachers.

Popham states that the purpose of formative testing is to “elicit evidence so that teachers can make accurate inferences about their students’ unseen knowledge and skills” and “adjust their ongoing instructional procedures”, and Wright (1980) reminds us that “the only justification for testing under these circumstances is the intention to use test results to help test takers” by providing information on students’ “strengths and weaknesses” (p. 196).

In other words, interpretation of results from formative tests may be helpful in the same way as criterion-referenced testing, but it is the interpretation that has to be criterion-referenced. To address this issue, we will do what Haertel (1985) claims that many others have done.  We will illustrate how to “press” quantitative test results from PIRLS 2016 “into service to meet the demands of criterion-referenced testing” to make them useful for formative classroom testing.

1.5 Criterion-referenced interpretation

The criterion-referenced interpretation that this paper describes is interpretation through scale anchoring, but it uses scale anchoring for two different purposes and departs from the way that Beaton and Allen (1992) did it. It derives criterion-referenced benchmarks by analysis of a large number of scale-anchored distributions, and it interprets the proficiencies of students within stages separated by benchmarks at the midpoints of the stages rather than at the benchmarks.

The assumptions of the criterion-referenced benchmarks are the same as the assumptions of the so-called construct maps defined by Wilson (2005, 2023). They assume that there is a qualitative order of levels or stages inherent in the construct and that it is possible to specify waypoints that separate stages with quantitatively and qualitatively different response distributions. Wilson uses such waypoints as benchmarks for the interpretation of test scores by Wright maps. Section 5 of this paper will show how to define and use waypoints as criterion-referenced benchmarks by a systematic analysis of a large range of scale-anchored distributions.

The set of criterion-referenced benchmarks is supposed to refer explicitly to the criteria to which interpretation refers, and it defines a model of a construct map where the waypoints defined by the benchmarks separate qualitative stages where students meet some but not all the criteria that interpretation refers to. The extreme waypoint should refer to students who meet all the criteria.

During our search for papers on criterion-referenced testing, we came across two papers that have been important for our work on criterion-referenced interpretation. Haertel (1985) outlines a conceptual and methodological framework for criterion-referenced assessment, and Clifford (2016) defined a framework for the interpretation of response probabilities in terms that refer to the development of abilities.

1.6 The outline of the paper

Development of a criterion-referenced assessment of educational tests is a stepwise procedure described in Figure 1.

Figure 1.

 The methodological components of the procedure for the definition of criterion-referenced proficiency categories.

Section 2 provides a brief description of PIRLS. The criteria on which our criterion-referenced classification of test results from PIRLS will be founded are implied by PIRLS’s theoretical framework.

Section 3 describes a measurement model that fits Danish data from PIRLS.

Section 4 discusses measurement issues. Measurement by log-linear Rasch models is quantitative, similar to measurement by conventional Rasch models. This section includes an example that illustrates the need for additional qualitative information on proficiency in addition to conventional quantitative measures of ability information on estimates of person parameters provided by IRT and Rasch models and defines the scale-anchored response distributions that we use to define item-referenced benchmarks.

Section 5 describes the three-step procedure that defines criterion-referenced categories. The first step estimates the scale-anchored distribution of responses to items and defines so-called item-referenced benchmarks. The second step defines domain-referenced benchmarks, and the third step defines the criterion-referenced benchmarks that we use to define criterion-referenced proficiency categories. Finally, this section proposes a criterion-referenced classification scheme that describes seven stages of development of reading literacy.

Section 6 explores the validity and reliability of criterion-referenced interpretation and examines the criterion validity by comparison of the stages of development defined in Section 5 and the PIRLS’s benchmarks and proficiency categories.

Finally, Section 7 recapitulates what we have attempted to do and discusses a number of methodological issues.

2. PIRLS

2.1 The theoretical framework

PIRLS distinguishes between two different reading purposes: reading for literary experience and reading to acquire and use information (Mullis & Martin, 2015). To take this into account, PIRLS selected and adapted texts with fictional and non-fictional contents together with questions or items associated with the texts.

In addition to this, PIRLS distinguishes between the following four comprehension processes and developed items where responses to questions only need one specific comprehension process.

Focusing on and retrieving explicitly stated information (INF)

Making straightforward Inferences (IFR)

Interpreting and Integrating Ideas and Information (I&I)

Evaluating and critiquing content and textual elements (E&C)

Haertel (1985) introduced the notion of item domains where “test items of a clearly defined type comprise an item domain” and claimed that “there is a general agreement that the criterion-referenced test score must be referable to a well-defined domain” (p. 24). Since responding to items in PIRLS items is supposed to depend on one of the four comprehension processes, it follows that PIRLS´s reading test covers four different item domains and that criterion-referenced assessment of a student´s reading proficiency must refer to the degree to which the student is able or unable to master all four comprehension processes.

2.2 Booklet 16

In 2016, the PIRLS assessment inventory consisted of 12 texts assembled in booklets. Our example uses data from Danish students’ responses to the items in Booklet 16, because PIRLS has published the texts of Booklet 16 together with the items[3]. Our analyses will illustrate how Booklet 16 would function as a stand-alone formative test.

Booklet 16 consists of two texts: “Mary’s Red Hen” providing literary experience, and “The Green Turtle” providing factual information. PIRLS used four types of items for each text:

Dichotomous multiple-choice (MC) items

Dichotomous items with open responses

Trinary items with open responses

Polytomous items with open responses scored from zero to three

Figure 2 shows the distribution of Booklet 16 items across the theoretical frame of reference. Since content validity requires that the items cover all parts of the theoretical framework, Figure 2 shows that the reading test of Booklet 16 is content valid.

Figure 2.

Distribution of Booklet 16 items across the PIRLS framework.

2.3 Scaling methodology

PIRLS applies non-Rasch IRT models for the analysis of the international data and Rasch models for secondary analyses of data at national levels (Martin et al. 2017) but does not explain why they use different models for international and national analyses.

In 2001, PIRLS fixed the origin and unit of measurement of their person parameter scale (θ_PIRLS) in such a way that the proficiency distribution across all PIRLS countries had a mean equal to 500 and a standard deviation (SD) equal to 100. To make results comparable across different PIRLS surveys, the definition of θ_PIRLS has been the same since 2001. Since many countries have joined PIRLS since 2001, it follows that the interpretation of θ_PIRLS values in terms of means and standard deviations of a relevant population no longer applies[4].

2.4 Interpretation through scale anchoring in PIRLS

PIRLS recognized that it takes much more than a value on a quantitative scale to be a meaningful statement on the level of a student’s proficiency and used direct scale anchoring to interpret test results. PIRLS used norm-referenced benchmarks in 2001 but decided on a different set of benchmarks that they have used since then (Martin, Mullis & Kennedy, 2006). Figure 3 shows these benchmarks together with the proficiency categories that PIRLS used in 2016. The benchmarks are close to the benchmarks of 2001, but they are not norm-referenced, because they do not refer to a specific standard population.

Figure 3.

PIRLS’s Benchmarks and proficiency categories used by PIRLS 2016

We regard the benchmarks that PIRLS has used as benchmarks of convenience, but this does not imply that the interpretation of the proficiency categories is an interpretation of convenience. PIRLS has invested time and effort in interpreting and documenting the abilities in reading at different levels in a way that makes sense in subject matter terms and is useful for accountability testing, but probably less useful for formative testing. The following sections will show how to address this issue with smooth estimates of scale-anchored probabilities and criterion-referenced benchmarks.

3. A measurement model for Booklet 16

3.1 The items

The Danish data from PIRLS 2016 is available online^[5]. Table 1 shows the Booklet 16 items with different colours, distinguishing between the comprehension processes needed to respond correctly to the items.

                                                                Table 1.                                                                      

Overview of Booklet 16 items.

Note: Black = Retrieving information; Green = Straightforward inferencing; Blue = Evaluating & Critiquing; Red = Interpreting & integrating

Table 2 shows the marginal distributions of responses to items for students with complete responses to Booklet 16 items. Items are presented in order defined by increasing manifest item difficulties, as defined by the average item scores divided by the maximum item score. Table 2 distinguishes between multiple choice (MC) items and items with open responses and uses the same colour codes as Table 1 to discriminate between different comprehension processes.

Table 2.

Distributions of responses to Booklet 16 items for students with responses to all items.

 Black = Retrieve information  Green = Straightforward inference  Blue = Evaluation & Critique   Red = Interpretation & integrating

Table 2 underscores two important differences between MC items and items with open responses. First, MC items are easier than items with open responses and second, that INF and IFR items are easier than E&C and I&I items.

It is not surprising that MC items are relatively easy items. The students are aware that the correct response must be one of the alternatives. Notice, however, that T06 and H15 are particularly easy IFR and E&C items with open responses.

3.2 The measurement model

We intend to interpret PIRLS scores by smooth anchor scaling and therefore need estimates of item parameters of a statistical measurement model. Beaton and Allen (1992) describe scale-anchored distributions where expected frequencies of responses to items are calculated according to a 3PL model. Their argument for this model is that “in many applications of test theory the 3PL has been found to fit test data reasonably well” (p. 201). We understand their argument as an argument for a statistical model of convenience, but we must insist on more than a model of convenience. To avoid bias and confounding of the interpretations of the scale-anchored probabilities of responses to items, we must have a measurement model that fits the data and is both realistic and meaningful. One complicating factor is that Booklet 16 consists of two so-called testlets with 16 items that share the same stimulus material. In models with or for testlets, the assumptions of local independence are neither plausible nor realistic and estimates by conventional models like the 3PL or the Rasch models will be biased. To analyze PIRLS’s data, we need a model describing the testlet structure where local dependence has been taken into account.

One possibility is the Rasch testlet model proposed by Wang and Wilson (2005), where local dependence among items within a testlet is generated by a random effect associated with the testlet. However, Wang and Wilson’s model would generate a two-dimensional structure with separate Rasch models for the two testlets. We considered these models, but tests of fit rejected the Rasch models for the two testlets, disclosing strong evidence of local dependence within both Rasch models.

Another possibility is the so-called Rasch model for item bundles proposed by Wilson and Adams (1995), where scores summarizing responses to items within bundles define super-items fitting Rasch models for polytomous items. The problem with this model is that it cannot be used for scale-anchoring, because the responses to the separate items are unavailable. Items that fit the Rasch model for item bundles must fit Kelderman’s (1984) log-linear Rasch model, where estimates of scale-anchored probabilities are available (Kreiner, 2026). For this reason, we fitted a log-linear Rasch model to the Danish Booklet 16 data and used this model to illustrate criterion-referenced interpretation. Appendix A provides some information on the item analysis by this model. The Rasch model for item bundles and the log-linear Rasch model share one important property of conventional Rasch models. The total raw score over items is sufficient for the person parameter, and the estimation of person parameters proceeds in exactly the same way as in conventional Rasch models. Kreiner and Nielsen (2026) provide one example where the estimates of the person parameters from the two models are close to indistinguishable. The major difference between the models is that the bundle model assumes that all items are locally dependent, whereas the log-linear Rasch model only assumes that there may be local dependence and regards the conventional Rasch model as a special case where all items are locally independent. We refer readers to the seminal papers by Kelderman (1984, 1989 & 1992) and to papers of Kreiner and Christensen (2002, 2004, 2007 and 2011a)^[6] for information on item analysis by log-linear Rasch models.

Figures 4 and 5 show two so-called chain graphs that visualize the association structure within the two testlets of Booklet 16. In chain graphs of log-linear Rasch models, a missing edge between two items means that they are locally independent^[7]. The association structure is complicated, but 13 out of the 32 items are nevertheless locally independent of the other items. To make the association structure interpretable in subject matter terms we use thick edges to indicate very strong association and color codes to distinguish between the different types of items. Retrieving info is white, Inferencing is green, Interpreting is red, and Evaluating is blue. Multiple-choice items have an M attached to the item name.

4. Measurement issues

IRT and Rasch models provide quantitative interval-scaled measurement by estimates of person parameters. In log-linear Rasch models, the estimate of the person parameter  is a function of the total score over items, because the score is sufficient for θ. From this it follows that  is a discrete variable with a range that consists of 41 real numbers that Rasch models refer to as logits.

Kreiner (2025) compared different estimators of person parameters in Rasch models. In this paper we use Warm’s (1989) weighted likelihood (WLE) because it is less biased than the other estimators.

Figure 4.

 IRT graph of the log-linear Rasch model of items related to “Macy’s Red Hen”.

Figure 5.

 IRT graph of the log-linear Rasch model of items related to “The Green turtle”.

Kreiner and Christensen (2013) describe how to estimate the exact distribution of  including bias and standard error of measurement (SEM). Table 3 presents Warm’s (1989) weighted likelihood (WLE) estimate of θ together with the bias and the standard error of measurement (SEM).  is virtually unbiased except for extreme values of the raw score.

4.1 Formative testing

Data on classroom applications of Booklet 16 is unavailable. We therefore use simulated data to illustrate how criterion-referenced interpretation may be useful for formative testing.

Table 4 shows the effect of an intensive reading course on the proficiency of a single student. The table includes the true values of the person parameter before and after the course, together with the simulated Booklet 16 scores and the estimates of the person parameters. Since we have an estimate of the distribution of the total score anchored at the person parameters, we have been able to simulate the scores without the responses to the items. The situation, therefore, is similar to a classroom application where responses to items are unavailable and where assessment of proficiency refers to the total scores or the estimates of the person parameters. In this case, the course increased the student’s scores by 9 points and the person parameter by close to 1 logit,  = 0.923. Since the difference between the scores is marginally significant^[8], we conclude that there is evidence of a positive effect of the course on the student’s proficiency.

Table 3.

WLE estimates of person parameters in the log-linear Rasch model for Booklet 16.

Table 4.

Measurement of reading before and after an intensive reading course.

4.2 Estimates of scale-anchored response distributions

Readers familiar with the Rasch model’s logit scale will probably conclude that an effect of this size is important^[9], but the problem with this conclusion is that the test results in themselves do not provide insights into what the student could do after the course, that he was unable to do before the course.

Knowing that  is measured by interval-scaled logits is of considerable interest to measurement specialists, but it is not helpful at all to the teacher who needs concrete information that can help her interpret what θ values equal to -0.948 and -0.025 say about what the student can do after the course that he could not do before. However, calculating estimates of scale-anchored response probabilities by inserting person parameters together with estimates of item parameters in the formulas that define the probabilities of responses to items may be helpful. Table 5 presents estimates of the response probabilities of the 17 items with open responses anchored at the two estimates of θ. We will use these probabilities to interpret the students’ abilities in terms of how difficult the items were to the student before the course and how easier they were after the course.

Table 5.

Scale-anchored distributions of response probabilities to items with open responses.

Note: Black = Retrieve information; Green = Straightforward inference; Blue = Evaluation & Critique; Red = Interpretation & integrating

Before the course, there were no easy items, and four items were close to unsolvable, with probabilities of incorrect responses larger than or equal to 90 %. After the course, there are three easy items (T06, H15, and T10) with probabilities of correct responses larger than or equal to 75 %, and only three difficult items (H08, H16, and H04) with probabilities of incorrect responses between 75 and 90 %. A closer look at the issues raised by these items^[10] may provide insight into what the student was able and unable to do after the course.

5. Criterion-referenced interpretation of Booklet 16 scores.

Criterion-referenced interpretation is supposed to describe the degree to which students   can read and understand relevant texts in terms that relate to their abilities, and that requires much more than information on the difficulties of a few items. The rest of this paper will describe one way to provide criterion-referenced interpretation by systematic and careful examination of scale-anchored probabilities interpreted in terms of the degree to which students’ proficiencies satisfy the criteria needed to read and understand texts similar to those in Booklet 16.

5.1 Item domains

Popham and Husek (1969, p.6) stated that “those who write criterion-referenced items are more attentive to defining the domain of relevant test results and the situation in which they should be required” and Glaser (1971) underscored that criterion-referenced tests should yield measurement which is interpretable in terms of performance standards defined by “representative samples of tasks” (p.37) drawn from item domains.

We refer to Berk (1980), Haertel (1985) and Reckase (2017) for information on item domains in criterion-referenced tests. In PIRLS, the comprehension processes define four item domains, and Mullis et al. (2017, pp. 85 and 97) describe the comprehension processes that characterize students at high and advanced levels of proficiency. Their summaries provide the criteria that students must meet to be able to read texts like Mary’s Red Hen and The Green Turtle. It is these criteria that our attempt to interpret Booklet 16 scores will refer to. The criteria are:

Retrieving and using information in texts:

• When reading literary texts, the student should be able to locate and distinguish significant actions and details embedded across the text.
• When reading informational texts, the student should be able to locate and distinguish information within a dense text or a complex table.

Straightforward inferencing depending on what they have read:

• When reading literary texts, the student should be able to explain relationships between intentions, actions, events, and feelings.
• When reading informational texts, the student should be able to provide explanations and reasons about logical connections.

Interpreting and integrating ideas and information:

• When reading literary texts, the student should be able to interpret story events and character actions to describe reasons, motivations, feelings, and character development.
• When reading informational texts, the student should be able to distinguish and interpret complex information from different parts of the text.

Evaluating and criticizing contents:

• When reading literary texts, the student should be able to evaluate the effect of the author’s language and style choices on the reader.
• When reading informational texts, the student should be able to evaluate visual and textual elements to consider the author’s point of view.

Hambleton (1980) provides an extensive list of requirements that items of criterion-referenced tests must meet in addition to the requirements of items for norm-referenced tests. These requirements are also relevant for criterion-referenced interpretation of measurement by IRT and Rasch models. Even though we assume that all items fit the log-linear Rasch model and therefore provide item information that adds to the precision of the quantitative measurement, it does not follow automatically that all the items are useful for criterion-referenced interpretation. Appendix B provides information on the items that we will use. We exclude multiple-choice items because the difficulty of an MC item is less than what the difficulty of the item would have been if students did not know that the correct response is one of the alternatives provided by the item, and we also exclude two very easy items (T06 and H15) with open responses.

This leaves us with 15 items with open responses that we refer to as the criterion-referencing CR-items. This is less than we would prefer if this had been more than an illustration of methodology, and we acknowledge that it is an open question whether the 15 items constitute a “representative” sample of items from the four domains that satisfy Hambleton’s list of requirements. However, it is enough to illustrate the methodology.

5.2 Item-referenced benchmarks

It is intuitively easy to interpret an estimate  in terms of scale-anchored difficulties of items, but the interpretation will be tied to the specific items that the students responded to, and criterion-referenced interpretation has to look beyond the specific texts and items of Booklet 16. It should attempt to describe the level of development of the student’s proficiency and assess whether students at this level are able to or unable to read and understand similar texts. We will address this issue by defining benchmarks in the same way as PIRLS, but we define benchmarks by a stepwise procedure that is fundamentally different from the way that PIRLS selected their benchmarks in 2006. The first step defines item-referenced benchmarks, the second defines domain-referenced benchmarks, and the third and final step defines criterion-referenced benchmarks.

Clifford (2006) characterized stages of development of abilities as either sporadic, emerging, developing or sustained depending on probabilities of incorrect and correct responses to dichotomous items. Table 6 classifies the stages of development in the same way. However, it includes polytomous items where partial credit is neither incorrect nor correct, and it adds two extreme stages that refer to absent and consolidated evidence of development.

Table 6.

Item-referenced classification of abilities and difficulties.

P₀ = PR(X_i = 0)  P₁ = PR(X_i = m_i)

Table 7 shows the probabilities of responses to items H06 and T08 anchored at WLE estimates of θ together with the item-referenced benchmarks defined by the lowest value of  that satisfies the requirements defined in Table 6

In Table 7, probabilities in bold numbers refer to item-referenced benchmarks. Sporadic is gray, emerging is red, developing is red, sustained is blue and consolidated is green.The emerging benchmark of H06 is equal to -2.34, where the probability of an incorrect score is less than 0.75 for the first time, while the emerging benchmark of the more difficult T08 is equal to -0.95.

The item-referenced benchmarks of locally independent dichotomous Rasch items would be consistently ordered at all levels of θ, because Rasch model for dichotomous items satisfy the requirement of invariant item ordering (IIO). Regrettably, Sijtsma and Hemker (1998) show that IIO only applies for polytomous items if they fit the rating scale model defined by Andrich (1978), and Kreiner & Christensen (2004) show that it never applies for locally dependent dichotomous items in log-linear Rasch models.

In Table 7, the developing, sustained and consolidated benchmarks of H06 lie above the benchmarks of T08, and the probability of a completely correct response is larger for the difficult T08 than for the easy H06 if θ > -0.84. This is not unexpected, but it complicates interpretation.

There is no straightforward algebraic way to predict the relationships between the benchmarks of CR-items in log-linear Rasch models with a mixture of dichotomous items and polytomous items. The best we can do is to collect the benchmarks in tables for each of the domains. Tables 8–11 present the benchmarks for each of the four comprehension processes.

5.3 Domain-referenced benchmarks

Identifying item-referenced benchmarks is only the first step. The second step defines domain-referenced benchmarks that encapsulate the information on item-referenced benchmarks within the domains. There is no simple formal way to do this. However, we suggest that students have reached a specific domain-referenced benchmark at the lowest score where most of the items associated with the domain have reached the item-referenced benchmark. Table 12 shows the resultsand  Figure 6 provides a visual illustration of the development of proficiency implied by the domain-referenced benchmarks. Retrieving information and straightforward inferencing develop faster than interpreting and evaluating texts, and interpreting develops, at least to begin with, faster than evaluating.

Table 7.

Scale-anchored probabilities of responses to items H06 and T08

Table 8.

Item-referenced benchmarks of items asking students to retrieve information in texts

Table 9.

Item-referenced benchmarks of items asking for straightforward inference

Table 10.

Item-referenced benchmarks of items asking for interpretation and integration

Table 11.

Item-referenced benchmarks of items asking for evaluation and critique