H1: Positioning the key response(s) at the edges increases the difficulty of the item

Of the eight NS items, where one or two key responses were placed at the ends, unlike their corresponding items, six showed absolute values in the expected order (Table 6, Figure 2). Notably, one of these (item 5) was significantly more difficult in Set C than in Set B (p=0.05: t=3.53), where both key responses were positioned at the edges.

For the PFC, when the absolute values were compared, all items were more difficult when the key response was positioned at one of the edges (Table 7, Figure 3). In one comparison (item 4) the difference was statistically significant when Sets F and G were compared (p=0.05: t=3.24).

Most results were directionally consistent with the hypotheses, yet did not attain statistical significance.

H2: Positioning the key response later in the order of response options increases the difficulty of the item

In terms of absolute values, the difficulty of the item increased in six of ten PFC items when the key response was positioned later in the order of response options (Table 7, Figure 3). Item 4 showed a statistically significant difference when Sets F and G were compared (p=0.05: t=3.24).

Most results were directionally consistent with the hypotheses, yet did not attain statistical significance.

H3: When two answers are required, positioning key responses in the incorrect order (i.e. presenting the second answer first) increases the difficulty of the item.

Five of eleven NS items were more difficult when the key responses were positioned in the incorrect order (Table 6, Figure 2). In one item (item 5) the difference was statistically significant (p 0.05: t=3.53).

The results are inconclusive with the hypothesis.

H4: When two answers are required, positioning the key responses further apart will increase the difficulty of the item.

The proximity of the key responses affected the difficulty of the items by increasing the difficulty – as expected – in five of six NS items; one of them (item 5) was statistically significant (p=0.05: t=3.53; Table 6, Figure 2). However, statistically, item 4 was significantly more difficult, but in the opposite direction (p 0.05: t=3.02).

Most results were directionally consistent with the hypotheses, yet two statistical significances were shown in different directions.

H5: Positioning the strongest distractor before the key response increases the difficulty of the item

When considering the a priori expected strongest distractors, items 2 and 6 in Set E were expected to be more difficult than the corresponding items in Set F, and item 5 in Set F was expected to be more difficult than the corresponding item in Set E. This was confirmed for items 2 and 5 when the absolute values of task difficulty were compared (Table 7, Figure 3). For the empirically strongest distractors, items 1 and 5 in Set E could have been expected to be more difficult than Set F (Table 5), but this was only confirmed for item 1 (Table 7, Figure 3). Nevertheless, there were no statistically significant differences in task difficulty.

Most results were directionally consistent with the hypotheses, yet did not attain statistical significance.

H6: Increasing the distance between the key response and the strongest distractor decreases the difficulty of the item

When considering the a priori strongest distractor (Table 7, Figure 3), as the distance increased, the items became more difficult for five of six items. This contrasts with what was hypothesized, and was statistically significant for item 4 (p=0.05: t=3.24). When considering the empirically strongest distractors, one of three items (item 3) became less difficult when the distance increased.

Most results were directionally inconsistent with the hypotheses, yet did not attain statistical significance.

4. Discussion

This pilot study has investigated how the position of response options in MC items affects the difficulty of items. Because the study only found a few statistically significant results, we must refrain from drawing definitive conclusions. Instead, the discussion will elaborate on the trends related to the overarching research questions and highlight the complexities inherent in this type of study.

How is the difficulty of an item affected by the placement of the key response?

In line with our first hypothesis, both NS and PFC showed a trend  – albeit not statistically significant or conclusive – of increased item difficulty when key responses were positioned at the edges, corroborating earlier findings (Attali & Bar-Hillel, 2003; Bar-Hillel & Attali, 2002; DeVore et al., 2016).

As far as the second hypothesis is concerned, the results are more inconsistent. In contrast to a proposed primacy effect, the results from four PFC items in this study revealed the opposite trend; items became easier when the key response was later in the order of response options. We speculate that this reversed effect is because response options three and four are spatially closer to the end of the folding and cutting sequence in the main picture, making comparison with these response options easier. This interpretation is further supported by several of the empirically strongest distractors tending to appear at response options three or four, and after the key response (see further discussion on its implications below). In turn, this might suggest that test takers are more likely to focus initially on response options that are spatially closer to the end of the sequence, rather than working from left to right. It could be related to the type of MC item being studied, i.e. one which comprises a sequential visuospatial main picture with horizontal response options. Such a design may lead to different response patterns compared to items with text-based options, or items with vertically stacked response options.

The NS items require two key responses to be selected in the correct order. To the best of our knowledge, no prior studies have investigated the impact of the placement of two key responses. Therefore, the rationales for H3 and H4 were based on logical reasoning rather than previous empirical evidence. However, this study included only a few items and found just one statistically significant difference, i.e. positioning the key responses further apart appeared to outperform incorrect ordering of key responses in terms of making items more difficult (items 9, 12, 16; Table 6, Figure 2). The interpretation here is that placing the key responses closer together makes comparison and integration of information easier, thereby reducing cognitive load. However, edge aversion is clearly a confounding factor. As the distance between key responses increases, the likelihood of one or both key responses being positioned at the edges also increases (Table 2). Consequently, even in larger studies, disentangling the effects of response distance from other factors influencing item difficulty may be challenging.

How is the difficulty of an item affected by the placement of distractors in relation to the key response?

Before discussing the results of the hypotheses, it is important to note that in nearly two-thirds of the items, the a priori expected strongest distractor was not the most frequently chosen incorrect response, i.e. the empirically strongest distractor. This discrepancy can likely be attributed to one or two factors: the positioning of the response options may influence which distractor becomes the most attractive, or the authors’ a priori judgement of similarity of distractor and key response may have been inaccurate. A qualitative inspection of the response options, and the proportion of participants selecting each option, suggests an interaction between these two factors. In some items, it is understandable why an alternative distractor emerged as the most attractive incorrect response. However, in other cases – particularly when the empirically strongest distractor was response option three or four – it is reasonable to conclude that the positioning itself may have influenced the likelihood of it being selected as the strongest distractor.

In three out of four PFC items, the item became more difficult when the strongest distractor was placed before the key response, supporting H5. However, in two of those three items, the key response was positioned at the edge. Thus, it is not possible to know whether the difference in difficulty is related to edge aversion (Attali & Bar-Hillel, 2003). Nor is it possible to know whether the response options before the key response were likely to divert test takers’ attention and interfere with the key response to the extent that they believed them to be correct (Kiat et al., 2018).

From PFC Sets F and G, our results showed that the test takers did not find it more difficult to differentiate between the key response and the strongest distractor, if they were placed close to one another. This is in contrast to an earlier proposal by Shin et al (2020). Nevertheless, as for H5, when the distance increased, the key response was often positioned at the edge, and this is likely confounding our ability to analyze the effect of the proximity of the key response to the strongest distractor.

Methodological considerations

The study design may have been overly ambitious in attempting to isolate the unique effects outlined in the hypotheses. One contributing factor is that even minor adjustments to response option positions could introduce unintended changes. For example, increasing the distance between key responses in the NS items, or between the key response and the strongest distractor in the PFC items, often resulted in the key response being positioned at the edge(s). While additional response options could have mitigated the impact of edge aversion when increasing distances, the number of response options in this study was fixed due to technical constraints. However, with more response options, another issue will arise. Distractors must have a certain degree of plausibility and contribute to the discrimination of responses, otherwise they become ineffective (Gierl et al., 2017). Therefore, disentangling the effects of response distance from other factors that influence item difficulty remains a challenge, because modifying one variable may inadvertently affect other aspects.

An additional challenge in addressing the hypotheses in this study was that the a priori expected strongest distractors in the PFC items did not always align with the empirically strongest distractors. As discussed above, two potential explanations for this discrepancy are author misjudgment and a possible position effect. This raises a further question: how can the effectiveness of distractors be objectively and quantitatively defined? Although the PFC test is considered a logical reasoning test, it also incorporates visuospatial components. In the field of visual perception, van der Helm (2000) discussed two key principles: the likelihood principle (Helmholtz, 1962) and the simplicity principle (Hochberg & McAlister, 1953). While the first principle states that the visual system will prefer the most likely interpretation, the latter states that the simplest interpretation is preferred. However, these principles may be difficult – or even impossible – to separate and they may result in the same predictions, but by means of very different lines of reasoning (van der Helm, 2000, p. 770). Furthermore, it has been emphasized the importance of distinguishing between relationships within the measurement mechanism and causal explanations of task difficulty (Melin & Pendrill, 2023). Specifically, the way measurement information propagates through a measurement system is distinct from how attributes are intrinsically associated with an object, and is not primarily influenced by the measurement mechanism itself. Thus, to determine whether a position effect influences which distractor emerges as the empirically strongest – and consequently affects task difficulty – it is essential to separate this effect from an objective definition of distractor effectiveness.

Recently, Hagenmüller (2021) claimed that it is possible to vary the item difficulty of knowledge items, regardless of item content, only by changing the position of the solution among the response options. Our results somewhat contradict that this applies regardless of item content. In this study, although not statistically significant, the easier NS items seem to be more affected by the changed position of distractors than more difficult NS items or the PFC items (Figure 2 and 3). Whether different item types, or items with different levels of difficulty, have greater or lesser sensitivity to changing the order of response options remains to be examined further.

In the present study, all test takers had completed regular cognitive tests, in up to 80 minutes, before they received the items under development for the updated SEB. As it was not explained to the test takers which items were the regular items, and which were under development can have, the test takers had no reason to disengage selectively, which likely helped sustain effort even in the presence of fatigue. Moreover, because the anchor sets (Set A for NS and Set D for PFC) was always administered first, any fatigue effects would be expected to influence the other items similarly; consequently, comparisons of task difficulty among those sets should be only minimally affected. A further consideration concerns the potential impact of unequal group composition: the proportions of test takers differed across age and selection groups. However, because we observed no DIF and no differences in estimated ability across groups in the anchor sets, any influence of these compositional differences on the results is likely negligible.

Distractor analysis is a crucial process for helping test developers understand why test takers make errors, by supporting diagnostic inferences about test performance (Gierl et al., 2017). This analysis can be conducted using descriptive statistics to identify nonfunctional distractors – those with low response frequencies – or through IRT models to examine distractor performance in relation to test takers’ abilities. In this study, due to the small number of items and complexity, we chose only a few basic methods. In forthcoming work, complements such as eye-tracking (Holzknecht et al., 2021) and a point‐biserial discrimination index of distractors (Attali & Fraenkel, 2000) may provide additional insights into the overall functionality of distractors in MC items.

Given the complexity of distractor patterns and the limited number of significant differences, questions arise regarding the practical implications of this type of research. However, it is widely accepted that the quality of test items – be they multiple-choice or other formats – is fundamental to the validity and reliability of assessments. Poorly constructed items, such as those with ambiguous wording or culturally biased content, can lead to misinterpretation or create inequities among test takers. Producing high-quality items requires careful attention during their construction, and robust review and validation procedures (Morrison & Embretson, 2018). As Rodriguez (1997) aptly noted, “Item writing has been, is, and always will be an art […] Item writing is also serious business,” underscoring the importance of combining creativity with rigorous standards in item development. Therefore, we argue that while the items fit the model, and their composition yields valid and reliable measures, a deeper understanding of how the positioning of key responses and distractors influences item difficulty will contribute to broader theoretical insights. This, in turn, can enhance item development and provide better guidance for constructing MC items.

5. Conclusion

The difficulty of an item may be influenced by the placement of the key response and the relative positioning of distractors. However, given the limited number of significant findings and occasional inconsistencies in trends, a key takeaway from this study is that examining the effects of ordering, and the influence of distractors in MC items is inherently complex. Isolating unique effects is challenging, as even minor adjustments to response option positions can introduce unintended changes.

Whether different item types, or items of varying difficulty, exhibit differing sensitivity to response option order remains an open question for further investigation. However, preliminary indications suggest that easier NS items may be affected more by distractor positioning than more difficult NS items or PFC items. Additionally, findings from the PFC items indicate that task-specific characteristics, particularly spatial organization, may interact with response placement. This interaction could create a visual focus that, in turn, influences item difficulty. Nevertheless, further research is required to disentangle the interplay between primacy effects, edge aversion, and spatial proximity in shaping response patterns across different item types.

Acknowledgment

The authors would like to express their appreciation to the whole research team working to update the existing Swedish Enlistment Battery: Dr. Maria Fors Brandebo, Dr. Stefan Annell, Dr. Eva Johansson, Prof. Gerry Larsson, Emma Oskarsson (all Swedish Defence University), and Dr. Daniel Bergh (University of Gothenburg), for valuable feedback during the study.

Funding details and disclosure statement

This study is a part of the Swedish Defence Conscription and Assessment Agency's work to update the existing Swedish Enlistment Battery. The work has been partially funded by the Swedish Defence Conscription and Assessment Agency. Beyond that, the authors have no conflicts of interest to declare.

Data Availability Statement

The data that supports the findings of this study has been retrieved from the Swedish Defence Conscription and Assessment Agency. Restrictions apply to the availability of this data, which was used under license for this study.

How to Cite

Melin, J., & Wahlkrantz, E. (2026). The interplay between key responses, distractors, and position effects in multiple-choice items: A complex and unresolved issue. Educational Methods & Psychometrics, 4 (SAMC 2024 Special Issue): 24.

Attali, Y., & Bar-Hillel, M. (2003). Guess where: The position of correct answers in multiple-choice test items as a psychometric variable. Journal of Educational Measurement, 40(2), 109–128.

Attali, Y., & Fraenkel, T. (2000). The point-biserial as a discrimination index for distractors in multiple-choice items: deficiencies in usage and an alternative. Journal of Educational Measurement, 37(1), 77–86.

Andrich, D., Sheridan, B., & Luo, G. (2012). RUMM2030+ [Computer software]. RUMM Laboratory Pty Ltd, Perth, WA.

Bar-Hillel, M., & Attali, Y. (2002). Seek whence: Answer sequences and their consequences in key-balanced multiple-choice tests. American Statististical Association, 56(4), 299–303. https://doi.org/10.1198/000313002623

Carnegie, J. A. (2017). Does correct answer distribution influence student choices when writing multiple choice examinations? The Canadian Journal for the Scholarship of Teaching and Learning, 8(1), Article 1. https://doi.org/10.5206/cjsotl-rcacea.2017.1.11

DeVore, S., Stewart, J., & Stewart, G. (2016). Examining the effects of testwiseness in conceptual physics evaluations. Physical Review Physics Education Research, 12(2), 020138. https://doi.org/10.1103/PhysRevPhysEducRes.12.020138

French, J. W., Ekstrom, R. B., & Leighton, A. P. (1963). Manual for kit of reference tests for cognitive factors (p. 127). Educational Testing Service.

Garcia-Segarra, P., Santamarta, V., & Falomir, Z. (2024). Educating on spatial skills using a paper-folding-and-punched-hole videogame: Gameplay data analysis. Frontiers in Education, 9. https://doi.org/10.3389/feduc.2024.1303932

Gierl, M. J., Bulut, O., Guo, Q., & Zhang, X. (2017). Developing, analyzing, and using distractors for multiple-choice tests in education: A comprehensive review. Review of Educational Research, 87(6), 1082–1116. https://doi.org/10.3102/0034654317726529

Guttman, L., & Schlesinger, I. M. (1967). Systematic construction of distractors for ability and achievement test items. Educational and Psychological Measurement, 27(3), 569–580. https://doi.org/10.1177/001316446702700301

Hagenmüller, B. (2021). On the impact of the response options’ position on item difficulty in multiple-choice-items. European Journal of Psychological Assessment, 37(4), 290–299. https://doi.org/10.1027/1015-5759/a000615

Haladyna, T., & Rodriguez, M. C. (2013). Selected-response format: Developing multiple-choice items. In Developing and Validating Test Items. Routledge.

Helmholtz, H. von. (1962). Treatise on physiological optics (J. P. C. Southall, Trans. and Ed.). Dover Publication.

Hobart, J., & Cano, S. (2009). Improving the evaluation of therapeutic interventions in multiple sclerosis: The role of new psychometric methods. 2009, 214.

Hochberg, J., & McAlister, E. (1953). A quantitative approach, to figural ‘goodness’. Journal of Experimental Psychology, 46(5), 361–364. https://doi.org/10.1037/h0055809

Hohensinn, C., & Baghaei, P. (2017). Does the position of response options in multiple-choice tests matter? Psicológica, 38, 93–109.

Holzknecht, F., McCray, G., Eberharter, K., Kremmel, B., Zehentner, M., Spiby, R., & Dunlea, J. (2021). The effect of response order on candidate viewing behaviour and item difficulty in a multiple-choice listening test. Language Testing, 38(1), 41–61. https://doi.org/10.1177/0265532220917316

Irvine, S. H., Dann, P. L., & Anderson, J. D. (1990). Towards a theory of algorithm-determined cognitive test construction. British Journal of Psychology, 81(2), 173–195.

Jonsson, E., Salo, M., Lillemäe, E., Steder, F. B., Ferst, T., Kasearu, K., Novagrockiene, J., Österberg, J., Sederholm, T., Svensén, S., Tresch, T. S., & Truusa, T.-T. (2024). Multifaceted conscription: A comparative study of six European countries (1). 7(1), Article 1. https://doi.org/10.31374/sjms.166

Kiat, J. E., Ong, A. R., & Ganesan, A. (2018). The influence of distractor strength and response order on MCQ responding. Educational Psychology, 38(3), 368–380. https://doi.org/10.1080/01443410.2017.1349877

Lions, S., Dartnell, P., Toledo, G., Godoy, M. I., Córdova, N., Jiménez, D., & Lemarié, J. (2023). Position of correct option and distractors impacts responses to multiple-choice items: Evidence from a national Test. Educational and Psychological Measurement, 83(5), 861–884. https://doi.org/10.1177/00131644221132335

Ludvigsson, J. F., Berglind, D., Sundquist, K., Sundström, J., Tynelius, P., & Neovius, M. (2022). The Swedish military conscription register: Opportunities for its use in medical research. European Journal of Epidemiology, 37(7), 767–777. https://doi.org/10.1007/s10654-022-00887-0

Melin, J., & Pendrill, L. R. (2023). The role of construct specification equations and entropy in the measurement of memory. In Jr. Fisher William P. & S. J. Cano (Eds.), Person-centered outcome metrology: Principles and applications for high stakes decision making (pp. 269–309). Springer International Publishing. https://doi.org/10.1007/978-3-031-07465-3_10

Morrison, K., & Embretson, S. (2018). Item generation. In The Wiley handbook of psychometric testing: A multidisciplinary reference on survey, scale and test development (1st edition). Wiley-Blackwell.

Rodriguez, M. C. (1997). The art & science of item-writing: A meta-analysis of multiple-choice item format effects.

Rodriguez, M. C. (2005). Three options are optimal for multiple-choice items: A meta-analysis of 80 years of research. Educational Measurement: Issues and Practice, 24(2), 3–13. https://doi.org/10.1111/j.1745-3992.2005.00006.x

Salgado, J. F. (2017). Using ability tests in selection. In H. W. Goldstein, E. D. Pulakos, J. Passmore, & C. Semedo (Eds.), The Wiley Blackwell handbook of the psychology of recruitment, selection and employee retention (1st ed., pp. 113–150). Wiley. https://doi.org/10.1002/9781118972472.ch7

Schroeder, J., Murphy, K. L., & Holme, T. A. (2012). Investigating factors that influence item performance on ACS exams. Journal of Chemical Education, 89(3), 346–350. https://doi.org/10.1021/ed101175f

Shin, J., Bulut, O., & Gierl, M. J. (2020). The effect of the most-attractive-distractor location on multiple-choice item difficulty. The Journal of Experimental Education, 88(4), 643–659. https://doi.org/10.1080/00220973.2019.1629577

Swedish Police Authority. (n.d.). polisen.se. Retrieved 22 January 2025, from https://polisen.se/jobb-och-utbildning/bli-polis/antagningskrav/

Tennant, A., & Conaghan, P. G. (2007). The Rasch measurement model in rheumatology: What is it and why use it? When should it be applied, and what should one look for in a Rasch paper? Arthritis & Rheumatism, 57(8), 1358–1362. https://doi.org/10.1002/art.23108

Thissen, D., Steinberg, L., & Fitzpatrick, A. R. (1989). Multiple-choice models: The distractors are also part of the item. Journal of Educational Measurement, 26(2), 161–176.

van der Helm, P. A. (2000). Simplicity versus likelihood in visual perception: From surprisals to precisals. Psychological Bulletin, 126(5), 770–800. https://doi.org/10.1037/0033-2909.126.5.770

Wahlkrantz, E., & Melin, J. (2025). Fair assessment of logical ability in military selection: A Rasch analysis of a paper fold and cut test. SAGE Open 15, nr 3 (2025): 21582440251372266. https://doi.org/10.1177/21582440251372266.

Manuscript Received: 26 FEB 2025

Final Version Received: 02 DEC 2025

Published Online Date: 15 FEB 2026

Appendix

The Interplay Between Key Responses, Distractors, and Position Effects in Multiple-Choice Items: A Complex and Unresolved Issue

Person-item histogram for NS. Level CD (blue) indicates Set A and B and level CE indicates Set A and C.

Person-item histogram for PFC. Level 1(blue) indicates Set A and B, level 2 indicates Set A and B, and level 3 indicates Set A and C.