Critical thinking has been defined as thinking which may involve inductive/deductive reasoning, and analysis of information, evaluation, judgement, problem solving, decision making, etc., in relation to evidence, claims, arguments and so on (van der Zanden et al., 2018). While definitions of critical thinking vary in their focus on specific aspects (e.g. Facione, 1990; Halpern, 2003; Khan, 2017; Petress, 2004; Pintrich et al., 1991), there is also agreement on what the core aspect of critical thinking are across these definitions: critical thinking is in essence concerned with questioning subject matter and making own decisions, evaluating theory, interpretations and conclusions, and evaluating alternatives (Nielsen et al., 2021; 2022). Such variations in definitions may also be at the root of differences in results when it comes to development of critical thinking throughout the university experience. While no previous studies evaluating whether different instruments based on different definitions of critical thinking would lead to different results could be identified, we find it resonable to pressume that different instrument would indeed lead to different results.  

Several definitions of critical thinking are closely connected with a specific assessment instrument, and different instruments are used depending on the purpose of the study in question. For example, the Critical Thinking Appraisal (WGCTA; Watson & Glaser, 1964) and the California Critical Thinking Skills Test (CCTST; Facione, 1991) have often been used in studies emphasizing individual assessment of detailed critical thinking profiles of higher education students (Staib, 2003; Facione, 1991). Both the WGCTA and the CCTST are complex instruments with 80 and 34 items, respectively, and they both require 45-60 minutes to complete the assessment. As such, they are detailed enough for achieving detailed profile information, but at the same time too resource demanding for larger N studies, or studies including many constructs to be measured. At the other end of the complexity spectrum, are the critical thinking scale from the Motivated Strategies for Learning Questionnaire (MLSQ; Pintrich et al., 1991) and a brief version of it (Nielsen et al., 2021), which consists of five and three items, respectively. These instruments appeal to usage in larger survey studies which include (several) other measures and/or include a large sample (e.g. Crede & Phillips, 2011; Duncan & McKeachie, 2005; Makransky et al., 2017; Nielsen et al., 2021; Nielsen et al., 2022), although they are not for detailed profiling.

The original MLSQ critical thinking scale (Pintrich et al., 1991) has been translated into more than 15 languages (Holland et al., 2018; Makransky et al., 2017; Nielsen et al., 2021), and it has been used to assess higher education students’ course-specific critical thinking as part of their meta-cognitive learning strategies (Crede & Phillips, 2011; Duncan & McKeachie, 2005). It was translated into Danish by Makransky et al. (2017) and used as an outcome in a study on the predictive validity of using admissions testing in higher education. They found that students admitted based on admission testing scored significantly higher on critical thinking and had significantly lower levels of dropout compared to students admitted based solely on grades. Makransky et al. (2017) reported that the Danish language version of the critical thinking scale was validated using the Partial Credit model (PCM; Masters, 1982), which is one parameterization of the implementation of the Rasch model (Rasch, 1960) for ordinal data. However, the psychometric properties of the Danish version of the MLSQ critical thinking scale were not documented, and no considerations about the content validity of the items of the scale were reported.

The MLSQ critical thinking scale (Pintrich et al., 1991) was translated into Spanish from Danish for a cross-cultural validation study, which aimed to test whether this brief scale could be used to compare critical thinking of higher education students from the two countries with similar disciplines (Nielsen et al., 2021). The study used Rasch models for the psychometric validation and included a thorough investigation of content validity. Nielsen et al. (2021) found that the MSLQ critical thinking scale contained two content-irrelevant items. A finding that was supported strongly by the psychometric evidence. Upon removal of the irrelevant items, the resulting three-item brief critical thinking scale was found to fit the ordinal Rasch model for the Danish sample as well as the Spanish sample separately. However, when analyzing the two country samples together, it was found that two items functioned differentially across countries, and thus cross-cultural invariance could only be achieved by splitting the person parameter estimates from the Rasch model (i.e. splitting for DIF; Hagquist et al., 2009) or equating the sum scores for DIF (Nielsen & Kreiner, 2024).

Previous research has not claimed that the brief critical thinking scale’s three-items cover all and detailed aspects of critical thinking. Rather that it covers the three aspects of critical thinking, which Nielsen et al. (2021) found to be common across three definitions of critical thinking (Facione, 1990; Halpern, 2003; Pintrich et al., 1991): the critical thinking aspects of how often the student questions things and decide about them, how the student decides about a theory, an interpretation or conclusion, and how the student looks for alternatives.

Previous validity studies (and other studies) have so far only used a course specific three-item scale (CTh) on samples of psychology students in various courses. Thus, there is currently a gap in the literature on the validity evidence for 1) it’s use with other university student groups, and for 2) a generalized (non-course specific) version of the scale.

1.1 The current study

In the current study, the aim is to conduct a broader validity study of a new non-course specific adaptation of the three-item critical thinking scale with a sample of university students from five different Bachelor’s degree programs belonging to five different academic faculties, while using the Rasch model for item analysis to ensure comparability to the previous studies. Thus, we will be testing whether the fit to the Rasch model extends to students from diverse disciplines with this more general version of the scale.  

2. Methods

2.1 Data

The data in the current study stems from the prospective cohort study of admission testing, pre-academic exam self-efficacy and retention by O’Neill and Nielsen (2024), where critical thinking was measured as one of several possible predictors of drop out. The data were originally collected within the first two weeks of the first semester in the included Bachelor’s degree programs, through an in-class online survey after a brief introduction of the project to motivate students to complete the survey, which resulted in a response rate of 77.3%.

The data used for the current study was critical thinking data collected with the General Critical Thinking scale (see section 2.2) and background data from 584 students in five different academic programs belonging to different faculties in one Danish university. Table 1 shows the distribution of the current study sample on the five background variables included for analyses of differential item functioning and invariance in the present study (see analyses section). Additional details on the study sample are provided in O’Neill and Nielsen (2024).

Table 1.

Characteristics of the study sample (N = 584)

2.2 Instrument

The General Critical Thinking (GCTh) scale used in the current study was a slightly adapted version of the CTh scale previously validated in both Danish and Spanish and found to fit the Rasch model (Nielsen et al., 2021; 2022). The adaptation was such that the scale was changed from measuring course specific CTh to general CTh. Thus, the only change to the items was a change of the reference “in this course” to a reference of “in my degree program”. The adaptations were done in collaboration with The Danish National Evaluation Institute. The GCTh items were rated according to how well they described the students in terms of frequency of the thinking, which was relayed in the items using a five-point response scale: 1 = never, 2 = rarely, 3 = sometimes, 4 = often, 5 = always.

Previous studies using the CTh scale reported reliabilities of 0.82 and 0.73 for Danish and Spanish psychology students taking courses in personality psychology and statistics, respectively (Nielsen et al. 2021), and 0.72 at baseline and 0.75 at one-year follow-up for Danish psychology students taking a statistics course (Nielsen et al., 2022). Reliability of the GCTh has not previously been reported. Table 2 shows the distribution of item responses, while Table 3 shows the raw score distribution of the GCTh prior to item analysis. The Danish items used in the current study are included in Table A1 in the Appendix.

Table 2.

 Distribution of responses to the General Critical Thinking scale (N = 584)

Note. The numbering of items follows the original numbering of items in the MLSQ and other publications employing the original CTh scale to facilitate easy comparison.

Table 3.

Raw score distribution on the General Critical Thinking Scale (N = 584)

2.3 Item analysis by Rasch models

Item analyses were conducted to investigate the psychometric properties and validity of the GCTh scale in detail. Both unidimensional Item Response Theory (IRT) and Rasch models (RM; Rasch, 1960; Andersen, 1983) assume that items in a scale adhere to (Kreiner, 2013; Mesbah & Kreiner, 2013):

Unidimensionality: The items of a scale assess one single underlying latent construct. In this study, the items should only measure the single construct of general critical thinking. This was established from the content of items, and the absence of negative local dependence among items (Marais, 2023; see below). 

Conditional independence of items (no local dependence; No LD): The response to a single item should be conditionally independent from the response to another item of the scale given the latent variable. In this study this means that responses to any one of the items in the scale should depend only on the level of general critical thinking, and not also on responses to the other items in the scale. This was tested for all item pairs.

Absence of differential item functioning (no DIF): Items and exogenous (i.e. background variables) should be conditionally independent given the latent variable. In this study this would entail that responses to any one item on the scale should only depend on the level of general critical thinking, and not also on subgroup membership (e.g. degree program, or gender). This was tested for each item and each exogenous variable.

Overall invariance: The entire set of items should function invariantly across subgroups (c.f. DIF, which was tested for a single item at a time).

In addition, RMs assume that item responses adhere to:

Overall homogeneity: The rank order of the item parameters (i.e. the item difficulties) should be the same across all persons regardless of their level on the latent variable. In this study, this means that the order of the three items should be the same in terms of difficulty no matter the level of general critical thinking, or put in another way, that the item requiring the least critical thinking to be endorsed and the item requiring the most should be the same for all students. Within the item analysis, homogeneity was tested for low and high scorers across the entire set of item parameters.

As items in the GCTh are ordinal polytomous items, we used the Rasch model for ordered polytomous data. This can be parameterized in different ways, which all provide the same results, and it is possible to go from one to another (Kreiner, 2013; Mesbah & Kreiner, 2013; Kreiner & Nielsen, 2023). The software package used (i.e. DIGRAM) uses the power series distribution for estimation, but presents for example. item parameters in Rasch’s original multiplicative form and as partial credit category thresholds and locations. As many readers are familiar with the Partial Credit Model (PCM; Masters, 1982), we include estimates of the partial credit thresholds and locations in the results.

2.3.1. Statistics

The RM is characterized by the sufficiency of the raw score for the person parameter estimates; A property adhering only to Rasch models. Therefore, the estimation of parameters and inference concerning the fit of items to the RM can be conditional. Conditional inference eliminates person parameters by using the conditional distributions of items that do not depend on the person parameters, and thus, conditional inference does not need assumptions concerning the distribution or sampling of persons (Kreiner, 2013; Rasch, 1961). All inference in DIGRAM is conditional and thus the person parameters are not used, when testing the model (Kreiner & Nielsen, 2013, 2023).

Item parameters were calculated using conditional maximum likelihood (CML) estimates, since these estimates are known to be consistent and do not require any assumptions on the distribution of the person parameters (cf. the above; Christensen, 2013; Kreiner, 2025; Kreiner & Nielsen, 2023). All tests used test the null hypothesis that item responses fit the Rasch model against different alternatives. Two overall tests of fit were conducted using Andersen’s conditional likelihood ratio test (CLR; Andersen, 1973; Kreiner & Christensen, 2013a): a test of global homogeneity by comparison of item parameters in low and high scoring groups defined to be as close to two equal-sized groups as possible, and a global test of no DIF across the entire set of items in the scale in relation to each background variable in turn, i.e., a test of overall invariance. Fit of the individual items to the RM was tested by conditional infit and outfit statistics (Christensen & Kreiner, 2013; Kreiner & Nielsen, 2023), and also by comparing the observed item-restscore correlations with the expected item-rest score correlations under the model (Christensen & Kreiner, 2013; Kreiner, 2011; Kreiner & Nielsen, 2023). Item-rest score correlations are correlations between and item and the score with the item excluded. Too strong item discrimination will result in an item-restscore correlation that is stronger than expected under the Rasch model, while too low item discrimination will result in the opposite (Christensen & Kreiner, 2013). Conditional independence of items (i.e. no local response dependence) and absence of DIF was tested using Kelderman’s likelihood-ratio test (Christensen & Kreiner, 2013; Kelderman, 1984; Kreiner & Christensen, 2013c).

DIF and invariance were tested in relation to five central background variables. Degree program (Medicine, Law, Biomedicine, History, Robot technology), Admission track (Grade-based admission, Test-based admission), Program priority (1^st priority, not 1^st priority), Gender (Female, Male), and Age groups (20 years and younger, 21 years, 22-25 years, 26 years and older).

Person fit was assessed with a two-step procedure. In the first step, the conditional probabilities of all response patterns for the different combinations of outcomes and exogenous variables are calculated. In the second step Martin-Löf’s exact test (Martin-Löf, 1977) is used to identify improbable response patterns. This test uses the conditional distribution of the response pattern given the total score as the test statistic, in the same way as Fisher’s exact test does in 2x2 tables. Conditioning with the sufficient score avoids using estimates of person parameters for the test of person fit (Kreiner & Nielsen, 2023, provides further explanations of the procedure, while Nielsen, 2024, provides another example of it’s use with scales fitting the dichotomous RM).

Once fit of the item responses to the Rasch model was established, person parameters were estimated and reliability and targeting were assessed. Person parameters were estimated using weighted maximum likelihood estimation, as these have been shown to be less biased than maximum likelihood estimates (Kreiner & Christensen, 2013b; Warm, 1989).

During analyses of reliability and targeting, we assumed that the distribution of the person parameter was normal, we estimated the mean and standard deviation of this distribution, and we used this distribution to assess reliability and targeting. To estimate the reliability, we generated simulated GCTh data on person parameters for 20,000 persons, calculated the expected true scores for all persons and generated two independent sets of item responses and person scores. Given this data, reliability can be estimated as it would be if the distribution of the person parameters had been normal. In the current study, we estimated reliability as the ratio of the variances of the expected and observed scores, and the Person Separation Index (PSI) as the measure of reliability for the theta scale.

Targeting was analyzed by comparing the best standard error of measurement (SEM) that the scale can provide to the average SEM in the study population. Since we are aware that measurement with a limited set of items is biased, we prefer to measure the so-called standard error of measurement by the RMSE(θ) instead of the SEM(θ). In addition, since we can define the exact RMSE as a function of θ, we can also find the person parameter where RMSE is minimized. We refer to this location as the target of the test, and refer to the test information and RMSE at this point as the target information and RMSE (Kreiner & Christensen, 2013b). Thus, the target of the test is the location on the q scale, where information is maximized and the RMSE is minimized. We assessed targeting by the two target indices (Kreiner & Christensen, 2013b): the target information index (the ratio between the average test information and the target information), and the target RMSE index (the ratio between the average RMSE and the RMSE at target). Both indices should have a value close to one. The target of the observed scores and the standard error of measurement of the observed scores were also calculated. Targeting was also illustrated graphically by plotting an item map showing the distribution of partial credit category threshold locations against the locations of the weighted maximum likelihood estimates of the person parameter, the person parameters for the population (assuming a normal distribution), and the test information function.

All the statistical tests used tested whether the data complied with the assumptions of the RM, with significant p-values providing evidence against the RM. Rather than applying a deterministic critical limit of 5%, p-values were evaluated as a continuous measure of evidence distinguishing between weak (p < 0.05), moderate (p < 0.01), and strong (p < 0.001) evidence against the model (Cox et al., 1977). The Benjamini-Hochberg procedure was applied to adjust the false discovery rate (FDR) taking multiple testing into account, when appropriate (Benjamini & Hochberg, 1995). All analyses were done using the Digram software package (Kreiner, 2003; Kreiner & Nielsen, 2013; 2023), while the item maps were created using R (R Core Team, 2021).

3. Results

The GCTh scale fitted the Rasch model for the total sample of 584 students in the five different degree programs belonging to five different faculties. There was no evidence against global homogeneity, and thus the scale functioned in the same way for low as well as high scorers (Table 4). There was no evidence against overall invariance, in the sense that the entire item set functioned in the same way across subgroups defined by degree programs, admission track, program priority on application, genders or age groups (Table 4). Nor was there any evidence against fit of the individual items to the Rasch model with any of the three tests used (Table 5).

Table 4.

Global tests of homogeneity and invariance for the final General Critical Thinking scale model (N = 577)

Notes. CLR = Conditional Likelihood Ratio test

a The test of homogeneity is a test of the hypothesis that item parameters are the same for persons with low or high GCTh scores.

Table 5.

Item fit statistics for the General Critical Thinking Scale (N = 577)

Notes. g = Goodman & Kruskal’s gamma coefficient (Goodman & Kruskal, 1954).

Tests for differential item functioning (DIF) for single items in relation to the aforementioned background variables, did not reveal any evidence of DIF. Likewise, test of conditional local independence of the items did not reveal any evidence against this (Table A2 in the Appendix).

3.1. Person fit

There was no evidence against person fit beyond what could be expected, when assessed across the total sample (Table A3 in the Appendix shows the misfit frequency at each score level, while Table A4 in the Appendix shows it broken down on subgroups defined by each exogenous variable).  The frequency of person misfit was 29 (5%), while the expected misfit frequency was 20 (3.4). The difference was very small and borderline insignificant (SD = 4.41, z = 1.9973, p = 0.0458). Thus, there was no basis for claiming significant (or relevant) misfit of persons, and thus we claim the Rasch model fits all persons.

3.2. Targeting and reliability

The target of the test only lies a little more than a half logit below the mean of the population, and the population averages of test information and RMSE are less than 5% below the target information and the RMSE. In our experience, this is extremely good targeting. The reason may be a standard deviation of the person parameter in this study population of

 1.35. This together with an average RMSE equal to 0.84 explains the relatively low reliability of the GCTS.

The GCTS is perfectly targeted, but it does not provide precise person level measurement. The excellent targeting is also illustrated in the item map, which shows that items cover almost the entire range of persons, and that information is almost equally high along most of the scale (Figure A1 in the Appendix). Reliability of the raw sum score scale and the estimated latent scale (q) was identical (c.f. PSI and r in Table 6), and at a level just sufficient for early stages of predictive or construct validation research, e.g. at least 0.70 (Nunnally & Bernstein, 1994). Category thresholds, item difficulties and item targets are provided in Table A5 for the interested reader.

Table 6.

Targeting and reliability of the General Critical Thinking Scale (N = 584)

Notes. The target info index = the ratio between the average test information and the target information. The target RMSE index = the ratio between the average RMSE and the RMSE at target.  PSI = Person Separation Index = the ratio between the variance of θ in the population and the variance of the estimates of θ. SEM = The standard error of measurement of the observed score. r = reliability estimated as the ratio between the variance of the expected true score and the observed mean score.