Modeling person guessing as a random effect: a Bayesian approach of the two-parameter logistic model

Abstract

Introduction:

Guessing behavior has been an enduring problem that undermines the validity and interpretability of scores from MC items. The present study implements a Bayesian random-effects extension of the 2PLE model which suggests that guessing is a latent individual trait rather than a single item parameter.

Methods:

We implemented a Monte Carlo simulation in a fully crossed design of sample sizes (N = 100–1,000) and test lengths (6–40 items), with 50 replications per condition. Item response data were simulated under the 2PLE model with heterogeneous guessing.

Results:

In all conditions the estimates of discrimination were larger with the 2PLE than with the 3PL. Gains were especially marked for item difficulty and lower-asymptote estimation that had noticeable distortion under the incorrect 3PL model. Bayesian predictive fit indices (i.e., Leave-One-Out Information Criterion, LOOIC; Widely Applicable Information Criterion, WAIC) consistently supported the 2PLE model under all sample sizes and test lengths. In the proposed framework, the person-level random effect δ_n reflects differences between individuals in guessing tendency and directly influences the lower asymptote of an item response function.

Discussion:

Through reallocating guessing variance from items to persons the 2PLE random-effects model can better capture diversified response patterns, and obtain a better psychometric performance. Findings are consistent with the conceptualization of guessing as a substantive trait-based process and underscore the utility and necessity of using person-specific guessing models to optimize inferences from test scores.

Introduction

The reliability and validity of test scores have always been essential for standardized educational and psychological measurement. One of the eminent threats to these properties according to broad-based (multifaceted) test development models is guessing in multiple-choice response formats because it compromises the construct relevance of measurement error associated with correct responses from guessing versus indifference rather than aptitude per se (Downing and Haladyna, 2004; Wise and DeMars, 2006). Consequently, the guessing behavior, random, educated, strategic, implies that interpretations of scores can also be distorted and the validity or reliability of test results may be compromised. Empirical evidence has indicated increased awareness that guessing can threaten test validity of constructs assessed by standardized tests, and alternative assessment and scoring procedures have been advocated to reduce its influence (e.g., Lee and Bolt, 2018). This study has broader implications for the field of educational and psychological measurement by providing an alternative analytical model which can accurately model guessing as a person attribute.

Psychometric issues and conceptual underpinnings of person guessing behavior

Guessing places construct-irrelevant error variance into test scores (Downing and Haladyna, 2004) and biases Item Characteristic Curves (ICCs) in Item Response Theory (IRT). The 3PL model includes a pseudo-guessing (c) that represents the lower asymptote of the item characteristic curve, i.e., the probability of a correct response for very low-ability examinees who may still benefit from partial knowledge or elimination of distractors (Birnbaum, 1968). This approach may capture the statistical deference of guessing but may not fully account for motivational or cognitive factors. This is explicated by Kane and Moloney (1978) who model the answer-until-correct (AUC) method. Their use of a modified Horst model suggests that item reliability under AUC is highly sensitive to examinees’ abilities to eliminate wrong choices, and the number of distractors included in the test. Guessing has, thus, the greatest negative impact on item and scale reliability.

In the measurement literature, several distinct forms of guessing have been discussed, each reflecting different cognitive or motivational processes. Random guessing refers to indiscriminate responding in the absence of item-specific knowledge, typically yielding chance-level accuracy. Lucky guessing denotes correct responses obtained by chance despite lack of knowledge and is therefore an outcome rather than a behavioral strategy. Rapid guessing describes a time-based disengagement behavior in which examinees respond quickly without attempting item solution (Wise and Kong, 2005 In contrast, ability-based guessing, as defined by the 2PLE model of Zhu et al. (2019), assumes that the probability of successful guessing when guessing happens is a deterministic function of ability, that is, examinees with the same ability have the same chance to guess right. We extend this model to incorporate person-based guessing, which we conceptualize as a latent trait reflecting individual differences in the tendency to guess (δ_n) systematically and nonrandomly across content beyond ability. This proposition gives expression to the idea that individuals at the same ability level under guessing may have different baseline probability of correct responding, corresponding to stable motivational or strategic differences rather than random noise.

Model description: moving from the 3PL to the 2PLE and 2PLE random model

The item response function of the 3PL IRT model is given by Equation 1:

Where P(θ) is the probability of a correct response given an ability level θ, a_i is the item discrimination parameter, b_i is the item difficulty parameter, and c_i is the lower asymptote representing the guessing effect at the item level (Birnbaum, 1968). In the 3PL model (Birnbaum, 1968), the lower asymptote c_i is treated as an item-specific fixed parameter and is typically estimated across all examinees under the assumption that the pseudo-guessing parameter c is an item-specific constant shared across examinees. Although its effect on the probability of a correct response is modulated by ability via the logistic term, the 3PL does not allow individual differences in guessing tendencies beyond what is implied by θ itself. While this formulation is computationally convenient, it fails to account for known inter-individual variability in guessing propensities due to differences in motivation, strategy use, or test-taking behavior (Wise and Kong, 2005).

In contrast, the 2PLE model proposed by Zhu et al. (2019) reconceptualized guessing as a person–item interaction, with a logistic function governing the pseudo-guessing as a function of person ability. The model incorporates a latent trait-conditioned guessing function that depends not solely on item-level characteristics but also on person-specific parameters, thereby allowing individual differences in guessing behavior to be explicitly modeled and quantified (for the multidimensional case see Youxiang et al., 2024). The 2PLE model introduced by Zhu, Wang, and Tao conceptualized guessing as a deterministic function of the person’s ability. Specifically, the guessing probability g_ni was defined as Equation 2:

In the above equation, γ is 1/k for k-options items (e.g., 0.25 for a 4-choice item). This baseline guessing formulation is specific to the original 2PLE model and is not retained in the present extension. In other words, in the Zhu et al model, there is no individual-specific guessing parameter; rather, for examinees who do guess, the conditional probability of a correct response increases with θ, reflecting the assumption that more able examinees can more effectively eliminate implausible options and thus guess more accurately (Zhu et al., 2019). This does not imply that high-ability examinees guess more often, only that their guesses, when they occur, are more likely to be correct. There is no individual-specific pseudo-guessing—rather, all persons with the same θ level have the same guessing probability. In other words, successful guessing is entirely governed by the levels of the latent trait θ and the properties of the discrimination and difficulty parameters a_i, and b_i. The Zhu et al. model introduced a dynamic, ability-based lower asymptote, in an effort to correct of potential shortcoming of the 3PL model, which treats guessing behavior as an item-level constant. In Zhu et al.’s (2019) model, guessing is entirely determined by ability with guessing being modeled at the person level. Thus, individuals with the same θ are predicted to have identical levels of successful guessing. In the present model we attempted to add a layer of individuality and variability in guessing for individuals with the same θ.

To visualize the substantive impact of the person-level guessing trait, ICCs for a representative item are plotted at low, medium, and high values of δ_n, illustrating how δ_n shifts the lower asymptote while preserving the ability-based slope. As δ_n increases, the person-level lower asymptote g_n increases, producing a systematically elevated baseline probability of success among low-ability examinees. This visualization makes explicit how the 2PLE random-effects formulation accommodates heterogeneity in guessing behavior that the 3PL model cannot represent.

Psychological Interpretation of the δ_n trait

The latent person parameter δ_n is conceptualized as an individual difference reflecting the examinee’s propensity to engage in guessing. Psychologically, δ_n may reflects a latent propensity toward non-construct-relevant responding, which may arise from disengagement, impulsivity, strategy use, or motivational factors. Modeling δ_n allows the response function to capture systematic deviations from construct-relevant behavior and helps disentangle whether low-ability examinees are performing poorly because of limited skill or because of erratic or disengaged response patterns. In this way, δ_n plays a role analogous to person-fit indices in classical test theory but is integrated directly into the IRT response model. Importantly, δ_n is not intended to represent unsystematic error or residual variance, but rather a latent behavioral tendency inferred probabilistically from response patterns, reflecting systematic individual differences in engagement and response strategy that are orthogonal to ability.

Current model specification and estimation: the 2PLE with random person guessing

The item response function of the proposed 2PLE model with random person guessing for person n on item i is modeled via Equation 3:

where θ_n denotes the latent ability of person n_i and a_i,b_i are the item discrimination and difficulty parameters, respectively. In contrast to traditional IRT models such as the 3PL, which employ a fixed itemlevel pseudo-guessing c_i, the proposed model introduces a person-specific guessing probability g_ni, reflecting interindividual variability in test-taking behavior.

The probability of guessing is modeled as Equation 4:

Where is a person-specific latent trait. The behavior of this formulation lies on influencing the lower asymptote, moving it in upward or downward fashion depending on individual’s proclivity to guess. The logistic function ensures that less able and higher-delta-responders have higher probabilities of guessing correctly (which is in line with whether these are disengaged or strategic responders). In this study, the lower asymptote at baseline was fixed as zero so that person- specific guessing parameter δ_n would range over the full [0, u] range instead of being raised from an artificially high starting point from 1/k in multiple choice testing as used in Zhu’s et al., original 2PLE model. By treating guessing as a latent individual-difference variable, the proposed model offers a more nuanced account of test-taking behavior as more than just aberrant responding.

In more detail the present model extends the framework of Zhu et al. (2019) by incorporating a random effect for person-level pseudo-guessing that accounts for variability among individuals in guessing. Thus, the 2PLE model assumes an ability-dependent deterministic guessing function of item-condensing guessing based on person-independent item-specific guessing (which is modulated by δn), while the present generalization in turn introduces a random latent variable δ(n) permitting systematic dispersion of guessing behavior across persons other than what can be predicted from classic IRT formulations such as the 3PL. By including factor δn, the model separates guessing from θ and permits better estimation of both ability and item parameters. This partition is a way to reduce construct-irrelevant variance which would otherwise be attributed either to θn (inflating ability estimates) or to ai (deflating discrimination) (Figure 1).

FIGURE 1

In conclusion, the model we proposed extends the dynamic person-item interaction framework of 2PLE by introducing person-specific random effect to the lower asymptote. This extension permits heterogeneity in guessing behavior for different examinees, even given ability. Functionally, the model (a) accommodates variance in test-taking behavior that is unrelated to trait ability, (b) enhances parameter estimation by accounting for systematic guessing bias, and (c) affords new person-level diagnostics: examinees with high or low δn can be flagged as “high guessers” or “low guessers,” respectively. By conceptualizing guessing as a latent individual difference variable, PP and Ppb provide a better understanding of test-taking behavior in terms of underlying motivational, cognitive, or affective tendencies and support the identification and interpretation of abnormal response patterns.

Computation of item guessing using person-based estimates

Following estimation, we computed average person-based guessing probability estimates at the item level i as follows Equation 5:

where g_ni denotes the estimated guessing probability for person n on item i, calculated from the posterior medians of the person and item parameters-namely a_i,b_i,θ_n, and δ_n. These item-level averages, , reflect a descriptive summary and not an estimated parameter of the lower asymptote of each item’s item characteristic curve (ICC), but unlike the fixed pseudo-guessing c_i in the 3PL model, they reflect a dynamic interaction between person traits and item properties.

Implications for measurement and practice

Estimating person- level guessing tendencies has several substantive advantages over the traditional approach of examining guessing at the item level. The first is that identification of examinees showing unusually high guessing behavior can help make inferences about the validity of a test, as well as the effort or strategy of the examinee. Second, it purifies ability estimates, particularly for low ability examinees who may respond in non-construct-relevant ways. Third, it strengthens the item-level diagnostics, the overall average guessing rate on an item level would give context sensitive information on item working under realistic testing conditions.

Analytical choice: opting for the Bayesian model

From a technical standpoint, Bayesian estimation offers distinct advantages over maximum likelihood (ML) methods when modeling item response theory (IRT) models that incorporate person-level guessing behavior. In traditional ML estimation, parameters such as person abilities θ_n, and item characteristics-including discrimination a_i, difficulty b_i, and possibly guessing c_i-are estimated by maximizing the likelihood function L(θ,a,b,c∣Y), where Y is the observed item response matrix. However, such point estimates can be unstable in the presence of guessing-related behaviors, which often flatten or distort the likelihood surface and lead to convergence difficulties or local maxima. Furthermore, ML estimation relies on asymptotic approximations to quantify uncertainty, which may be invalid in complex models or with sparse data. By contrast, Bayesian methods rely on full posterior inference (Equation 6):

where p(θ,a,b,c) denotes prior beliefs about parameters. This approach inherently carries the uncertainty through the model and gives full posterior distributions for all parameters, leading to more reliable conclusions. According to Dienes (2014), Bayesian statistics can evaluate competing hypotheses, such as ability-based processing vs. deviant guessing. Bayesian estimation also characterizes parameter identifiability problems via the use of informative or weakly informative priors. For example, restricting pseudo-guessing to a plausible range (e.g., [0.2, 0.3] for four-point multiple-choice items) enhances stability, particularly when data are corrupt with noise or careless responding. Such regularization is crucial to accurately model systematic variation in guessing behavior across individuals, as in person-fit analyses, or random person-level pseudo-guessing models. In conclusion, the Bayesian estimation provides a principled and flexible approach to modeling complex response behaviors in IRT formulations. It enhances parameter recovery, allows for model-based uncertainty, and facilitates form-fit comparisons using criteria such as LOOIC or Bayes factors. These benefits are particularly crucial in models that have the goal to separate ability from irrelevant variance (e.g., strategic guessing or disengagement scenarios) for which ML approaches are often not suitable. In the 3PL comparison model, we employed an identical Bayesian MCMC framework again with weakly informative priors on c to keep it within plausible values [e.g., c ∼ Beta(α, β) centered around 1/k for k-option items], which follows standard practice in Bayesian estimation of 3PL pseudo-guessing parameters.

Research on test security has proposed a broad suite of statistical and algorithmic approaches to detecting outlier behavior, including similarity indices (Wollack, 1997), person-fit indices like lz and Ht (Cizek and Wollack, 2017; van der Linden and Sotaridona, 2006), divergence-based measures based on Kullback–Leibler or Jensen–Shannon distances (Fox and Marianti, 2016; Lin, 1991; Sinharay, 2016). These methods target behaviors such as: Answer copying, pre-knowledge, random responding and lucky guessing. While informative, these methods operate diagnostically after the response pattern is generated. The current model complements this literature by embedding aberrant tendencies directly into the response function through the δ_n parameter, thus integrating aberrance detection and ability estimation within a single framework. Whereas traditional test security indices aim to flag aberrant response patterns after test administration, the proposed 2PLE framework probabilistically attributes guessing-related aberrance during model estimation, thereby integrating detection and correction within a single psychometric model

A related literature investigates rapid guessing using response times (Schnipke and Scrams, 2002; Wise and Kong, 2005). Such models distinguish solution behavior from disengaged responding based on speed, but do not directly model the probability of success conditional on guessing. The present approach is complementary: whereas response-time models infer disengagement behaviorally, the 2PLE framework models guessing probabilistically within the item response function. This provides a psychometrically principled method for representing individual differences in guessing even in the absence of response-time data.

Empirical evaluations of response-time–based rapid-guessing methods generally support their utility for identifying low-effort responding, especially in low-stakes settings; however, their effectiveness depends on the availability and quality of response-time data and on defensible thresholding or mixture assumptions that can vary across tests, devices, and examinee subgroups. Moreover, response-time approaches primarily classify effort states (e.g., solution vs. rapid responding) rather than estimating a person-level propensity that directly parameterizes the probability of a correct response under guessing. Consequently, even when response times are available, they do not provide a fully probabilistic decomposition of observed accuracy into construct-relevant ability versus guessing-related success. The present 2PLE random-effects approach addresses this gap by modeling heterogeneity in guessing success directly in the item response function, enabling identification and adjustment for person-level guessing tendencies even in datasets where response times are unavailable.

The primary research question guiding this study is whether modeling guessing as a person-level latent trait improves psychometric performance relative to the traditional Three-Parameter Logistic (3PL) model when guessing behavior varies across individuals. Specifically, we examine whether a Two-Parameter Logistic Extension with random person-level guessing (2PLE) yields superior parameter recovery, reduced bias, and improved predictive model fit compared to the 3PL model. To address this question, we employ a Monte Carlo simulation design in which response data are generated under a person-level guessing mechanism and analyzed using both models. Model performance is evaluated using parameter recovery indices and Bayesian measures of predictive fit, including LOOIC and WAIC. The following section describes the simulation design, data generation procedures, estimation approach, and evaluation criteria in detail.

Materials and methods

Simulation design

The simulation study was conducted according to Monte Carlo procedures (de Ayala, 2009; Muthén and Muthén, 2002), where multiple datasets are generated under known true conditions to check parameter recovery and model fit. Sample sizes (N = 100–1,000) and test lengths (I = 6–40) were chosen based on educational/psychological testing ranges typically encountered in empirical studies (Lord, 1980; de Ayala, 2009). Fifty replications per condition were selected, given that previous Monte Carlo research has shown this number is sufficient for estimating validly parameter estimates, thus, being computationally feasible. Person-level guessing propensities δ_n were generated from normal distributions with varying standard deviations (σδ) and transformed to the unit interval via the logistic function to obtain valid guessing probabilities g_n ∈ (0,1). This transformation ensures probabilistic coherence while allowing unbounded latent heterogeneity on the δ_n scale. Two values of σδ (0.5 and 0.8) were selected to represent moderate and moderately high guessing heterogeneity; additional values are examined in sensitivity analyses reported in Supplementary Appendix.

Data generation

The present study adopts a Monte Carlo simulation design in which dichotomous item responses are generated under a two-parameter logistic extension model with random person-level guessing (2PLE random-effects model). This design allows direct evaluation of parameter recovery and model fit when guessing behavior varies systematically across individuals.

Generating person-level parameters

For each replication and simulation condition, person abilities were drawn independently from a standard normal distribution (Equation 7),

reflecting the conventional assumption of normally distributed latent ability in unidimensional IRT models. In addition, each person was assigned a latent guessing propensity (Equation 8),

where σ_δ controls the degree of heterogeneity in individual guessing behavior. In the present study, σ_δ = 0.8 was selected to represent a moderate-to-high level of interindividual variability in guessing propensity, consistent with empirical findings on disengaged and aberrant responding in low-stakes and operational testing contexts. Person-specific guessing probabilities were obtained via a logistic transformation (Equation 9):

which maps the unbounded latent trait δ_n onto the unit interval (0,1), yielding valid lower-asymptote probabilities. In the present implementation, the baseline lower bound γ was fixed at 0, such that all lower-asymptote variation is captured by the person-specific parameter ϑ_n. Across simulation conditions, the implied distribution of g_n exhibited substantial heterogeneity, with values spanning nearly the full unit interval (see Table 1 for specifications of generated parameters).

Generating item-level parameters

Item discrimination and difficulty parameters were generated jointly to reflect realistic psychometric conditions observed in applied testing. Specifically, item parameters (a_i,b_i) were drawn from a correlated bivariate distribution (Equation 10),

with the correlation ρ_ab = −0.30, reflecting empirical evidence that more discriminating items tend to be moderately easier in operational assessments. Discrimination parameters were exponentiated to ensure that non-negative values are present, resulting in a lognormal distribution for a_i, while difficulty parameters followed a normal distribution centered around zero.

Response probability model

Using person and item parameters, the probability of a correct response for individual n on item i was computed in line with the 2PLE random-effects model as follows Equation 11:

where g_n reflects the person-specific lower asymptote and u the upper asymptote. In the present study, we fixed the upper asymptote at u = 0.98, allowing some minor variability from perfect performance.

Dichotomous item responses were generated from Equation 12:

With individual differences in guessing behavior directly contributing to the baseline probability of success, especially at low-levels of theta, while preserving the standard logistic relationship between ability and item difficulty at higher levels of theta. For each simulation condition, summary statistics of the generated person- and item-level parameters are shown in Table 1.

Model estimation procedure

All simulated datasets were analyzed within a Bayesian framework using Stan via the CmdStanR interface. Models were estimated using Hamiltonian Monte Carlo with the No-U-Turn Sampler (NUTS) as implemented in CmdStan (version 2.36.0) and CmdStanR (version 0.8.0), executed in R (version 4.3.x). For each dataset, four Markov chain Monte Carlo (MCMC) chains were run with 2,000 iterations per chain, including 1,000 warm-up iterations and 1,000 post-warm-up sampling iterations (see Supplementary Appendix 1 for Stan code).

Person-level latent parameters for ability (θ_n) and guessing propensity (δ_n) were estimated using a non-centered parameterization, whereby standardized latent variables were drawn from standard normal distributions and subsequently transformed via location and scale parameters. This parameterization was adopted to improve sampling efficiency and reduce posterior correlations between latent effects and their variance components. Item parameters were estimated using centered parameterizations with weakly informative priors.

Weakly informative priors were specified using the same functional families as the data-generating distributions but with hyperparameters chosen to span a broad range of plausible values rather than to closely reproduce the generating mechanism. Specifically, item discrimination parameters were assigned lognormal priors, item difficulty parameters normal priors, person abilities normal priors, and person-level guessing propensities normal priors. During estimation, item and person parameters were assigned independent priors, even when they were generated from correlated distributions, to avoid artificially favoring the data-generating model.

Convergence was assessed using multiple diagnostic criteria. Across all simulation conditions, the Gelman–Rubin statistic (R̂) was below 1.01 for all parameters, and both bulk and tail effective sample sizes exceeded 400. No divergent transitions were observed. In addition, trace plots for representative parameters were visually inspected to confirm stable mixing. Posterior predictive checks further indicated that the fitted models adequately reproduced key characteristics of the simulated data.

For comparability, the same MCMC settings, priors, and convergence criteria were applied to the Bayesian 3PL model. In the 3PL specification, guessing is modeled as an item-level fixed parameter rather than a person-level latent trait and therefore does not involve a hierarchical variance component.

Posterior samples were used to compute the pointwise log-likelihood of observed responses, which served as the basis for Bayesian model comparison. Two widely used measures of out-of-sample predictive accuracy were calculated: (a) the Leave-One-Out Information Criterion (LOOIC) using Pareto-smoothed importance sampling, and (b) the Widely Applicable Information Criterion (WAIC). Both indices were computed using the loo package in R. For each simulation condition, fit statistics were aggregated across the 50 replications to obtain mean and standard deviation estimates of LOOIC and WAIC, providing stable summaries of model performance across varying sample sizes and test lengths.

Prior predictive checks

Prior predictive simulations were conducted to ensure that the chosen priors yielded plausible item response probabilities and asymptote behavior. The person-specific guessing probability was defined as ϑ_n = logit⁻¹ (δ_n), and the response probability followed Equation 13:

Prior predictive checks confirmed that the implied distributions of q_n remained within psychometrically reasonable ranges.

Evaluation criteria

For each condition and parameter type (θ,σ,b,c/g), we computed (Equation 14):

where is the mean posterior estimate in replication r,ψ is the true value, and R = 50 is the number of replications. Bias was defined as Equation 15:

Coverage was the proportion of replications in which the 95% posterior credible interval for ψ contained the true value. Global model fit was assessed using the leave-one-out information criterion (LOOIC) and the widely applicable information criterion (WAIC), computed from the pointwise log-likelihoods using the loo package; lower values indicate better out-of-sample predictive performance.

Model comparison

We computed PSIS-LOO–based stacking weights and pseudo-BMA weights (see Supplementary Appendices 2–4).

Results

A Monte Carlo simulation was conducted to compare the performance of the Three-Parameter Logistic (3PL) model with the Two-Parameter Logistic model incorporating a latent person-specific guessing component (2PLE). The simulation design crossed six sample sizes (N = 100, 150, 200, 300, 500, 1,000) with five test lengths (I = 6, 10, 20, 30, 40), yielding 30 unique conditions. For each condition, 50 replications were generated and evaluated with respect to parameter recovery and predictive performance. Evaluation criteria included root mean square error (RMSE), bias, and coverage for all parameters, as well as two Bayesian model-fit indices: the Leave-One-Out Information Criterion (LOOIC) and the Widely Applicable Information Criterion (WAIC). MCMC diagnostics demonstrated stable estimation across all conditions, with , adequate effective sample sizes, and zero divergent transitions (see Supplementary Appendix 2).

Parameter recovery

For all the considered sample sizes and test lengths, the 2PLE model results in lower RMSE and absolute bias than those obtained for all parameters (θ, a, b and guessing) with the 3PL model. Coverage rates for the 2PLE model were close to nominal in all but one condition (see Supplementary Appendix 2), whereas the 3PL showed systematic under coverage of difficulty and guessing parameters as a result of model misspecification. Importantly, no reversals were observed across the N × test-length design, indicating that the superiority of the 2PLE model is stable rather than condition-specific.

Item discrimination (a)

Across all sample sizes and test lengths, the 2PLE model consistently demonstrated superior recovery of item discrimination parameters relative to the 3PL model (Table 2). RMSE values for a_i were uniformly lower under the 2PLE model. For the smallest design condition (N = 100, I = 6), RMSE for discrimination decreased from 0.347 under the 3PL to 0.290 under the 2PLE model. This pattern was more pronounced with increasing sample size and test length; at N = 500 and I = 6, RMSE values were 0.322 for the 3PL and 0.272 for the 2PLE, respectively, and similar differences were observed across lengthier tests (Figure 2).

FIGURE 2

Patterns of bias also reflected this trend (Table 3). Similarly, the 3PL model was increasingly negatively biased in discrimination estimates with increasing sample size (e.g., −0.090 at N = 500, I = 6), whereas the bias for the 2PLE was small and relatively constant across conditions (e.g., −0.028 at N = 500, I = 6). These findings are consistent with the fact that when heterogeneous guessing behavior is forced into a fixed item-level lower asymptote, as in the 3PL, residual variance ends up being absorbed by the discrimination parameter, leading to systematic distortion. Because the 2PLE model accounts for guessing at the person level, it does bias the discrimination parameters in any way.

Item difficulty (b)

Interestingly, more pronounced differences emerged for the item difficulty parameters. Across conditions, the 3PL model produced large RMSE values for b_i, oftentimes exceeding 1.40, with substantial negative bias (see Tables 2,3). For instance, at N = 500 and I = 6, the RMSE estimate for difficulty was 1.446 under the 3PL model, with a corresponding bias of −1.381. Similar patterns were observed across all test lengths (Figure 3).

FIGURE 3

In contrast, the 2PLE model yielded dramatically lower RMSE values for item difficulty (e.g., 0.274 at N = 500, I = 6) and bias values that were near zero or slightly positive (e.g., 0.008 at N = 500, I = 6). This pattern is theoretically consistent with the simulation design: when guessing varies across individuals but is constrained to be constant at the item level, as in the 3PL, item difficulty absorbs unexplained response variability, resulting in systematic underestimation. Allowing guessing to vary at the person level resolves this confounding and produces substantially more accurate and stable difficulty estimates.

Guessing/lower-asymptote parameters (c or g)

Differences in the recovery of guessing-related parameters were similarly large. The 3PL model’s item-level pseudo-guessing parameter c_i showed consistently high RMSE values (approximately 0.42 across all conditions) and large negative bias (approximately −0.38; Tables 2,3). In contrast, the 2PLE model’s person-level guessing probability g_n was recovered more accurately with RMSE values for g_n ranging between 0.14 and 0.17 across conditions. Bias estimates were consistently around zero for all sample sizes and test lengths. When heterogeneous guessing tendencies are forced into a single item-level constant, as in the 3PL, systematic underestimation appears to be unavoidable.

Latent ability (θ)

Recovery of person ability also favored the 2PLE model across all conditions (Tables 2,3). In the smallest design condition (N = 100, I = 6), RMSE for ability decreased from 0.928 under the 3PL to 0.901 under the 2PLE model. Both models showed improved recovery with increasing sample size and test length; however, the relative advantage of the 2PLE model remained stable and, in some cases, increased slightly for longer tests (Figure 4).

FIGURE 4

Bias estimates for the 2PLE model were consistently close to zero across all conditions (e.g., −0.002 at N = 100, I = 6), whereas the 3PL model exhibited small but systematic bias (e.g., 0.018 at N = 100, I = 6). These results indicate that allowing guessing to vary across individuals improves the recovery of latent ability when heterogeneous guessing behavior is present, even though ability itself was generated independently of guessing propensity.

Model fit and predictive accuracy

Model comparisons were conducted using LOOIC and WAIC, with lower values indicating better expected out-of-sample predictive accuracy. Differences were computed as ΔLOOIC = LOOIC(2PLE) − LOOIC(3PL), such that negative values favor the 2PLE model. To assess the reliability of these differences, we report the standard error of ΔLOOIC, computed from pointwise log-likelihood differences using Pareto-smoothed importance sampling as implemented in the loo package. Following established recommendations, differences exceeding approximately twice their standard error were interpreted as providing meaningful evidence in favor of one model over the other (Table 4). Both indices approximate out-of-sample predictive accuracy, with lower values indicating better performance. Pareto-k diagnostics indicated that PSIS-LOO estimates were reliable across all conditions, with mean k values well below the critical threshold of 0.7 and virtually no observations exceeding this cutoff (Supplementary Appendix 3). Model-averaging weights using stacking and pseudo-BMA showed preference for the 2PLE model across all test lengths (see Supplementary Appendix 4).

The magnitude of the effects was relatively small in absolute terms, typically ranging from 1 to 20 points; however, it was consistent across groups. For instance, in the N = 100, I = 6 condition, LOOIC was reduced from 652.25 under the 3PL to 650.91 under the 2PLE model, and WAIC decreased from 650.63 to 650.00. In the largest sample condition (N = 1000, I = 6), the 2PLE model also yielded lower LOOIC (6443.70 vs. 6445.48) and WAIC (6436.57 vs. 6439.53). This trend was found to hold also for tests of length 10, 20, 30, and 40 items.

The stability of such differences suggests that the superior performance of the 2PLE is not contingent on specific sample sizes or test lengths. Rather, it is an inherent structural advantage due to modeling guessing as a person-level function rather than a fixed item-level constant. Such flexibility enables the 2PLE to model response styles across a wider range of ability and hence increase the out-of-sample predictive performance.

Posterior predictive checks

Posterior predictive p-values (PPCs) were computed for all simulation conditions to evaluate model adequacy (Table 4). Absolute model adequacy was evaluated by posterior predictive p-values (PPCs) for all 30 simulation conditions (Table 4). Posterior predictive p-values were tightly concentrated around 0.74 under all simulation conditions, which meant that there was no systematic bias between observed and reproduced data (Figure 5). While PPC values around 0.50 frequently get highlighted as perfect, PPC over the range 0.70–0.80 are typical of correctly specified hierarchical models and correspond to conservative test statistics rather than model misspecification. The lack of extreme PPC values and steady trends over N or test length constitute compelling evidence for model appropriateness. Together with the smooth monotonic improvement in RMSE and the invariant direction of ΔLOOIC across conditions, these results indicate that the 2PLE model performs robustly across a wide range of realistic testing scenarios.

FIGURE 5

Robustness across design conditions

Taken together, the results demonstrate that the advantages of the 2PLE model are robust across a wide range of realistic testing scenarios. RMSE, bias, LOOIC, WAIC, and PPC values all varied smoothly with increases in sample size and test length and consistently favored the 2PLE model. No reversals or instabilities were observed for any evaluation metric.

Real data example using PIRLS 2019

Models were fitted on data of a reading comprehension scale from PIRLS 2019 and the country Saudi Arabia. PIRLS is an international assessment, which was developed to assess the reading comprehension of fourth-grade students by measuring their ability to construct meaning from written text in literary and informational domains. Items were contextualized within narrative and information passages about animal behavior and survival—a typical PIRLS theme designed to be engaging for young readers, while tapping different levels of comprehension. Each item was scored as correct (1) or incorrect (0) and involved students in responding to explicitly stated information, making simple inferences, or combining local cues with prior knowledge. In the present evaluation participants were 413 4th graders, with 207 being females (50.1%) and 206 males (49.9%). Among them, 357 (86.4%) were of Arabic origin and 56 (13.6%) were international students.

Model fit and predictive comparisons

The reading comprehension subscale was used in the comparison between the 2PLE and 3PL models using as criteria the Widely Applicable Information Criterion (WAIC) and Pareto-smoothed importance sampling leave-one-out cross-validation (PSIS-LOOIC). Across analyses, the two models demonstrated remarkably similar predictive performance [LOOIC_2PLE = 2889.604, S.E._2PLE = 43.263; LOOIC_3PL = 2888.117, S.E._3PL = 43.597; WAIC_2PLE = 2878.797, S.E_2PLE. = 42.986; WAIC_3PL = 2889.029, S.E._3PL = 43.449]. Differences in both WAIC and LOOIC were small in absolute magnitude (Δ < 2 points) and trivial relative to the associated standard errors, indicating that neither model achieved a substantively meaningful advantage in expected out-of-sample prediction.

Person-level versus item-level guessing: substantive considerations

Although differences between the 2PLE and 3PL models were marginal, models differed fundamentally in the location of guessing behavior. Within the 3PL model, guessing is modeled as an item-specific lower asymptote that indirectly accounts for unexpected correct responses from low-ability examinees based on characteristics of individual items (e.g., distractor quality or cueing). In 2PLE, however, guessing is represented as a latent person-level variable that accounts for stable individual differences in the likelihood of guessing across items.

The findings from the 2PLE model showed pronounced individual differences in the person-level guessing probabilities with posterior mean estimates indicating that guessing was not restricted to few items but rather was an important response process at the examinee level. Interestingly, the person-level guessing variance was estimated without compromising overall predictive accuracy relative to the 3PL model. Thereby, the inclusion of a latent guessing trait did not result in overfitting or deterioration of out-of-sample performance evident by similar LOOIC and WAIC values across models.

Rationale for preferring the 2PLE model against the 3PL model

As the predictive fit was empirically equivalent between model comparisons, substantive interpretability and conceptual alignment to the measurement construct, as opposed to incremental changes in information criterion, were used for model selection. Reading comprehension tests like PIRLS do not simply evaluate reading, but individual response tendencies to read complex texts in specific timeframes. Within this context, guessing can reasonably be construed as a person- level response tendency (e.g., with respect to item-specific thresholds as opposed to category-boundaries) that represents inter-individual variation in test- taking strategy, motivation, risk willingness or partial understanding rather than merely an item-level artifact. The 2PLE model directly parameterizes this individual-difference process by estimating posterior distributions of person-level guessing probabilities. Indeed, such an attribute allows the possibility of psychometrically meaningful inferences which cannot be provided within the 3PL model. Thus, the 2PLE model offers a conceptually coherent framework for modeling guessing behavior in PIRLS reading comprehension data. Figure 6 displays the ICCs of the reading comprehension items along with their credible intervals which are notable much more narrow in the case of the 2PLE model.

FIGURE 6

Discussion

The present study examined the consequences of modeling guessing behavior as a latent person-level trait rather than as a fixed item-level constant. We compared the traditional 3PL model and a corresponding variation of the Two-Parameter Logistic Extension with person-specific guessing (2PLE random-effects model) using an extensive Monte Carlo study. Across various sample sizes and test lengths, the results tended to prefer 2PLE model in parameter recovery, bias reduction, and predictive fit. Collectively, these findings demonstrate that when guessing behavior varies systematically across individuals, constraining guessing to the item level induces substantial and predictable distortions in item and ability estimation. These are consistent with prior work documenting limitations of item-level guessing parameters in the presence of heterogeneous examinee behavior which suggest that fixed guessing parameters can distort item difficulty estimates and inflate discrimination when guessing varies across individuals (Reise and Waller, 2009; van der Linden, 2007). Our results corroborate these findings by demonstrating systematic bias in difficulty estimates under the 3PL model when person-level guessing heterogeneity is present. Beyond confirming prior concerns, the present study extends the literature by formally modeling guessing as a person-level latent trait rather than as an item-level nuisance parameter. Whereas earlier approaches either fixed guessing at the item level or treated aberrant responding as a post hoc diagnostic problem, the 2PLE model integrates guessing directly into the measurement model, allowing individual differences in guessing propensity to be estimated simultaneously with ability.

Theoretical implications

Psychometrically, estimating guessing as an individual-level construct is congruent with classical and contemporary measurement models. Guessing is usually subsumed under random error variance in Classical Test Theory with the implicit assumption that its effects are purely unchanging and unsystematic. These results therefore question the assumption that guessing is unsystematic and represents measurement error. From the standpoint of latent trait theory this perspective aligns naturally with the representation of examinee-specific tendencies, such as engagement, risk-taking, or response strategies, as latent variables that interact with ability to produce observed responses.

A central theoretical contribution of this is that guessing should not be considered only an item characteristic or a nuisance parameter. Rather, the simulation data are consistent with guessing being conceptually represented as an individual-level unobservable trait that mediates interaction between ability and response. This view is consistent with research on person-fit and aberrant response behavior which suggests that behaviors such as rapid guessing, disengagement, and impulsive responding often reflect stable individual tendencies rather than random noise (Karabatsos, 2003; Wise and Smith, 2011). From this perspective, the failure of the 3PL model is not only one of estimation but also a psychometric one in that guessing is likely not an item characteristic but a person trait.

Importantly, the results are critical in understanding the misspecification caused by applying heterogeneous guessing at item-level. If guessing varies between persons but is constrained to be the same within items, as it is in the 3PL model, residual response variability is switched systematically to other parameters such as item difficulty and discrimination. This mechanism was evident with large amounts of negative bias and inflated RMSE in the difficulty parameters under the 3PL model across all simulation conditions. Through moving the guessing random variable from items to persons, the 2PLE model separates out these two sources of uncertainty and provides parameter estimates that align more closely with true examinee behavior.

Methodological implications

From a methodological standpoint, this study provides compelling evidence that person-level guessing models can substantially improve both local parameter recovery and global predictive performance. This effect was true across all 30 simulation conditions with the 2PLE model being associated with lower RMSE and bias for item discrimination, item difficulty, guessing-related parameters, and latent ability. Interestingly the effects were consistent even with small sample sizes and test lengths. Differences in global fit (LOOIC and WAIC) were modest in magnitude but consistent in that the 2PLE model outperformed the 3PL model across all conditions. In Monte Carlo simulations, consistent directional advantages across conditions add strength to the superiority of one methodology over another.

Importantly, the benefits of using the 2PLE model were most pronounced when there was substantial person-level guessing heterogeneity—conditions where unmodeled response behavior is most susceptible to biasing parameter estimates. Under these conditions, the 3PL model was consistently biased toward item difficulty and overly inflated uncertainty in ability estimates whereas the bias of the 2PLE model did not grow. These results also indicate that it is especially important to account for examinee engagement in realistic testing conditions and where we should not assume person-level guessing to be equal across all individuals.

Implications for ability estimation

Enhanced recovery of latent ability under the 2PLE model has serious consequences for score interpretation and validity. Although ability and guessing tendencies were independently simulated, the impact of heterogeneous guessing behavior on ability estimation was observed under the 3PL model misspecification. Allowing guessing to vary at the person level, the 2PLE model also improved bias and RMSE in ability estimates across all conditions. This result highlights that accurate ability estimation is not simply a matter of modeling ability alone, but also involves representing the ancillary response processes that produce performance on observed tests.

In practice, these findings indicate that inadequate consideration of individual differences in response to guessing may result in biased proficiency estimates, particularly for examinees with relatively low ability or who demonstrate less attentive modes of responding. In contrast, person-level guessing models provide a principled means to tease apart true ability from response strategies and thus strengthen construct validity.

Applied interpretation of guessing models in PIRLS reading comprehension

The differentiation between the 2PLE and 3PL models has implications for large-scale reading comprehension assessments like PIRLS, where scores are utilized to make inferences at the policy level. Items on the PIRLS reading assessment are constructed to measure comprehension processes from literal recall to inferential reading and integration. In these assessments, false positives and negatives by weaker learners can occur not just through an absence of understanding but for strategic reasons, partial grasp, disengagement or educated guessing in the face of uncertainty. From an applied measurement perspective, the central question is therefore not merely which model yields marginally better global fit, but where the model locates the source of unexpected correct responses. The 3PL model attributes these relations directly to item-specific characteristics, assuming guessing to be a function of distractor quality or the extent of cueing from item presentation. In contrast, the 2PLE model treats guessing as person-level response style by which individual differences in test-taking behavior are more directly modeled within reading ability. In comparing such models in practice, then attention needs to swift from information criteria to which model is more substantively interpretable for the latent construct measured.

In the current analyses, differences in relative fit between the two models were negligible and we believe that the 2PLE model is more interpretable and thus more optimal. The advantage of the 2PLE model over alternative item response models in PIRLS lies in its ability to derive estimation on differences of guessing propensity without reliance on substantive understanding of student response behavior, which is crucial for valid interpretation

Practical implications

The results have straightforward implications for both educational and psychological measurement, particularly in situations where the engagingness of examinees to test items may vary. In standardized testing, professional licensure exams and other low-stakes assessment environments heterogeneous guessing is likely common and may introduce threats to validity due to guessing. The 2PLE model offers a more flexible and theoretically motivated approach to this problem by treating guessing as a person level property rather than an item level attribute.

The relevance of the 2PLE model is not limited to better parameter recovery, however. Person-level guessing estimates might also be of use for detecting when low-ability examinees are responding in ways that deviate from human models of answering behavior (e.g., by guessing very quickly or not at all) in test security contexts to supplement currently available methods. Detaching the guessing effect from ability in computerized adaptive testing can be beneficial for initial-item selection and ability estimation, especially when little information is offered. More generally, by controlling for the confounding of item characteristics and examinee behavior, the 2PLE model enables accurate measurement with fairer inferences from scores in educational and psychological testing.

In addition, the person-level guessing trait as estimated by the 2PLE may have clinical utility beyond enhancing parameter estimation. While the current investigation did not test content interpretations of this parameter, it likely represents aspects like test-taking engagement, impulsivity or response caution. As with other person-fit indices and response-time–based measures, it could augment the utility of existing measures by providing a unique perspective on examinee behavior without needing to rely on additional sources of data.

Limitations and future directions

The present study is subject to several limitations. First, the current formulation models guessing solely as a person-level latent trait. An important extension would be to develop person × item guessing models, in which individual guessing propensities interact with item features such as format, cognitive demand, or number of response options. Such models could further disentangle strategic guessing from item-specific affordances. Second, the guessing as modeled by the present instantiation is only through person-level latent trait. An important extension will be to further model response bias in a person × item format (guessing models) in which one’s guessing propensity is influenced by characteristics of the items, by their level of cognitive demand or number of alternatives that might be guessed. Such models might serve to disentangle strategic guessing from item-specific affordances. Finally, future studies should expand the model to other measurement settings such as low-stakes testing, large-scale assessment and diagnostic testing venues, or adopt joint modeling outcomes that incorporate RTs or engagement. Taken together, these extensions would help to explicate the function of person-level guessing in educational and psychological measurement, and expand the utility of this model.

Conclusion

The current simulation study has shown that modeling guessing as a person-specific latent trait provides large measurement advantages when some individuals guess while others do not. In contrast to the standard 3PL model, the 2PLE random-effects model produced superior recovery of item and person parameters, less bias, and consistently better predictive fit over a wide range of test circumstances. These benefits are not simply a consequence of model nonlinearity but result from more accurate encoding of the process generating responses. Collectively, these results suggest that guessing is not simply a nuisance or deviant behavior to be screened out during the post-hoc analysis but is an important part of the responding process that can—and should—be specifically modeled when individual differences in test-taking are present.

References

• 1

  Birnbaum A. (1968). “Some latent trait models and their use in inferring an examinee’s ability,” in Statistical Theories of Mental Test Scores, edsLordF. M.NovickM. R. (Reading, MA: Addison-Wesley), 397–479.

  • Google Scholar

• 2

  Cizek G. J. Wollack J. A. (2017). Handbook of Quantitative Methods for Detecting Cheating on Tests.New York, NY: Routledge.

  • Google Scholar

• 3

  de Ayala R. J. (2009). The Theory and Practice of Item Response Theory.New York, NY: The Guilford Press.

  • Google Scholar

• 4

  Dienes Z. (2014). Using Bayes to get the most out of non-significant results. Front. Psychol.5:781. 10.3389/fpsyg.2014.00781

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 5

  Downing S. M. Haladyna T. M. (2004). Validity threats: Overcoming interference with proposed interpretations of assessment data. Med. Educ.38, 327–333. 10.1046/j.1365-2923.2004.01777.x

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 6

  Fox J.-P. Marianti S. (2016). Joint modeling of ability and response time with the generalized linear mixed modeling framework.Psychometrika8147–70. 10.1007/s11336-014-9431-1

  • CrossRef
  • Google Scholar

• 7

  Kane M. Moloney J. (1978). The effect of guessing on item reliability under answer-until-correct scoring. Appl. Psychol. Meas.2, 41–49.

  • Google Scholar

• 8

  Karabatsos G. (2003). Comparing the aberrant response detection performance of thirty-six person-fit statistics.Appl. Meas. Educ.16277–298. 10.1207/S15324818AME1604_2

  • CrossRef
  • Google Scholar

• 9

  Lee S. Bolt D. M. (2018). An alternative to the 3PL: using asymmetric item characteristic curves to address guessing effects.J. Educ. Meas.5590–111. 10.1111/jedm.12165

  • CrossRef
  • Google Scholar

• 10

  Lin J. (1991). Divergence measures based on the Shannon entropy.IEEE Trans. Informat. Theory37145–151. 10.1109/18.61115

  • CrossRef
  • Google Scholar

• 11

  Lord F. M. (1980). Applications of Item Response Theory to Practical Testing Problems.Hillsdale, NJ: Lawrence Erlbaum Associates.

  • Google Scholar

• 12

  Muthén L. K. Muthén B. O. (2002). How to use a monte carlo study to decide on sample size and determine power. Struct. Equ. Modeling9, 599–620. 10.1207/S15328007SEM0904_8

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 13

  Reise S. P. Waller N. G. (2009). Item response theory and clinical measurement. Annu. Rev. Clin. Psychol.5, 27–48.

  • Google Scholar

• 14

  Schnipke D. L. Scrams D. J. (2002). “Exploring issues of examinee behavior: Insights gained from response-time analyses,” in Computer-Based Testing, edsMillsC. N.PotenzaM. T.FremerJ. J.WardW. C. (Mahwah, NJ: Lawrence Erlbaum Associates), 237–266.

  • Google Scholar

• 15

  Sinharay S. (2016). An overview of statistics for assessing item and test functioning.J. Educ. Meas.53265–287. 10.1111/jedm.12114

  • CrossRef
  • Google Scholar

• 16

  van der Linden W. J. (2007). A hierarchical framework for modeling speed and accuracy on test items. Psychometrika72, 287–308. 10.1007/s11336-006-1478-z

  • CrossRef
  • Google Scholar

• 17

  van der Linden W. J. Sotaridona L. S. (2006). A statistical test for detecting answer copying on multiple-choice tests.J. Educ. Meas.431–19. 10.1111/j.1745-3984.2006.00001.x

  • CrossRef
  • Google Scholar

• 18

  Wise S. L. DeMars C. E. (2006). An application of item response time: The effort-moderated IRT model. J. Educ. Meas.43, 19–38.

  • Google Scholar

• 19

  Wise S. L. Kong X. (2005). Response time effort: a new measure of examinee motivation in computer-based tests.Appl. Meas. Educ.18163–183. 10.1207/S15324818AME1802_2

  • CrossRef
  • Google Scholar

• 20

  Wise S. L. Smith L. F. (2011). “A model of examinee test-taking effort,” in High-Stakes Testing in Education: Science and Practice in K–12 Settings, edsBovairdJ. A.GeisingerK. F.BuckendahlC. W. (Washington, DC: American Psychological Association), 139–153. 10.1037/12330-009

  • CrossRef
  • Google Scholar

• 21

  Wollack J. A. (1997). A nominal response model approach to detecting answer copying.Appl. Psychol. Meas.21307–320. 10.1177/01466216970214002

  • CrossRef
  • Google Scholar

• 22

  Youxiang J. Xiaofang Y. Yan C. Dongbo T. (2024). A multidimensional IRT model for ability-item-based guessing: the development of a two-parameter logistic extension model.Commun. Stat. Simulat. Comput.533068–3080. 10.1080/03610918.2022.2097694

  • CrossRef
  • Google Scholar

• 23

  Zhu R. Wang C. Tao G. (2019). A two-parameter logistic extension model: An efficient variant of the three-parameter logistic model.Appl. Psychol. Meas.43449–463. 10.1177/0146621618800273

  • Pubmed Abstract
  • CrossRef
  • Google Scholar