IRT studies of many groups: the alignment method

Abstract

Asparouhov and Muthén (2014) presented a new method for multiple-group confirmatory factor analysis (CFA), referred to as the alignment method. The alignment method can be used to estimate group-specific factor means and variances without requiring exact measurement invariance. A strength of the method is the ability to conveniently estimate models for many groups, such as with comparisons of countries. This paper focuses on IRT applications of the alignment method. An empirical investigation is made of binary knowledge items administered in two separate surveys of a set of countries. A Monte Carlo study is presented that shows how the quality of the alignment can be assessed.

1. Introduction

Asparouhov and Muthén (2014) presented a new method for multiple-group confirmatory factor analysis (CFA), referred to as the alignment method. The alignment method can be used to estimate group-specific factor means and variances without requiring exact measurement invariance. A strength of the method is the ability to conveniently estimate models for many groups, such as with comparisons of countries. The method is a valuable alternative to the currently used multiple-group CFA methods for studying measurement invariance that require multiple manual model adjustments guided by modification indices. Multiple-group CFA is not practical with many groups due to poor model fit of the scalar model and too many large modification indices. In contrast, the alignment method is based on the configural model and essentially automates and greatly simplifies measurement invariance analysis. The method also provides a detailed account of parameter invariance for every model parameter in every group.

This paper focuses on IRT applications of the alignment method. An empirical investigation is made of binary knowledge items administered in two separate surveys of a set of countries. A Monte Carlo study is presented that shows how the quality of the alignment can be assessed. Mplus inputs are provided in the Supplementary Material.

2. Multiple-group IRT

Consider the response to item y expressed by the two-parameter logit model for individual i in group g,

where g = 1, …, G and G is the number of groups, i = 1, …, N_g where N_g is the number of independent observations in group g, and η_ig is a latent variable, η_ig ~ N(α_g, ψ_g). Using item response theory (IRT) language, a_g is the discrimination parameter and b_g the difficulty parameter. For a recent overview of IRT for psychologists, see e.g., Reise et al. (2013).

Neither approach is scalable—both are very cumbersome when there are many groups, such as 50 countries (50 × 49/2 = 1225 pairwise comparisons for each item). The correct model may well be far from either of the two starting points, which may lead to the wrong model. Asparouhov and Muthén (2014) proposed a new method referred to as alignment which is suitable for analysis of many groups. The alignment method is based on the idea of starting from the configural model with no invariance and attempting to find as much invariance as possible by letting the factor means and variances vary across groups.

3. The alignment method

Asparouhov and Muthén (2014) considers the model for a continuous item y_ipg,

where p = 1, …, P and P is the number of observed indicator variables, g = 1, …, G and G is the number of groups, i = 1, …, N_g where N_g is the number of independent observations in group g, η_ig is a latent variable and we assume that ε_{ipg ~ N(0, θpg)}, η_ig ~ N(α_g, ψ_g). This expression is relevant also for binary outcomes when letting the dependent variable in (2) be a continuous latent response variable y^*_ipg underlying the observed binary variable y_ipg, where using a threshold parameter τ,

and the variance of the residual ε_ipg is standardized as π²/3 in line with the logistic model (with the alternative probit modeling, the residual variance is standardized as one). Using (2), the IRT parameters of (1) are obtained as

Asparouhov and Muthén (2014) illustrates the reason for the choice of the term alignment for this new method as in Figure 1 using continuous items. Consider group-invariant intercepts and loadings for 10 items and two groups with factor means 0 and −1 and factor variances 1 and 2. The configural model of the first step of alignment fixes the factor means and variances to 0 and 1, respectively, in both groups. The plot at the top shows the configural intercept parameters which due to group differences in factor means and variances are not equal across the two groups despite the perfect measurement invariance of the original parameters. The plot at the bottom shows the invariance across groups of the original parameters where the correct factor means and variances have been taken into account. Going from the top to the bottom plot, the intercept parameters have been aligned.

Figure 1

3.1. The alignment fitting function

Denote the estimates of the configural model by ν_{pg, 0} and λ_{pg, 0}. Asparouhov and Muthén (2014) show that for every set of parameters α_g and ψ_g there are intercept and loading parameters ν_pg and λ_pg that yield the same likelihood as the configural model. These parameters can be obtained as follows

We want to choose α_g and ψ_g so that we minimize the amount of measurement non-invariance. The α_g and ψ_g parameters are, however, not identified in the configural model and are fixed to zero and one, respectively for each group. Adding a simplicity function gives the necessary restrictions to identify the model. The simplicity function minimizes with respect to α_g and ψ_g the total loss/simplicity function F which accumulates the total measurement non-invariance over the items,

The function F implies that for every pair of groups and every intercept and loading parameter we add to the total loss function the difference between the parameters scaled via the component loss function (CLF) f. CLF has been used in EFA analysis, see for example Jennrich (2006) and it is used similarly here. One good choice for the CLF is

where ϵ is a small number such as 0.0001. Thus, the total loss function F will be minimized at a solution where there are a few large non-invariant measurement parameters and many approximately invariant measurement parameters rather than many medium-sized non-invariant measurement parameters. This is similar to the fact that EFA rotation functions aim for either large or small loadings, but not mid-sized loadings.

The alignment method is carried out using maximum-likelihood estimation of the configural model. In addition to the logit model, probit can also be handled. More than one factor can also be accommodated in which case the alignment is done for each factor. Cross-loadings are not, however, allowed. To handle national surveys, the estimation allows complex survey data with stratification, weights, and clustering, where standard errors are computed using the Huber-White sandwich estimator.

Muthén and Asparouhov (2013a) make a comparison of the alignment method and two-level IRT modeling. In the former approach the groups are viewed as a fixed mode of variation, whereas in the latter approach they are viewed as a random mode of variation. A key advantage of the alignment method is that a specific distributional assumption such as normality of the item parameter distributions across groups is not required. For example, a subset of the groups may show large non-invariance, whereas the remaining groups may show little invariance. Information about which groups contribute to non-invariance is also more readily available with the alignment method.

3.2. Alignment quality and degree of non-invariance

The Asparouhov and Muthén (2014) simulation results show that alignment parameter biases increase with increasing degree of measurement non-invariance, decreasing group sample size, and increasing number of groups. For 60 groups, satisfactory results were obtained with groups sizes of 1000 and at most 20% non-invariant measurement parameters. A key issue is the quality of the ranking of groups based on factor means. Monte Carlo simulations in Muthén and Asparouhov (2013a) focused on the correlation between the population factor means and the estimated alignment factor means computed over groups and averaged over replications. Correlations of at least 0.98 were deemed to produce reliable factor mean rankings. Correlations of this magnitude were seen even in cases with higher than 20% non-invariant measurement parameters. As a rough rule of thumb, a limit of 25% non-invariance may be safe for trustworthy alignment results, while with higher percentages a Monte Carlo simulation study is recommended. Such a study is illustrated below.

4. An illustration comparing countries in two cross-sectional surveys

The IEA (International Association for the Evaluation of Educational Achievement) civic knowledge test of 1999 consists of 38 dichotomously scored items. This test, referred to as CIVED, was administered to nearly 90,000 14-year-old students in 28 countries (Torney-Purta et al., 2001; Schultz and Sibberns, 2004). A later survey referred to as ICCS (International Civic and Citizenship Education Study) was carried out in 2009 including 17 link items to make scores comparable to those of 1999 (Schultz et al., 2010). ICCS surveyed over 140,000 eight grade students in 38 countries. 17 countries had comparable national samples and test items and therefore allow comparisons to be made between CIVED achievement and ICCS achievement. Three of these countries had missing data for everyone on at least one of the items at one of the surveys, leaving 14 countries to be compared between the 1999 CIVED and the 2009 ICCS in the current analyses. To further sharpen the comparison, the analyses are restricted to 14-year olds. The IRT alignment analyses to be reported thereby focus on the 17 link items and 29,449 students in 14 countries of CIVED and 10,643 students in 14 countries of ICCS. The 14 countries (country number and country acronym given in parentheses) are: Chile (04; CHL), Colombia (05; COL), Czech Republic (07; CHE), England (09; ENG), Finland (11; FIN), Greece (13; GRC), Italy (16; ITA), Latvia (17; LVA), Norway (19; NOR), Poland (20; POL), Slovak Republic (24; SVK), Slovenia (25; SVN), Sweden (26; SWE), and Switzerland (27; CHE).

Before doing the alignment analysis it is of interest to study measurement invariance using traditional methods, namely comparing the configural, metric, and scalar models (see Muthén and Asparouhov, 2013a). The metric model specifies invariant loadings. The scalar model is of particular interest because it specifies measurement invariance of both thresholds and loadings, a requirement for comparing factor means using traditional methods. Table 1 shows the results for the 1999 CIVED data, the 2009 ICCS data, and the combined data. It is clear that both the metric and the scalar models are rejected by the likelihood-ratio chi-square tests. Part of the reason for this is that the sample sizes are large so there is considerable power to reject invariance. Although criteria such as difference in global fit indices like CFI or RMSEA (Chen, 2007) or detection of local misspecification (Saris et al., 2009) have been proposed to somewhat mitigate this power issue, they are not available with the maximum-likelihood estimation of binary items considered here.

Whatever step-wise non-invariance search method is applied, a large effort is required to find subsets of items that fulfill scalar invariance sufficiently well in different subsets of the groups. The advantage of the alignment method is that metric and scalar invariance are not required. Instead, factor means are made comparable while minimizing measurement non-invariance.

A 14-group alignment analysis of the 17 items is performed for the 14 countries in each of the two surveys, followed by a 28-group alignment analysis of the two surveys jointly. The joint analysis makes it possible to compare factor means and factor variances not only across countries but also across the two surveys. The survey-specific analyses are used to check that the ordering of countries is not largely affected by considering the two surveys together. It was found that the country ordering was almost exactly the same within studies as in the joint 28-group alignment analysis.

The results of the 28-group joint analysis are shown in Tables 2, 3 in factor analysis metric for thresholds and loadings, respectively. The tables indicate which item parameters are non-invariant in which groups by putting groups in parentheses. It is seen that even after alignment many item parameters remain significantly non-invariant in many of the groups. An interesting feature of alignment is that this does not invalidate the alignment method. Thirty three percent of the thresholds and 11% of the loadings are found non-invariant, averaging to 22% non-invariance. Using the 25% rule of thumb mentioned earlier, this implies trustworthy alignment results. To support this conclusion, Monte Carlo simulations reported in Section 5 based on these parameter estimates show that the factor means are well estimated so that a group comparison can be made.

The results in Tables 2, 3 can be augmented by the contributions each item and each group makes to the simplicity function (7). It is of interest to see which items and which groups contribute the most and the least to the non-invariance as quantified by this function. The results can be studied for thresholds and loadings separately or together for an item. It is found that the two least invariant items are items 2 and 9 and the most invariant item is item 4. This largely agrees with the significance findings in Tables 2, 3. Further inspection of these items is therefore warranted. None of the 28 groups stands out as contributing substantially more to the simplicity function, while three groups stand out as contributing the least to the simplicity function: 225 (Slovenia at the second survey), 213 (Greece at the second survey), and 219 (Norway at the second survey).

The aligned factor means are shown in Table 4. The tables also show results of testing for significant factor mean differences between the countries. Figure 2 gives a graphic representation of factor means at the two surveys. It is seen that a majority of the countries decrease in achievement over the 10 years. Exceptions are Finland, the Czech Republic, Sweden, Columbia, and Chile. The variation in the factor means is also diminished such that fewer countries are at the high end on the factor in 2009 as compared to 1999. It is of interest for test developers to investigate if the causes of these features are partly due to testing artifacts. Such an investigation may include studying differences in item order in the testing booklets, different missing data patterns, and different motivation among the students.

Figure 2

5. Monte Carlo investigation

A useful augmentation of the alignment analysis is to carry out a Monte Carlo simulation study to check how well the factor means are captured. Studies may show a large degree of measurement non-invariance, that is, many measurement parameters show large non-invariance in many groups. The concern may then be that the factor means are not well enough estimated to afford a trustworthy comparison across the groups.

The Monte Carlo study can be done using the same features as in the real-data analysis. The features include the degree of measurement non-invariance, the group-varying factor means and variances, the number of items, the number of groups, and the sample sizes in the groups. Such a Monte Carlo analysis is easily carried out using Mplus. The estimated parameters in the real-data alignment analysis can be saved and used for data generation. A large number of replications (random samples of observations) is used. Summary statistics are provided that include the correlation between the generated and estimated factor means for the countries. A near-perfect correlation is required for the ordering of groups with respect to the factors to be trustworthy. Muthén and Asparouhov (2013a) observed that a correlation of at least 0.98 is needed. For the current 28-group analysis a correlation of 0.996 is observed suggesting excellent alignment despite the non-invariance. The parameter values are also well recovered. Mplus input excerpts for both the real-data and Monte Carlo analyses are shown in the Supplementary Material.

6. Conclusions

The alignment method provides a convenient and powerful method to study IRT modeling in many groups. In recent research 92 groups has proved feasible (Munck et al., 2014). With country comparison it is expected that a large degree of non-invariance is present due to cultural and other country differences. Existing methods are simply not practical for handling such complexity. In the current paper maximum-likelihood estimation was used but Bayesian analysis is also available as discussed in Muthén and Asparouhov (2013a). Bayesian analysis also makes it possible to relax the assumptions of the configural IRT model, for example by allowing certain residual correlations among the items. Bayesian analysis also makes it possible to base the alignment on a model with approximate measurement invariance as discussed in Muthén and Asparouhov (2013b).

Future developments of the alignment method for IRT applications include allowing for different booklets administered to different student groups, adding covariates to the alignment method, and the possibility to create plausible values of the factor scores for secondary analyses. These developments should make IRT alignment an even more valuable addition to the IRT methods arsenal.

Conflict of interest statement

The authors are developers of the Mplus software used in the paper. The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

References

• 1

  Asparouhov T. Muthén B. (2014). Multiple-group factor analysis alignment. Struct. Equ. Modeling21, 1–14. 10.1080/10705511.2014.919210

  • CrossRef
  • Google Scholar

• 2

  Chen F. F. (2007). Sensitivity of goodness of fit indexes to lack of measurement invariance. Struct. Equ. Modeling14, 464–504. 10.1080/10705510701301834

  • CrossRef
  • Google Scholar

• 3

  Jennrich R. (2006). Rotation to simple loadings using component loss functions: the oblique case. Psychometrika71, 173–191. 10.1007/s11336-003-1136-B

  • CrossRef
  • Google Scholar

• 4

  Kim E. S. Yoon M. (2011). Testing measurement invariance: a comparison of multiple-group categorical CFA and IRT. Struct. Equ. Modeling18, 212–228. 10.1080/10705511.2011.557337

  • CrossRef
  • Google Scholar

• 5

  Lee J. Little T. D. Preacher K. J. (2010). Methodological issues in using structural equation models for testing differential item functioning, in Cross-Cultural Analysis: Methods and Applications, ed DavidovE.SchmidtP.BillietJ. (New York, NY: Routledge), 55–84.

  • Google Scholar

• 6

  Munck I. Barber C. Torney-Purta J. (2014). Measurement Invariance in International Comparisons of Youth Attitudes Towards Immigrants: the Alignment Method Applied to IEA CIVED 1999 and ICCS 2009. (in press).

  • Google Scholar

• 7

  Muthén B. Asparouhov T. (2013a). New Methods for the Study of Measurement Invariance with Many Groups. Technical report. Available online at: http://statmodel2.com/download/PolAn.pdf.

  • Google Scholar

• 8

  Muthén B. Asparouhov T. (2013b). BSEM Measurement Invariance Analysis. Mplus Web Note 17. Available online at: http://www.statmodel.com/examples/webnotes/webnote17.pdf.

  • Google Scholar

• 9

  Reise S. P. Moore T. M. Haviland M. G. G. (2013). Applying unidimensional item response theory models to psychological data, in APA Handbook of Testing and Assessment in Psychology, Vol. 1: Test Theory and Testing and Assessment in Industrial and Organizational Psychology. APA Handbooks in Psychology, eds KurtF.BrackenB. A.CarlsonJ. F.HansenJ. C.KuncelN. R.ReiseS. P.et al. (Washington, DC: American Psychological Association, xxix), 101–119. 10.1037/14047-006

  • CrossRef
  • Google Scholar

• 10

  Saris W. E. Satorra A. van der Veld W. M. (2009). Testing structural equation models or detection of misspecifications?Struct. Equ. Modeling16, 561–582. 10.1080/10705510903203433

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 11

  Schultz W. Sibberns H. (2004). IEA Civic Education Study Technical Report. Amsterdam: The International Association for the Evaluation of Educational Achievement (IEA).

  • Google Scholar

• 12

  Schultz W. Ainley J. Fraillon J. Kerr D. Losito B. (2010). ICCS 2009 International Report: Civic Knowledge, Attitudes, and Engagement Among Lower-Secondary School Students in 38 Countries. Amsterdam: The International Association for the Evaluation of Educational Achievement (IEA).

  • Google Scholar

• 13

  Stark S. Chernyshenko O. S. Drasgow F. (2006). Detecting differential item functioning with confirmatory factor analysis and item response theory: toward a unified strategy. J. Appl. Psychol. 91, 1292–1306. 10.1037/0021-9010.91.6.1292

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 14

  Thissen D. Steinberg L. Wainer H. (1993). Detection of differential item functioning using the parameters of item response models, in Differential Item Functioning, ed HollandP. W.WainerH. (Hillsdale, NJ: Lawrence Erlbaum), 67–113.

  • Pubmed Abstract
  • Google Scholar

• 15

  Torney-Purta J. Lehman R. Oswald H. Schulz W. (2001). Citizenship and Education in Twenty-Eight Countries. Amsterdam: The International Association for the Evaluation of Educational Achievement (IEA).

  • Google Scholar