Aberrant responses, known as inconsistent or unexpected responses compared to the overall responses in a test, can provide diagnostic information for persons and items (Kogut, 1986; Seol, 1998). Indices developed for detecting a person’s aberrant response patterns were usually labeled as person-fit indices, while for detecting an item’s aberrant responses was called item-fit indices. Some indices are based on the characteristics of a group, and others are based on item response theory (IRT) models (Harnisch and Linn, 1981; Kogut, 1986; Meijer and Sijtsma, 1999). Based on the group-based Guttman (1944) principles (1944) (i.e., able persons should answer those items correctly, which exhibited difficulty levels lower than those persons’ ability levels), the dichotomous BW indicators (Huang, 2002) were designed to detect persons’ aberrant responses patterns, i.e., person-fit oriented, and were subsequently extended to detect aberrant responses of items, i.e., item-fit oriented. The person-fit BW indicators originally (Huang, 2002) measured a person’s tendency of “guess” (B indicator) and “carelessness” (W indicator) and later (Huang and Wu, 2013) extended to measure a person’s tendency of “mastery” (C indicator) and “misconception” (M indicator), respectively. Huang (2011) investigated the robustness of BW aberrance indices against test length and found that the person-fit BW indicators were almost unrelated to test length. Later, Huang (2012) compared the aberrance detection powers among the person-fit BW indicators, other four group-based indicators (SCI, MCI, NCI, and Wc & Bs), and five IRT-based indicators (OUTFITz, INFITz, ECI2z, ECI4z, and lz) under the conditions of content category, type of aberrance, the severity of aberrance, and the ratios of aberrance persons. He found the person-fit BW indicators and the four group-based indices exhibited higher detection rates (over 90%) than the five IRT-based indicators, and furthermore, the BW indicators exhibited the best stability across different situations.

According to symmetrical characteristics, the properties of the item-fit BW indicators are the same as those of the person-fit BW indicators. Specifically, the dichotomous BW item-fit indicators (Huang, 2002) were developed to detect the tendency of “disturbance” (b indicator) and “hint” (w indicator) in an item beyond and within the item’s difficulty level, respectively. In addition to the two aberrant indicators, Huang and Wu (2013) incorporated another two indicators with normal responses for an item, called “power” (c indicator) and “defenselessness” (m indicator) within and beyond the item’s difficulty level, respectively. Here, normal responses mean those responses that obey the Guttman’s principles. In their study, a cognitive diagnostic model based on the WBstar program was used to examine the quality of a teacher-made test (22 items) on the contents of fractions and decimals and found that the BW-based cognitive diagnostic model performed effectively for detecting the misfit of items in small-sample scenario of 32 fourth-grade students.

For applying to a DIF scenario, Huang and Lin (2017) conducted a Monte Carlo study to investigate the predictive effects of the dichotomous BW indicators on DIFs. They simulated data under five conditions (sample size, item number, DIF type, DIF ratio, and DIF severity) and found that the power indicator and the defenselessness indicator could significantly explain the variances of DIFs. Moreover, the four dichotomous BW indicators also exhibited over 90% of the accuracy rate of prediction on the flagged-DIF items. However, Huang and Lin’s study was limited in the dichotomously scoring system and not suitably interpreted in polytomously scoring scenarios. In recognition of limited usage of the dichotomous BW indicators, Huang and Lu (2017) extended these indicators to fit in polytomously scored scenarios, called the Poly-BW indices. In their study, they developed the PWBstar1.0 program¹ for estimating the Poly-BW indices, but they did not investigate the associations between the Poly-BW indices and DIF measures. Therefore, the predictive effectiveness of the Poly-BW indices to DIF measures is unknown.

Similar to the dichotomous situations, the main idea of the Poly-BW indices was based on the concept of a discrepancy, which would reflect the level of aberrant or normal responses. The discrepancy distances between persons’ abilities and an item’s difficulty would be calculated according to normal or aberrant responses beyond and within an item’s difficulty level, respectively. Specifically, suppose in a K-item test participated by N examinees, we can first rank ability levels by individual person scores from the bottom (low) to top (high) and then rank difficulty levels by individual item scores from left (low) to right (high). Let the jth item has a maximum score of s_j and the ith person has an earned score x_ij on the jth item, then x_ij ≤ s_j. Since the N examinees are ranked by individual’s total earned scores on the K-item test from bottom to top, we can define ti=∑j=1Kxij/∑j=1Ksj as the ith person’s potential ability. Then we correspondingly calculate the accumulated maximum score by defining a specified I_j (1 ≤ I_j ≤ N) such that Ij×sj≤Qj=∑i=1N(sj-xij)<(Ij+1)×sj for all N examinees from bottom to top, indicating a certain item score (Q_j) defined as the difference between the maximum score and the earned score is falling between two sequential accumulated maximum scores and thus exhibits the item’s robustness so as for all N examinees not to answer it correctly. Ideally, the ith person who possesses the complete potential ability t_i should gain the accumulated maximum score on the jth item.

Therefore, the true item difficulty (tQj*) for the jth item can be expressed by two sequential persons’ potential ability levels (t_i and t_i+1) through interpolation technique with a 0.5 error unit as Equation 1 shows:

Adding a 0.5 error unit in Equation 1 is because the score unit is 1 and using half of the score unit for correction will be helpful to distinguish difficulty levels when the examinee sample is too small.

With the true item difficulty (tQj*) and based on the dichotomous logic, the Poly-BW indices (cp, mp, bp, and wp) designed to measure the degree of power, defenselessness, disturbance, and, hint for an item, respectively, are given from Equations 2 to 5 as follows:

The properties of the four Poly-BW indices are delineated as follows. First, the power index (cp) refers to the power of an item, that is, the examinees would answer it incorrectly just truly due to their ability is lower than the item difficulty (i.e., within the difficulty level). Second, the defenselessness index (mp) refers to the possible inefficiencies of an item, that is, examinees whose ability levels were higher than the difficulty level of the item (i.e., beyond the difficulty level) would easily answer this item correctly. The cp and mp indices differ from the difficulty index in the classical test theory (CTT). In CTT, the computation of the difficulty index does not distinguish “within” or “beyond” an item difficulty level. In this study, the cp index measures the part of “not correct” responses within the item difficulty, and the mp index measures the part of “correct” responses beyond the item difficulty. An item with a greater value of power index is more likely to be answered “incorrectly” than another item with a lower value of power index. Therefore, when comparing power index between two groups (gender groups), a greater value of power index for a particular item in the focal group (e.g., men) indicates that this item is more likely to be answered incorrectly by male examinees than by female examinees, given those examinees’ ability within this item difficulty. In contrast, an item with a greater value of defenselessness index is more likely to be answered “correctly” than another item with a lower value of defenselessness index. Therefore, when comparing the defenselessness index between two groups (gender groups), a greater value of defenselessness index for a particular item in the focal group (e.g., men) means that this item is more likely to be answered correctly by male examinees than by female examinees, given those examinees’ ability beyond this item difficulty.

On the other hand, the bp and wp indices indicate that an item displays possible aberrances of disturbances or hints that examinees might encounter, respectively. The disturbance index (bp) measures the property of an item with possible pitfalls, that is, examinees answer the item incorrectly even though their ability levels are beyond the difficulty level of the item. An item with a greater value of disturbance index is more likely to be answered “incorrectly” than another item with a lower value of disturbance index. Therefore, when comparing the disturbance index between two groups (gender groups), a greater value of disturbance index for a particular item in the focal group (e.g., men) suggests that this item has more likelihood to be answered wrongly for male examinees than female examinees, even given those examinees’ abilities beyond the item difficulty. In contrast, the hint index (wp) measures the possible prompts in an item, that is, examinees answer the item correctly even though their ability levels are within the difficulty level of this item. An item with a greater value of hint index is more likely to be answered “correctly” than another item with a lower value of hint index. Thus, a greater value of hint index for a particular item in the focal group (e.g., men) delineates that this item has more likelihood to be answered “correctly” for male examinees than female examinees, even given those examinees’ abilities are within the item difficulty.

Figure 1 provides a visualization for the idea of the four Poly-BW indices in different situations of normal and aberrant responses. Ideally, an item is supposed to be answered incorrectly when examinees’ abilities are lower than the item difficulty (i.e., power) and to be answered correctly when examinees’ abilities are higher than the item difficulty (i.e., defenselessness). On the other hand, when the ideal principles of responses are violated, the levels of discrepancy will reflect the levels of aberrance. The disturbance situation occurs when an item is still answered incorrectly even though examinees’ abilities are higher than the item difficulty. The hint situation occurs when an item is still answered correctly even though examinees’ abilities are lower than the item difficulty.

A total of 1,439 seventh-grade students from central Taiwan (51.01% men and 48.99% women) participated in the study. They were selected from 40 classes in eight junior high schools. The students completed a math test in a math class within around 40 min. In Taiwan, all math textbooks are developed based on directions governing the Basic Education Curricula and the contents of math textbooks should be approved by the Ministry of Education. Therefore, all junior-high-school students with the same grade are taught similar contents or units of math. Before administering this math test, all participants were supposed to have learned the contents of arithmetic, algebra, and geometry during their math class. We confirmed that all teachers had taught these contents before giving this math test.

The math test was developed by our research team, i.e., three junior-high-school teachers and two experts in math and psychological measurement. It was a criterion-referenced and formative assessment. The objective of this test was to evaluate what the students had learned and assessed whether students obtained the required knowledge. Therefore, all items were developed according to the directions governing the Basic Education Curricula in Taiwan. The test comprised 34 items involving the three major components of math: 14 items for arithmetic (such as addition/subtraction of integers, multiplication/division of integers, operations on fractions, and the concept of factors/multiples), 16 items for algebra (such as the operation/application of integers and fractions), and 4 items for geometry (such as the concepts of the number line and absolute numbers). Each item required more than two problem-solving steps to solve and was scored on the basis of the number of steps taken by the students (according to the rubric of the math test). Supplementary Appendix A presents the specifications of items by contents and formats. Supplementary Appendix B presents the original teacher-made math test.

The rubric was established on the basis of the rationale of the partial credit model (PCM) (Masters, 1982). According to the PCM, to proceed with the next problem-solving step, the previous step is required to be correct. For instance, in the case of a three-step item, if a student answers the first step correctly but fails in the second step, it is reasonable to assume that he/she cannot answer correctly in the third step. Thus, based on the requirement of the PCM, each correct step, given that the previous steps were correct, was given one point, but a step was not given a point if any previous step was incorrect. Thus, the possible scores for each item in this study, ranging from 0 to 2, 0 to 3, 0 to 4, and 0 to 5, depended on the total number of solution steps and the corresponding previous correct steps for each item. To validate the step scores, two raters were asked to judge the step scores and to add the total scores for all the examinees on each item. The Pearson product-moment correlations were calculated between the two columns of total scores from the two raters for all the examinees on each item and ranged from 0.88 to 0.99, indicating good levels of inter-rater consistency for all items. All intracorrelations between items were significant at 0.01 alpha level and shown in Supplementary Appendix C by the male and female groups, respectively.

The Poly-SIBTEST, a confirmatory and theory-driven approach to detect associates of DIF, was designed to address the problem of DIF identification in polytomously scoring situations (Chang et al., 1996). Poly-SIBTEST is a non-parametric method (i.e., it did not assume the parametric shape of the item response function) and it was a Shealy-Stout multidimensional model (MDM; Shealy and Stout, 1993) used for DIF detection. The MDM model delineates that a DIF is a result of the second construct that is not intended to measure in a test (Shealy and Stout, 1993). Put differently, a DIF item measures more than one construct and the focal group (studied group) have different scores in the second construct. Therefore, according to MDM, a DIF occurs when the reference and focal groups that are matched on the same level of the intended (main) construct have different scores (distributions) on the second construct.

Based on Poly-SIBTEST theory and prior research, when items are hypothesized to have a common secondary construct, they could be bundled together and assessed for differential bundle functioning (DBF). Specifically, the inequality T_jF(θ) < T_jR(θ), where θ represents the measured target ability, T represents the marginal item response functions on item j for the focal group (F) and the reference group (R), respectively. The difference between the subtest response functions gives a preliminary index of DBF, given the examinees’ ability level. If all ability levels are considered, an index of DBF, i.e., Bu, can be estimated. The estimate of Bu can be tested by the standardized statistic, which has an approximately normal distribution with a mean 0 and SD of 1 for a large sample. Items with values greater than 1.96 or less than − 1.96 are deemed as DIF items. In Poly-SIBTEST analysis, we set the female group as the reference group so that the values of Bu greater than 0 indicate male-favoring potential and values of Bu less than 0 indicate female-favoring potential. However, for a DIF-flagged item, only the values of the standardized Bu greater than 1.96 or less than − 1.96 were considered.

Before DIF estimation, a valid matching subset was required previously in the Poly-SIBTEST procedure. There were 18 of 34 items of interest, such as mathematics content (arithmetic, algebra, and geometry), number type (fraction and integer), and item format (operation and word problem), which were classified into five bundles of the studied subtest. The remaining 16 items, after precluding the items in the studied subtest, were used in the matching subtest for the purpose of purification in the first automatic DIF analysis (Stout and Roussos, 1995). After conducting automatic DIF analysis three times and canceling five items that displayed significant DIFs, a valid matching subtest consisting of 11 non-DIF items was found (item 1, 6, 8, 10, 12, 13, 15, 20, 22, 27, and 32). These uncontaminated items were used to identify the ability levels of male and female students in Poly-SIBTEST analysis.

In this study, we set women as the referenced group in both Poly-SIBTEST analysis and Poly-BW differences estimation. A positive DIF measure in Poly-SIBTEST analysis indicated a male-favoring item and a negative DIF measure indicated a female-favoring item. A positive index difference in Poly-BW analysis implied a certain Poly-BW index estimated from the male group greater than that estimated from the female group, and vice versa. Thus, for research question 1, the Poly-BW indices were calculated for the male and female groups separately through the PWBstar1.0 program, and the differences in individual indices between the two groups were used as predictors for the DIF measure (Bu) obtained from Poly-SIBTEST through stepwise multiple regression analysis. For research question 2, the accuracy of classification for a DIF-flagged item by the four Poly-BW indices from the APT procedure was assessed by multiple discriminant analysis.

Specifically, the study used a 100-repetition APT procedure through the following steps: (1) setting three types of cp values (high, middle, and low, using 33% ranks as cutoffs); (2) setting two types of levels (high and low) by using upper or lower the 95% percentile value for each mp, bp, and wp index in the male and female groups, respectively. After these APT steps, all information on the item classifications can be provided by the PWBstar 1.0 program by labeling the types of item classifications as (letter + number)’, in which the letters represent high (H), middle (M), and low (L) power levels of the cp index; while the numbers 1, 2, 3, and 4 refer to normal, disturbance, hint, and hybrid (disturbance plus hint) types of responses, respectively. Further, the prime symbol (′) indicates a significantly high defenselessness value of the mp index. The item classifications provide insights into the associates of DIF. For example, if a DIF item is labeled as L1′ and L2′ for men and women, respectively, it implies that this DIF item displays a low level of power (L) and a high level of defenselessness (′) for both genders, but with normal performances (“1”) for the male groups and aberrant disturbances (“2”) for women. Suspiciously, disturbance might be an associate of DIF.

Descriptive statistics of item level performance and parameters estimated by the Poly-WB item indices formula for men (N = 734) and women (N = 705) were displayed in Supplementary Appendix D. Due to different problem-solving steps required in individual items, the scores of items were ranged from 0 to 2, 0 to 3, 0 to 4, and 0 to 5. As can be seen in Supplementary Appendix D, the mean scores on an item earned by male students ranged from 0.54 (max score 3 in item 29) to 3.48 (max score 5 in item 26) and ranged from 0.78 (max score 2 in item 30) to 3.85 (max score 5 in item 26) by female students. The smallest SDs for both groups occurred on item 19 (0.81 and 0.74 for men and women, respectively) and the largest SDs occurred on item 23 (2.34 and 2.42 for men and women, respectively). Before examining DIFs, the dimensionality of these 34 items was preliminarily checked. Principal components analysis of standardized residuals showed that unexplained variance explained by the main dimension was only 0.8%, indicating this math test exhibited a unidimensionality.

Table 1 summarizes the stepwise multiple regression analysis. Two Poly-BW indices explain almost 78% of the variance (63.8% from the predictor of mp_M–F and 14.2% from the predictor of cp_M–F) of the DIF measures (Bu) obtained from Poly-SIBTEST analysis (F_4,29 = 28.53, p < 0.001). Specifically, the defenselessness (mp) index has the largest contribution (63.8%), followed by the power (cp) index (14.2%). This finding indicates that the defenselessness (mp) and power (cp) indices could be used to predict the gender DIF measures in the math test (β = 0.734, p < 0.001 for mp_M–F; β = − 0.286, p < 0.05 for cp_M–F)². Because the women were regarded as the referenced group in both Poly-SIBTEST analysis and Poly-BW difference estimations, a positive regression coefficient implied that the larger the defenselessness differences (mp_M–F), the larger are the DIF measures (i.e., the items were more likely to favor men when they had a relatively large defenselessness for men). Likewise, a negative regression coefficient indicated that the larger the power differences, the smaller are the DIF measures (i.e., the items were more likely to favor women when they had a relatively large power for men). In summary, the defenselessness (mp) and power (cp) indices were significant predictors of DIF properties in this case. From the definitions of these two indices, these findings indicated that if an item was perceived as more defenseless (easy to be answered correctly beyond the item’s difficulty) or powerful (hard to be answered correctly within the item’s difficulty) by one gender group, then the item was more likely to exhibit a DIF. Interestingly, the abnormal indices (hint wp and disturbance bp) did not contribute significantly to the DIF properties. This finding indicated that these two abnormal indices were less likely to be the possible associates of DIFs in this case.

Furthermore, to demonstrate the Poly-BW indices outperformed the difficulty index of CTT, i.e., P = (P_H + P_L)/2 in terms of DIF variance explained, we compared DIF variance explained from the P index and the Poly-BW indices. Results showed that the CTT difficulty index explained 53.4% variance of DIF measures (F_1,32 = 36.68, p < 0.001), but it was lower than the variance explained by the defenselessness (mp) index (63.8%). Since the difficulty index based on CTT is calculated based on total scores, it confounded the within-difficulty and the beyond-difficulty effect in a person’s responses. Instead, the within-difficulty and beyond-difficulty effects can be distinguished by the power (cp) index and the defenselessness (mp) index, respectively, so that the intrinsic properties in DIFs can be more disclosed.

For research question 2, we examined how accurately the four Poly-BW indices predicted the DIF-flagged items using multiple discriminant analysis. Based on the standardized values of Bu obtained from the Poly-SIBTEST procedure (i.e.,|Bu| > 1.96), 12 items were detected with DIFs: 6 items favoring men (items 2, 5, 7, 11, 14, and 17) and 6 items favoring women (items 4, 18, 19, 21, 29, and 31). The remaining 22 items were neutral to both genders. The average hit rate of the DIFs (see Figure 2) was 82.4%. Without considering the group favored by the DIF items, the DIFs were perfectly (100%) predicted by the four Poly-BW indices. But the neutral category was predicted with an accuracy of 72.7% with six undetermined items. There were five neutral items (items 22, 23, 26, 27, and 28) classified as female-favoring items and one neutral item (item 25) classified as a male-favoring item.

When examining the values of Poly-BW indices (see Table 2), we found that five of six male-favoring items (except item 7) have higher mp values in the male group than in the female group, and all six female-favoring items have higher mp values in the female group than in the male group. For the undetermined items, the six neutral items displayed higher cp values in the male group than in the female group, but different magnitudes of mp values in both groups. All the five female-favoring items displayed higher mp values in the female group than in the male group, but in contrast, the male-favoring item (item 25) displayed a higher mp value in the male group than in the female group (0.094 vs. 0.076). This implies that the mp index (defenselessness) dominates the key reasons of item transformation from no DIF to favoring one group. Other Poly-BW indices values of remaining items can be seen in Supplementary Appendix D.

Furthermore, after excluding the six undetermined items, the remaining 28 items predicted by the defenselessness (mp) and power (cp) indices as male-favoring, female-favoring, and neutral are shown in Figure 3. As can be seen, in terms of the Q values from CTT, men mostly perceived an item as more difficult than women across all items, suggesting that CTT item difficulty cannot explain the possible associates of gender DIFs. Nevertheless, the mp and cp indices provide useful information on the associates of a DIF. First, in the female-favoring part (the left part of Figure 3), most of the female-favoring items exhibited low values of defenselessness (mp) but high values of power (cp) between men and women. Here, the values of defenselessness (mp) and power (cp) were the difference values between the gender groups (where women were the referenced group). Accordingly, when the items had lower defenselessness and higher power for men, they were likely to be female-favoring items. Second, in the male-favoring part (the right part of Figure 3), most of the male-favoring items exhibited high values of defenselessness (mp) but low values of power (cp) between men and women. Thus, when the items had higher defenselessness and lower power for men, they were likely to be male-favoring items.

Furthermore, the item classifications from the APT yielded more insightful information for DIF item treatment. In the female-favoring part, three items (i.e., items 4, 18, and 19) with high values of defenselessness (mp) exhibited consistent item types (L2′, L1′, and L1′, respectively) for both men and women. This implied that men and women consistently perceived the three items as defenseless. Moreover, they consistently perceived item 4 as defenseless and with disturbance (L2′), and items 18 and 19 as normal (L1′). Two other DIF items (items 31 and 21) with relatively high values of power (cp) exhibited consistent item types (M1 and H1, respectively) for both men and women, indicating that the two groups perceived the two items as having middle to high levels of power, respectively. On the other hand, only the item classification of item 29 was different for the gender groups: H3 for women and H1 for men. Thus, item 29 was perceived as having a high level of power for both groups but with a hint only for the female group. The Hint was likely an associate of this DIF item (item 29).

In the male-favoring part, six items exhibited male-favoring properties. Two of them (items 5 and 2) exhibited the same defenselessness item types (L1′) and three of them (items 7, 17, and 14) exhibited the same power item types (M1, H1, and, M1, respectively) for both groups. Thus, these findings implied that men and women consistently perceived the former two items as defenseless but the latter three items as having middle to have high levels of power. Interestingly, only item 11 was labeled as L1′ and M1 for men and women, respectively, suggesting that men perceived this item as more defenseless than women did, which likely resulted in its male-favoring property. In the neutral part, all 15 of 16 items performed consistently across the male and female groups. Only item 13 was labeled as M3 and M1 for men and women, respectively, indicating the item performed middle power for both groups, but men perceived more hints than women. Interestingly, item 13 did not perform DIF. This might be due to middle power performed by the item and, according to Table 1, the variances of DIFs explained by the power (cp) index are low (only 14.2%).

In summary, a 100-repetition APT found only two items (items 29 and 11) to be perceived with different item classifications for men and women among the 12 DIF items.