AUTHOR=Li Guangming , Pan Yuxi , Wang Weijun 

TITLE=Using Generalizability Theory and Many-Facet Rasch Model to Evaluate In-Basket Tests for Managerial Positions

JOURNAL=Frontiers in Psychology

VOLUME=Volume 12 - 2021

YEAR=2021

URL=https://www.frontiersin.org/journals/psychology/articles/10.3389/fpsyg.2021.660553

DOI=10.3389/fpsyg.2021.660553

ISSN=1664-1078

ABSTRACT=Purpose: This study aimed to analyze an interview data collected from a series of in-basket tests during the managerial personnel recruitment in a local Chinese company, taking advantage of a combination use of Generalizability Theory (GT) and Many-facet Rasch Model (MFRM), rather than the Classical Test Theory (CTT).
Design/methodology/approach: Participants included 78 candidates (Mage = 37.76, SD = 6.36; 35.9% female) interviewed for eight managerial positions in a local Chinese company. Data were collected based on a series of ten in-basket interview tests, and a 20-item rating scale (The In-Basket Tests Rating Scale; IBTRS) was developed and piloted, and five expert raters rated participants on their performance in five aspects (planning, communication and coordination, capital operations and management, analysis and problem-solving, and empowerment and controlling). Data were analyzed using a crossed design of p×i×r, where p represents person, i represents item and r represents rater. Effects of candidate (person), test item, rater, and the interaction of item and rater were examined.
Findings: The combination use of GT and MFRM was able to provide accurate, comprehensive information over the process of in-basket interview tests. Specifically, GT analysis showed good generalization coefficient and reliability index (0.893 and 0.871 respectively), and the variation of candidates explained most of the total variance (53.22%). Candidates were high on the dimension of empowerment and controlling. There were differences in the severity of raters. Three raters should be sufficient to ensure a good scoring stability. 
Originality/value: This study used a combination use of GT and MFRM to assess the interview data instead of using a CTT approach.