Automated coding of communication data using large language models: a comparison of hierarchical and direct prompting strategies

Abstract

Coding communication data is essential for assessing 21st-century skills such as collaboration and communication, but large-scale human coding is labor-intensive. Large language models (LLMs) such as ChatGPT offer a scalable alternative, yet their accuracy depends on both coding framework complexity and prompting strategy. Using a communication coding framework with five main categories and seventeen subcategories, we compared two prompting strategies: a hierarchical strategy that first assigns main categories and then codes subcategories, and a direct strategy that directly codes subcategories in a single step. Coding accuracy was evaluated against human coding using Cohen’s Kappa and mixed-effects logistic regression. Both strategies achieved agreement comparable to human–human reliability (overall κ ≈ 0.57–0.59). However, direct prompting consistently outperformed hierarchical prompting, yielding an approximately 18% increase in the odds of agreement. Hierarchical prompting was more susceptible to error propagation when main categories were misclassified, whereas direct prompting produced more stable subcategory coding. These results provide guidance for using LLMs to code communication data under complex coding frameworks.

1 Introduction

Communication and collaboration are widely recognized as critical 21st-century skills, as reflected in large-scale international assessments such as PISA 2015 (OECD, 2017). However, assessing these skills at scale remains a significant challenge. One of the primary barriers is the need to code large volumes of communication data from collaborative tasks to extract evidence of the targeted skills. This coding process is traditionally labor-intensive and difficult to scale, relying on trained human raters to apply a predefined coding framework. While advances in natural language processing and machine learning have enabled automated coding based on human-labeled training data, creating such training sets remains resource-intensive. This constraint limits the feasibility of exploring other coding frameworks.

The emergence of large language models (LLMs), such as ChatGPT, offers a new paradigm for automated data coding. Instead of training a model on large human-annotated datasets, LLMs can be prompted directly with a coding framework to assign codes to raw communication data with minimal human involvement. Previous work (Hao et al., 2025a) has shown that this approach can produce coding outputs comparable to those of human raters, particularly at the level of broad categories. Furthermore, it has been shown that the coding also does not show bias against different demographic groups (Hao et al., 2025b). However, the coding accuracy tends to decline as the complexity of the coding framework increases. For example, one can imagine that a coding framework with 20 categories is generally more challenging to code as compared to the one with two categories.

When human raters work with complex coding frameworks containing many categories, a common strategy is to organize the framework hierarchically. By grouping large numbers of fine-grained categories (sub-categories) into a smaller set of higher-level categories (main categories), the coding task can be broken into manageable layers. This nested structure effectively transforms one large, cognitively demanding task into a sequence of simpler decisions, each involving fewer categories. Such hierarchical coding has long been used to improve consistency, reduce cognitive load, and mitigate errors in human annotation. This raises an important question for automated coding: Can large language models (LLMs) benefit from the same hierarchical strategy when handling complex coding frameworks with many categories? If so, prompting LLMs to reason through coding decisions layer by layer may provide a practical pathway for scaling fine-grained communication coding to large datasets.

Using communication data from three distinct collaborative tasks, this study empirically examines whether ChatGPT can accurately apply a hierarchical coding framework consisting of five main categories and seventeen nested subcategories. Our primary goal is to compare two prompting strategies for automated coding: a hierarchical strategy, in which ChatGPT identifies main categories first and then assigning subcategories in each main category, and a direct strategy, in which ChatGPT is directly prompted to code subcategories. This comparison allows us to evaluate how different prompting approaches influence coding accuracy across levels of granularity. Specifically, this study is guided by the following two research questions:

RQ1: To what extent does ChatGPT-based coding align with human coding at subcategory levels across tasks?

RQ2: Can breaking a complex coding framework into simpler hierarchical layers improve ChatGPT-based coding?

2 Literature review

2.1 Automated coding of communication data to assess 21st-century skills

Twenty-first-century competencies such as collaboration and communication are widely recognized as essential for success in education, the workforce, and daily life (Andrews-Todd and Forsyth, 2020; Graesser et al., 2018; Griffin and Care, 2015; OECD, 2017). These skills are multidimensional, dynamic, and process-oriented (Andrews-Todd et al., 2023; Jiang et al., 2025; Liu et al., 2016), making them difficult to measure through traditional assessments that focus primarily on final products (e.g., responses to multiple-choice items; Hao et al., 2017b). As a result, researchers rely on communication data such as chat logs and discourse moves to infer the dynamic processes through which collaboration and communication unfold (Andrews-Todd and Kerr, 2019; Fiore et al., 2017). Such data offer rich evidence about how individuals share information, negotiate meaning, coordinate actions, and co-construct solutions within teams, processes fundamental for the assessment of collaboration skills (Hao et al., 2019).

Traditionally, the analysis of communication data has relied on manual qualitative coding that classify communication into meaningful categories based on theory-driven coding frameworks (Andrews-Todd et al., 2023; Andrews-Todd and Kerr, 2019; Kyllonen et al., 2023; Liu et al., 2016;). Despite its interpretive value, manual coding is time-consuming, labor-intensive, and difficult to scale, especially for large datasets involving a massive volume of communication data (Campbell et al., 2013). Human raters require extensive training and calibration, and reliability must be continuously monitored, creating barriers for large-scale assessment and for refining or expanding coding frameworks.

To address these challenges, researchers have increasingly applied machine learning and natural language processing (NLP) techniques to automate classification of communication data and improve efficiency. Early work using feature-based machine learning models, such as n-gram lexical features or bag-of-words representations, showed that these models can identify collaboration behaviors with modest accuracy (Flor et al., 2016; Flor and Andrews-Todd, 2022; Hao et al., 2017a; Zhang et al., 2024). Later deep learning-based approaches, such as transformer-based models like BERT, demonstrated higher accuracy and greater cross-task generalizability compared to n-gram based classifiers (Pugh et al., 2021), suggesting that embedding-based representations capture richer semantic information than traditional lexical features. Nevertheless, supervised machine learning approaches still require substantial amounts of high-quality, human-coded training data, often comprising 10–25% of the full dataset (Campbell et al., 2013). This annotation requirement remains a major bottleneck for scaling automated coding, especially for frameworks with many fine-grained categories.

2.2 Large language models for qualitative coding

The emergence of large language models (LLMs) has introduced new opportunities for automating the qualitative coding of communication data. Unlike traditional supervised models, LLMs can be prompted directly with coding frameworks to assign labels without large pre-labeled training sets (Gilardi et al., 2023; Morgan, 2023; Zambrano et al., 2023). Beyond applying predefined codebooks, LLMs have been used to assist with inductive coding and codebook development, generating candidate codes, themes, and analytic structures from raw text (Barany et al., 2024). Hybrid human–AI workflows have also been proposed to integrate LLMs into collaborative qualitative analysis, lowering barriers to participation while preserving methodological rigor (Gao et al., 2024; Li et al., 2025). Related work has examined multi-agent LLM systems for automating thematic analysis (Borchers et al., 2025; Sankaranarayanan et al., 2025) and compared the consistency of coding produced by humans and multi-agent systems (Simon et al., 2025).

Growing evidence shows that LLMs can produce coding accuracy that is comparable to or in some cases surpasses human raters for many types of discourse coding (e.g., Hao et al., 2025a; Hou et al., 2024; Jiang et al., 2025; Liu et al., 2025; Long et al., 2024; Na and Feng, 2025; Peczuh et al., 2025; Wellert et al., 2025; Xiao et al., 2023; Zhang et al., 2024). For example, Long et al. (2024) found that ChatGPT could classify classroom dialogues with near-human accuracy while reducing annotation time from hours to minutes. Similarly, Hao et al. (2025a) and Jiang et al. (2025) demonstrated that GPT-4o could reliably code communication across diverse types of tasks (e.g., problem-solving, decision-making, negotiation, and science simulation tasks) and coding frameworks, producing agreement levels similar to trained human raters.

LLM performance on coding varies by the complexity and design of coding frameworks and the nature of the tasks (Hao et al., 2025a; Koco’n et al., 2023). Unlike fields such as computer vision or speech recognition, automated coding of communication data using LLMs has not yet converged on shared benchmark datasets, standardized coding frameworks, or uniform evaluation protocols. Existing studies typically evaluate LLM performance within task-specific coding schemes using distinct prompt designs (Liu et al., 2025), making direct quantitative comparison across studies infeasible. As a result, reported accuracy metrics reflect performance relative to a particular coding framework and task rather than a universal state-of-the-art baseline.

Recent work further highlights how prompt design, task properties, and construct characteristics interact to shape LLM performance in qualitative coding (Borchers et al., 2025; Liu et al., 2025; McClure et al., 2024). In a multi-dataset evaluation, Liu et al. (2025) showed that LLM performance decreases for abstract or conceptually nuanced constructs, and that model performance is sensitive to prompt specificity (e.g., detailed instructions, examples, contextual information). For example, few-shot prompts are more effective for constructs that are difficult to define, while providing contextual information benefits constructs requiring local interpretation and surrounding context. This study underscores the need for systematic exploration of prompt engineering strategies to optimize coding quality (e.g., apply prompt chaining to reduce the load on LLMs for complex coding frameworks).

2.3 Hierarchical coding frameworks for human and automated annotation

Human raters frequently rely on hierarchical coding frameworks when working with coding schemes that contain many fine-grained categories or conceptually nuanced distinctions. Hierarchical frameworks organize codes into higher-level main categories and nested subcategories, which reduces cognitive load, enhances consistency, and facilitates high inter-rater reliability (Andrews-Todd and Kerr, 2019; Campbell et al., 2013; Liu et al., 2016). This organization aligns with cognitive load theory, which posits that decomposing complex tasks into sequential sub-decisions at each layer mitigates the burden on working memory (Sweller, 1988). Hierarchical coding is widely used in research such as collaborative problem solving (CPS) (OECD, 2017) and classroom dialogue analysis (Song et al., 2020; Tao and Chen, 2023). The advantages of hierarchical organization in human coding raise an important question: Do LLMs benefit from similar hierarchical structure when coding complex communication data?

Emerging LLM research suggests that multi-step prompting strategies, such as chain-of-thought prompting and prompt chaining, can improve model reasoning by breaking tasks into smaller, structured steps (McClure et al., 2024; OpenAI, 2024). For example, Na and Feng (2025) instructed LLMs to reason step by step when coding two constructs (communicative acts and communicative events) through the chain-of-thought technique. They found that separating the coding of communicative events and communicative acts into two prompts led to better performance over a single-shot prompt where LLMs code both constructs simultaneously. This finding highlights how prompt decomposition and prompt chaining can reduce cognitive load for LLMs and enhance coding quality.

However, limited empirical research has closely compared whether LLM-based coding with a hierarchically structured prompt can effectively outperform a direct prompt in complex communication contexts. The present study addresses this gap by comparing two strategies: (1) Hierarchical approach, in which the model identifies a main category before selecting a subcategory, mirroring human annotation workflows, (2) Direct approach, in which the model directly assigns a fine-grained subcategory without relying on higher-level predictions.

3 Methods

This section presents detailed information on the study design, collaborative tasks and data, coding framework, LLMs, and prompt design.

3.1 Study design

To address the two research questions, we chose communication data from three different types of collaborative tasks. Two trained human raters independently applied the coding framework (see next subsection) to assign both main-category and subcategory codes to each chat turn. A final human rater for each chat turn was then determined based on the two raters’ consensus; if the raters disagreed, the code was adjudicated to create a final code. This final human code served as the reference standard for evaluating coding agreement between the human raters and ChatGPT.

We instruct ChatGPT to code the data using two prompting strategies: hierarchical and direct prompting, as introduced earlier. For the direct approach, we prompt the ChatGPT to directly code the subcategories. For the hierarchical strategy, we further split into two different implementations: the single-step and two-step hierarchical approaches. In the single-step hierarchical approach, ChatGPT received one comprehensive prompt containing definitions for all main categories and their respective subcategories. The ChatGPT was explicitly instructed to first decide on the main category and then assign subcategories in a single pass, resulting in two labels per chat turn.

In the two-step hierarchical approach, ChatGPT was prompted in two separate rounds, rather than receiving a single combined prompt. In the first round, only main categories are included and ChatGPT was prompted to assign a main category to each chat turn, generating an initial coded dataset. In the second round, and for each main category, ChatGPT was prompted to assign subcategories to the corresponding subset of chat turns (i.e., individual messages or utterances within a multi-turn dialogue). This sequential procedure was designed to decompose the coding task into smaller, more focused steps, potentially making the overall process easier for the model to manage.

3.2 Datasets

The dataset used in this study was collected from three CPS tasks focusing on Decision-Making, Negotiation, and Letter-to-Number, respectively (Kyllonen et al., 2023). In each task, four participants formed a team and collaborated online through text-based chat to complete the assigned activity. In the Decision-Making task, team members discussed and selected the best apartment from several candidates. In the Negotiation task, team members worked together to reach a mutually agreed solution for planning a fundraising event. The Letter-to-Number task, adapted from a classic reasoning paradigm (Newell and Simon, 1972), required participants to jointly uncover the hidden correspondence between letters and numbers. Each task took approximately 40 min to complete and generated, on average, between 50 and 100 chat turns. More details of the tasks can be found in Hao et al. (2025a). Data for these three tasks were collected through Prolific¹ and reviewed and approved by the Institutional Review Board (IRB) of Educational Testing Service (ETS). For this study, random samples of completed sessions were drawn from each task, resulting in roughly 2,500–3,000 chat turns per task. Descriptive statistics of session and chat level characteristics are presented in Table 1.

3.3 Coding framework

The coding framework consists of five main categories, each containing two to five subcategories, for a total of 17 subcategories (Kyllonen et al., 2023; Hao et al., 2025a). Table 2 presents the coding categories and definitions for the hierarchical coding framework.

3.4 LLMs and prompt design

In this study, we used OpenAI’s GPT-4o model (version: 2024-05-13), deployed via the Azure cloud platform, as the underlying large language model for ChatGPT-based coding. The temperature parameter was set to 0, and a fixed random seed was used to ensure consistency and reproducibility across coding runs. GPT-4o represents the final generation of OpenAI’s general-purpose LLMs prior to the introduction of models explicitly optimized for multi-step reasoning, such as those in the GPT-o series and GPT-5. In prior work (Hao et al., 2025a), we systematically compared the coding performance of GPT-4o with that of reasoning-oriented models released through 2025, including GPT-o3, and found that GPT-4o exhibited strong and stable coding performance. Notably, the reasoning-oriented models did not outperform GPT-4o in coding accuracy and, in some cases, showed inferior performance. Accordingly, we chose GPT-4o in the present study. We note, however, that future advances toward artificial general intelligence may alter the relative performance of different model architectures, and we will continue to monitor such developments in future work.

We followed the best practices recommended by OpenAI (n.d.) to develop prompts for both hierarchical and direct coding strategies. To guide ChatGPT in applying the hierarchical coding framework, we designed structured prompts that clearly specified the task goal and the expected output format. Each prompt also contained a description of the coding categories, and roughly 10 expert-created examples (few shots). For both strategies, the same general template was used, but task-specific examples were customized to reflect the context of each collaborative task. Example prompts for the different strategies are included in the Appendix.

3.5 Analysis

We conducted two complementary statistical analyses to address the two research questions. First, we computed Cohen’s kappa (Cohen, 1960) to quantify agreement between ChatGPT-based coding and human coding, as well as the agreement between human raters, both within individual collaborative tasks and aggregated across all tasks. This descriptive analysis provides an overall evaluation of agreement patterns across coding strategies but does not support formal statistical inference regarding differences among strategies.

To address the second research question, whether differences in coding accuracy across prompting strategies were statistically significant, we fitted mixed-effects logistic regression models to predict the probability that a ChatGPT-generated code agreed with the corresponding human code. Agreement was treated as a binary outcome (1 = agreement, 0 = disagreement), with prompting strategy and task included as categorical fixed effects. To account for the nested and repeated-measures structure of the data, random intercepts were specified for person and chat turn. This modeling choice reflects that multiple responses were contributed by the same individual and that the same response content was evaluated under multiple prompting strategies. Including a turn-level random intercept accounts for shared, unobserved response difficulty and prevents pseudo-replication when comparing strategies on identical content.

Two models were fitted. The first model (M1) estimated overall differences among prompting strategies aggregated across tasks:

The second model (M2) extended M1 by adding a prompting-strategy-by-task interaction to examine whether the relative performance of prompting strategies varied by task:

In both models, the two-step hierarchical prompting strategy and the Decision-Making task served as reference categories. All models were estimated using maximum likelihood with a logit link via the glmer function in the lme4 package in R (Bates et al., 2015). Complete model outputs, including coefficient estimates and diagnostic information, are provided in the Appendix.

4 Results

To address the research question 1, Table 3 summarizes subcategory-level coding agreement, measured by Cohen’s Kappa, between human raters and different coding approaches. Across tasks, the Kappa values exhibit a consistent descriptive pattern in which direct prompting yields the highest agreement with human coding among the three ChatGPT-based strategies. This advantage is evident both in the aggregated results (κ = 0.591) and within each individual task, where direct prompting either matches or exceeds the hierarchical variants. In contrast, the two-step and one-step hierarchical strategies exhibited nearly identical overall levels of agreement (κ = 0.573 and 0.574, respectively). At the task level, the two-step approach performed worse than the one-step approach in the Decision Making and Negotiation tasks but much better in the Letter to Number task, indicating no consistent advantage of either hierarchical variant.

Overall, agreement between ChatGPT-based and human coding is comparable to human–human agreement, with the direct prompting strategy yielding agreement levels closest to human inter-rater reliability across all tasks. Because Cohen’s Kappa provides a summary of agreement rather than a statistical test, these results should be interpreted as indicating overall performance tendencies.

To address the research question 2 with statistical rigor, we used mixed-effects logistic regression to examine the statistical significance of the differences among prompting strategies. In the main-effects model aggregated across tasks, the direct prompting strategy showed statistically significantly higher agreement with human coding than the two-step hierarchical strategy (log-odds = 0.162, z = 3.41, p < 0.001), corresponding to an odds ratio of 1.18 (≈18% higher odds of agreement). The single-step hierarchical strategy did not differ from the two-step hierarchical strategy (log-odds = 0.001, p = 0.98).

When allowing prompting strategy effects to vary by task, we got a modest improvement in model fit (ΔAIC = 16.7), indicating limited task-specific heterogeneity. Most prompting-strategy and task interaction terms were not statistically significant, suggesting that the relative performance of prompting strategies was largely stable across tasks. The sole exception was the Letter-to-Number task, in which the single-step hierarchical strategy showed a significant decline in agreement relative to its performance in the reference task (log-odds = −0.445, p < 0.001). In contrast, the corresponding interaction for the direct prompting strategy in Task P was not significant (p = 0.41), and no within-task contrast revealed a significant difference between direct and two-step hierarchical prompting. Overall, the pattern of results indicates that the observed advantage of direct prompting is best characterized as a general main effect rather than evidence of task-specific differences. The limited detection of task-specific differences may be attributable to modest effect sizes.

It is worth noting that in the two-step hierarchical approach, coding errors during the initial main-category coding stage can propagate to the subsequent subcategory assignments. To illustrate this error propagation, Figure 1 presents the confusion matrix for main-category coding under the two-step hierarchical strategy.

Figure 1

5 Discussion

This study addressed two research questions concerning the use of ChatGPT to code communication data using a complex, multi-category coding framework. The first question examined the extent to which ChatGPT-based coding aligns with human coding when the number of categories increases substantially. Prior research has shown that ChatGPT can achieve agreement comparable to human raters when coding frameworks involve a small number of categories. The present findings extend this work by demonstrating that comparable performance can be maintained even when the coding framework expands to 17 subcategories, which require more fine-grained interpretive judgment.

As expected, increasing the number of categories reduced agreement for both human–human and human–ChatGPT comparisons. Human–human agreement ranged from 0.446 to 0.637 across tasks, while ChatGPT-based agreement ranged from 0.508 to 0.622, depending on prompting strategy. In the Decision-Making and Letter-to-Number tasks, ChatGPT’s agreement closely matched that observed between human raters. Notably, in the Negotiation task where human raters exhibited lower consistency, ChatGPT demonstrated comparatively higher agreement, suggesting that AI-based coding may offer more stable performance in contexts characterized by complex or ambiguous discourse.

The second research question examined whether hierarchical decomposition of a complex coding framework improves ChatGPT-based coding. The results indicate that hierarchical prompting does not confer the same benefits for ChatGPT as it does for human raters. For humans, reducing the number of categories through higher-level groupings can lower cognitive load and improve reliability. In contrast, for ChatGPT, hierarchical decomposition introduced additional sources of error. In particular, the two-step hierarchical strategy was vulnerable to error propagation: misclassifications at the main-category level (e.g., see Figure 1) directly constrained and distorted subsequent subcategory assignments. As a result, breaking the task into sequential hierarchical steps reduced, rather than improved, agreement at the subcategory level.

This finding contrasts with prior work showing benefits of prompt decomposition for LLM-based coding (e.g., Na and Feng, 2025). However, in those studies the decomposed constructs were independent rather than hierarchically nested. The present results suggest that while prompt chaining can be effective in some settings, hierarchical frameworks with dependent decision stages may undermine performance by amplifying early errors. Thus, the effectiveness of prompt decomposition appears to depend critically on the structural relationships among categories.

Across all tasks, the direct prompting strategy yielded the strongest subcategory-level performance. By coding subcategories directly, ChatGPT was able to capture nuanced discourse functions without being constrained by an initial high-level classification. The single-step hierarchical strategy performed reasonably well but did not outperform direct prompting, indicating that explicitly presenting the hierarchy in a single prompt does not improve accuracy relative to working directly at the fine-grained level. Taken together, these findings suggest that hierarchical decomposition is not inherently beneficial for ChatGPT-based coding and may be detrimental when category dependencies are strong.

Finally, consistent with prior work, coding agreement varied systematically across tasks at subcategory level too. Communication in the Negotiation task proved more challenging for both human raters and ChatGPT than communication in the Decision-Making and Letter-to-Number tasks. The present study extends this pattern to fine-grained subcategory coding, indicating that increased specificity in category definitions does not fully offset task-related differences in discourse structure and interaction dynamics. These results underscore the importance of task characteristics in shaping coding reliability and highlight the need for careful validation of both human and AI-based coding systems across diverse interaction contexts.

This study has several important limitations. First, the findings are based exclusively on the GPT-4o model. Although our prior work found it to be the best performer among the GPT family for communication-data coding, this does not preclude the emergence of more capable models in the future. As AI technology continues to evolve rapidly, the results presented here must be interpreted in the context of their time. Second, our results showed that in the two-step hierarchical prompting approach, errors made at the main-category level tended to propagate to the subcategory level, ultimately reducing overall coding accuracy. This limitation reflects the specific prompting strategies we examined rather than an inherent constraint of LLM-based coding. It is conceivable that more sophisticated prompting designs or multi-stage reasoning procedures could mitigate or prevent such error propagation. Future research may identify prompting schemes that better insulate subcategory decisions from upstream misclassifications.

Finally, the study examined only three collaborative tasks and one specific hierarchical coding framework. Although these tasks differ meaningfully in interactional characteristics, they do not encompass the full range of communication patterns found in educational or collaborative settings. Future work should test the generalizability of these findings across a broader set of tasks, alternative frameworks, and different domains of discourse. Additional research incorporating diverse datasets and coding schemes will help clarify the extent to which the observed patterns hold and whether the advantages of bottom-up prompting extend to other contexts.

6 Conclusion

This study examined whether ChatGPT can reliably code communication data at a fine-grained subcategory level and whether hierarchical decomposition of a complex coding framework improves coding performance. Extending prior work at the main-category level, the results show that ChatGPT achieves agreement comparable to trained human raters even when coding requires more detailed and nuanced distinctions. In contrast, hierarchical decomposition did not improve performance: the two-step hierarchical prompting strategy was susceptible to error propagation, and the single-step hierarchical prompting offered no clear advantage over direct prompting. Together, these findings contribute to the field by demonstrating the feasibility of scalable, AI-based coding for complex communication frameworks, and they offer practical guidance for researchers and practitioners by indicating that direct prompting is a more effective and reliable approach than hierarchical prompting strategies.

References

• 1

  Andrews-ToddJ.ForsythC. M. (2020). Exploring social and cognitive dimensions of collaborative problem solving in an open online simulation-based task. Comput. Human Behav.104:105759. doi: 10.1016/j.chb.2018.10.025

  • CrossRef
  • Google Scholar

• 2

  Andrews-ToddJ.JiangY.SteinbergJ.PughS. L.D’MelloS. K. (2023). Investigating collaborative problem solving skills and outcomes across computer-based tasks. Comput. Educ.207:104928. doi: 10.1016/j.compedu.2023.104928

  • CrossRef
  • Google Scholar

• 3

  Andrews-ToddJ.KerrD. (2019). Application of ontologies for assessing collaborative problem-solving skills. Int. J. Test.19, 172–187. doi: 10.1080/15305058.2019.1573823

  • CrossRef
  • Google Scholar

• 4

  BaranyA.NasiarN.PorterC.ZambranoA. F.AndresA. L.BrightD.et al. (2024). “ChatGPT for education research: exploring the potential of large language models for qualitative codebook development” in International conference on artificial intelligence in education. eds. ChenL.ChenP. (Cham: Springer Nature Switzerland), 134–149.

  • Google Scholar

• 5

  BatesD.MaechlerM.BolkerB.WalkerS.ChristensenR. H. B.SingmannH.et al. (2015). Package ‘lme4’. Convergence12:2. Available online at: http://dk.archive.ubuntu.com/pub/pub/cran/web/packages/lme4/lme4.pdf

  • Google Scholar

• 6

  BorchersC.ShahrokhianB.BalzanF.TajikE.SankaranarayananS.SimonS. (2025). Temperature and persona shape llm agent consensus with minimal accuracy gains in qualitative coding. arXiv preprint arXiv. doi: 10.48550/arXiv.2507.11198

  • CrossRef
  • Google Scholar

• 7

  CampbellJ. L.QuincyC.OssermanJ.PedersenO. K. (2013). Coding in-depth semistructured interviews: problems of unitization and intercoder reliability and agreement. Sociol. Methods Res.42, 294–320. doi: 10.1177/0049124113500475

  • CrossRef
  • Google Scholar

• 8

  CohenJ. (1960). A coefficient of agreement for nominal scales. Educ. Psychol. Meas.20, 37–46. doi: 10.1177/001316446002000104

  • CrossRef
  • Google Scholar

• 9

  FioreS. M.GraesserA.GreiffS.GriffinP.GongB. (2017). Collaborative problem solving: Considerations for the national assessment of educational progress.

  • Google Scholar

• 10

  FlorM.Andrews-ToddJ. (2022). Towards automatic annotation of collaborative problem-solving skills in technology-enhanced environments. J. Comput. Assist. Learn.38, 1434–1447. doi: 10.1111/jcal.12689

  • CrossRef
  • Google Scholar

• 11

  FlorM.YoonS. Y.HaoJ.LiuL.von DavierA. (2016). “Automated classification of collaborative problem-solving interactions in simulated science tasks,” in Proceedings of the 11th workshop on innovative use of NLP for building educational applications (31–41).

  • Google Scholar

• 12

  GaoJ.GuoY.LimG.ZhangT.ZhangZ.LiT. J. J. (2024). “CollabCoder: a lower-barrier, rigorous workflow for inductive collaborative qualitative analysis with large language models,” in Proceedings of the 2024 CHI conference on human factors in computing systems (1–29).

  • Google Scholar

• 13

  GilardiF.AlizadehM.KubliM. (2023). ChatGPT outperforms crowd workers for text-annotation tasks. Proc. Natl. Acad. Sci. USA120:e2312268120. doi: 10.1073/pnas.2305016120

  • CrossRef
  • Google Scholar

• 14

  GraesserA. C.FioreS. M.GreiffS.Andrews-ToddJ.FoltzP. W.HesseF. W. (2018). Advancing the science of collaborative problem solving. Psychol. Sci. Public Interest19, 59–92. doi: 10.1177/1529100618808244,

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 15

  GriffinP.CareE. (2015). Assessment and teaching of 21st century skills: Methods and approach. Cham: Springer.

  • Google Scholar

• 16

  HaoJ.ChenL.FlorM.LiuL.von DavierA. A. (2017a). CPS-rater: automated sequential annotation for conversations in collaborative problem-solving activities. ETS Res. Rep. Ser.2017, 1–9. doi: 10.1002/ets2.12184

  • CrossRef
  • Google Scholar

• 17

  HaoJ.CuiW.KyllonenP.KerzabiE. (2025b). Can ChatGPT code communication data fairly?: empirical evidence from multiple collaborative tasks. arXiv preprint arXiv. doi: 10.48550/arXiv.2510.20584

  • CrossRef
  • Google Scholar

• 18

  HaoJ.CuiW.KyllonenP.KerzabiE.LiuL.FlorM. (2025a). Automated coding of communications in collaborative problem-solving tasks using ChatGPT. J. Educ. Meas. 62, 809–837. doi: 10.48550/arXiv.2411.10246

  • CrossRef
  • Google Scholar

• 19

  HaoJ.LiuL.KyllonenP.FlorM.von DavierA. A. (2019). Psychometric considerations and a general scoring strategy for assessments of collaborative problem solving. ETS Res. Rep. Ser.2019:12276. doi: 10.1002/ets2.12276

  • CrossRef
  • Google Scholar

• 20

  HaoJ.LiuL.von DavierA. A.KyllonenP. C. (2017b). Initial steps towards a standardized assessment for collaborative problem solving (CPS): practical challenges and strategies. Innov. Assessment Collab.22, 135–156.

  • Google Scholar

• 21

  HouC.ZhuG.ZhengJ.ZhangL.HuangX.ZhongT. (2024). “Prompt-based and fine-tuned GPT models for context-dependent and-independent deductive coding in social annotation,” in Proceedings of the 14th learning analytics and knowledge conference (518–528).

  • Google Scholar

• 22

  JiangY.HaoJ.CuiW.KerzabiE.KyllonenP. (2025). “Uncovering transferable collaboration patterns across tasks using large language models,” in Proceedings of the 26th international conference on artificial intelligence in education (AIED 2025) (320–335).

  • Google Scholar

• 23

  Koco’nJ.CicheckiI.KaszycaO.KochanekM.SzydłoD.BaranJ.et al. (2023). ChatGPT: jack of all trades, master of none. Inf. Fusion99:101861. doi: 10.1016/j.inffus.2023.101861

  • CrossRef
  • Google Scholar

• 24

  KyllonenP.HaoJ.WeeksJ.KerzabiE.WangY.LawlessR., (2023). “Assessing individual contribution to teamwork: design and findings,” in Presentation given at NCME 2023, Chicago, IL, USA.

  • Google Scholar

• 25

  LiZ.TangL.XiaM.LiX.ChenN.GaševićD.et al. (2025). When LLMs fall short in deductive coding: model comparison and human AI collaboration workflow design. arXiv preprint arXiv. doi: 10.48550/arXiv.2512.21041

  • CrossRef
  • Google Scholar

• 26

  LiuL.HaoJ.von DavierA. A.KyllonenP.Zapata-RiveraJ. D. (2016). “A tough nut to crack: measuring collaborative problem solving” in Handbook of research on technology tools for real-world skill development. eds. RosenY.FerraraS.MosharrafM. (London: IGI Global Scientific Publishing), 344–359.

  • Google Scholar

• 27

  LiuX.ZambranoA. F.BakerR. S.BaranyA.OcumpaughJ.ZhangJ.et al. (2025). Qualitative coding with GPT-4: where it works better. J. Learn. Anal.12, 169–185. doi: 10.4018/978-1-4666-9441-5.ch013

  • CrossRef
  • Google Scholar

• 28

  LongY.LuoH.ZhangY. (2024). Evaluating large language models in analyzing classroom dialogue. NPJ Sci. Learn.9:60. doi: 10.1038/s41539-024-00273-3

  • CrossRef
  • Google Scholar

• 29

  McClureJ.SmyslovaD.HallA.JiangS. (2024). “Deductive coding's role in AI vs. human performance,” in Proceedings of the 17th international conference on educational data mining (809–813)

  • Google Scholar

• 30

  MorganD. L. (2023). Exploring the use of artificial intelligence for qualitative data analysis: the case of ChatGPT. Int J Qual Methods22:16094069231211248. doi: 10.1177/16094069231211248

  • CrossRef
  • Google Scholar

• 31

  NaY.FengS. (2025). “LLM-assisted automated deductive coding of dialogue data: leveraging dialogue-specific characteristics to enhance contextual understanding” in International conference on artificial intelligence in education. eds. ConatiC.HeffernanN. (Cham: Springer Nature Switzerland), 248–262.

  • Google Scholar

• 32

  NewellA.SimonH. A. (1972). Human problem solving. Englewood Cliffs, NJ: PrenticeHall.

  • Google Scholar

• 33

  OECD (2017). PISA 2015 assessment and analytical framework: Science, Reading, mathematic, financial literacy and collaborative problem solving, revised edition, PISA. Paris: OECD Publishing.

  • Google Scholar

• 34

  OpenAI. (2024). Best practices for prompt engineering with the OpenAI API. Available online at: https://help.openai.com/

  • Google Scholar

• 35

  PeczuhM. C.KumarN. A.BakerR.LehmanB.EisenbergD.MillsC.et al. (2025). Toward LLM-supported automated assessment of critical thinking subskills. arXiv preprint arXiv. doi: 10.48550/arXiv.2510.12915

  • CrossRef
  • Google Scholar

• 36

  PughS. L.RaoA.StewartA. E.D’MelloS. K. (2021). “Automatic modeling of collaborative problem-solving skills from student speech,” in Proceedings of the 14th international conference on educational data mining (EDM 2021) (55–67).

  • Google Scholar

• 37

  SankaranarayananS.BorchersC.SimonS.TajikE.AtaşA. H.CelikB. (2025). Automating thematic analysis with multi-agent LLM systems.

  • Google Scholar

• 38

  SimonS.SankaranarayananS.TajikE.BorchersC.ShahrokhianB.BalzanF. (2025). “Comparing a human’s and a multi-agent system’s thematic analysis: assessing qualitative coding consistency,” in International Conference on Artificial Intelligence in Education (60–73) Cham: Springer Nature Switzerland.

  • Google Scholar

• 39

  SongY.HaoT.LiuZ.LanZ. (2020). “A systematic review of frameworks for coding towards classroom dialogue,” in International symposium on emerging Technologies for Education (226–236) Springer: Cham

  • Google Scholar

• 40

  SwellerJ. (1988). Cognitive load during problem solving: effects on learning. Cogn. Sci.12, 257–285. doi: 10.1207/s15516709cog1202

  • CrossRef
  • Google Scholar

• 41

  TaoY.ChenG. (2023). Coding schemes and analytic indicators for dialogic teaching: a systematic review of the literature. Learn. Cult. Soc. Interact.39:100702. doi: 10.1016/j.lcsi.2023.100702

  • CrossRef
  • Google Scholar

• 42

  WellertL.BraunA.BeckerS.FrickP.BrendelW.LachnerA. (2025). Theory-prompt-validation: A practice-oriented approach to using LLMs for verbal coding in the learning sciences.

  • Google Scholar

• 43

  XiaoZ.YuanX.LiaoQ. V.AbdelghaniR.OudeyerP. Y. (2023). “Supporting qualitative analysis with large language models: combining codebook with GPT-3 for deductive coding,” in Companion proceedings of the 28th international conference on intelligent user interfaces (75–78).

  • Google Scholar

• 44

  ZambranoA. F.LiuX.BaranyA.BakerR. S.KimJ.NasiarN. (2023). “From nCoder to ChatGPT: from automated coding to refining human coding,” in International Conference on Quantitative Ethnography (470–485). Cham: Springer Nature Switzerland.

  • Google Scholar

• 45

  ZhangJ.BorchersC.AlevenV.BakerR. S. (2024). “Using large language models to detect self-regulated learning in think-aloud protocols,” in Proceedings of the 17th International Conference on Educational Data Mining. Washington, DC:National Center for Education Statistics. (157–168).

  • Google Scholar