Artificial intelligence and critical thinking: a case study with educational chatbots

This article presents a qualitative investigation into the evolution of critical questioning that occurred in dialogic relationships between artificial intelligence (AI) and a group of higher education students. Drawing from students' class records, it attempts to understand the development of critical thinking within formal educational contexts, utilizing a five-level questioning framework: General and Defining, Specific, Applied, Integrative, and Critical Engagement. The results indicate the presence of questioning patterns centered on levels four and five—namely, the use of Integrative and Critical Engagement questions—emerging from contextualized discussions. Students' interactions with AI enabled them to observe: the diversity of questioning levels, the progression of critical thinking, areas for improvement within each working group, the encouragement of reflexivity and metacognition, engagement with complex concepts, visualization of practical concept applications, and the expansion of interdisciplinary thinking. This study contributes to the literature on education and technology by offering insights into how to structure effective dialogic interactions between students and AI systems.

Introduction

Developing critical thinking in formal educational settings is a multifaceted challenge that requires a holistic approach beyond merely acquiring cognitive skills. Elder and Paul (2020) emphasize the importance of cultivating intellectual dispositions such as humility and open-mindedness, which are fundamental for effective critical questioning. This perspective is complemented by Browne and Keeley's (2007) proposal, which highlights students' ability to formulate pertinent questions and deconstruct complex arguments. For this study, the question was raised: how does critical questioning evolve in a dialogical relationship between students and artificial intelligence (AI)?

Nosich (2009), Facione (2015), and Elder and Paul (2020) suggest that structuring critical thinking into five levels of questioning provides a conceptual framework crucial for analyzing and promoting this evolution. This structure represents a continuum of development that requires consistent and reflective practice. By systematically fostering these questioning skills, we are preparing students to navigate the complexities of the modern world more adeptly. When this paradigm of critical questioning is consciously and systematically integrated into the educational process, it can radically transform how students interact with knowledge, fostering a more active, reflective, and engaged attitude toward their learning.

The integration of chatbots in education

The incorporation of chatbots in education is transforming teaching and learning processes. The emergence of new forms of interaction and knowledge construction in formal educational contexts prompts reflection on the development of students' cognitive abilities and knowledge construction through dialogue with AI systems.

In a meta-analytic study, Wu and Yu (2024) examined the impact of AI chatbots on student learning outcomes. Analyzing 24 randomized studies, they investigated the effects of AI chatbots on learning outcomes and their moderating effects on educational levels and intervention duration. The results indicated that AI chatbots had a significant positive effect on student learning outcomes. The impact was greater in higher education compared to primary and secondary education. Short-term interventions showed a stronger effect than longer ones, possibly due to the novelty effect. AI chatbots significantly improved learning performance, motivation, self-efficacy, interest, and appreciation of learning, while also reducing anxiety. According to the authors, AI chatbots have enormous potential as educational tools. They can act as partners, assistants, and mentors in learning environments, with effects that can vary depending on the context and duration of use. The authors suggest that future designers and educators should enhance learning outcomes by equipping AI chatbots with human-like avatars, as well as other elements of gamification and emotional intelligence.

The construction of knowledge through dialogical processes between students and AI

The integration of AI into educational contexts has opened new avenues for exploring dialogical processes in knowledge construction, with various scholars highlighting its potential to enhance learning experiences. Johansen et al. (2019) and Mollick and Mollick (2022) emphasize AI's capacity to adapt to individual student needs, promoting personalized and dialogical learning experiences. Johansen et al. focus on how chatbots and intelligent tutors can act as personalized guides, while Mollick and Mollick argue that large-scale language models offer real-time, adaptive dialogues that foster personalized learning environments. Tegos et al. (2020) extend this conversation by advocating for intelligent conversational agents in virtual learning environments, suggesting that such agents not only facilitate better engagement but also enhance the quality of educational dialogues.

Building on this foundation, Luckin et al. (2016) and Engeness and Lund (2020) provide a theoretical framework for understanding AI's role in scaffolding student learning. Luckin et al., through a Vygotskian lens, view AI as a “more capable partner” that offers adaptive scaffolding, critical for the student's cognitive development. Engeness and Lund expand on this by analyzing how AI mediates learning activities, restructuring the traditional processes of knowledge construction. Together, these perspectives illustrate how AI serves as an active participant in learning, not just facilitating knowledge acquisition but transforming the nature of educational dialogues.

The role of AI in collaborative learning is also explored by Baker et al. (2019) and Holmes et al. (2019), who examine how AI systems can stimulate critical thinking and collaborative knowledge building. Baker et al. focus on AI's ability to foster meaningful student discussions, while Holmes et al. delve into how AI can facilitate productive dialogues that promote metacognitive reflection. These contributions align with Zawacki-Richter et al. (2019), who highlight the growing use of AI in intelligent tutoring systems and conversational agents, pointing to the increasing relevance of AI-facilitated dialogue in educational settings.

However, critical voices like Selwyn (2019) caution against uncritical adoption, raising ethical concerns about AI's role in education. The potential biases in AI systems, as well as their implications for equity, demand careful consideration. While Goel and Polepeddi (2018) highlight successful implementations such as Jill Watson, an AI-based teaching assistant, they also underscore the importance of ensuring that AI not only complements but also enhances student learning. Overall, the dialogical processes between students and AI, though still in their early stages, show promising potential to reshape knowledge construction, offering new ways to foster critical thinking, problem-solving, and collaboration.

The evolution of critical thinking through dialogical processes between students and chatbots

The evolution of critical thinking has been the subject of study by various theorists, with its development closely tied to other cognitive and socio-emotional capacities. Elder and Paul (2020) argue that effective critical thinking requires more than cognitive skills; it also demands intellectual dispositions such as humility and open-mindedness. Browne and Keeley (2007) add to this discussion by emphasizing the importance of students' ability to ask pertinent questions and analyze complex arguments. Together, these perspectives highlight the multifaceted nature of critical thinking, extending beyond basic cognitive abilities to include emotional and intellectual dispositions.

To further understand the progression of critical thinking, Nosich (2009) and Facione (2015) propose a five-level question framework, which Elder and Paul (2020) also support. The first level, General and Defining Questions (PGD), lays the foundation for knowledge acquisition. Nosich (2009) argues that this initial phase is crucial for building a solid understanding of the subject matter, as it encourages students to establish basic knowledge before delving deeper. As students' progress to the second level, Specific Questions (PE), Facione (2015) highlights the importance of this stage in developing analytical skills, as students begin to refine their understanding by exploring more specific aspects of a topic.

Moving forward, the third level, Applied Questions (PA), focuses on knowledge transfer to practical contexts. Elder and Paul (2020) describe this as “substantive thinking,” where students apply theoretical knowledge to real-world situations, enhancing their ability to critically engage with practical problems. The fourth level, Integrative Questions (PI), emphasizes the creation of connections across different knowledge areas. Browne and Keeley (2007) stress that this phase is essential for fostering a holistic understanding, as students begin to see how different concepts interrelate.

At the highest stage, Critical Engagement Questions (PEC), Elder and Paul (2020) refer to this as “high-level thinking,” representing the peak of critical thinking development. This level requires students to synthesize their learning and engage deeply with complex ideas, challenging them to approach problems from multiple perspectives. However, Facione (2015) cautions that progression through these levels is neither linear nor automatic; it is a continuous process that requires consistent practice. Nosich (2009) also underscores the role of educators in modeling these questioning processes, suggesting that teachers must demonstrate how to formulate questions at each level to guide students in their development.

In the context of formal education, this five-level question framework has the potential to transform classroom dynamics. By systematically fostering the development of questioning skills, educators can better prepare students to navigate the complexities of contemporary society. Nosich (2009) emphasizes that this structured approach to questioning not only enhances critical thinking but also encourages students to engage more deeply with their learning, equipping them with the tools needed for thoughtful and informed decision-making.

Methodological procedures

This article addresses the proposed semester-long project for the course unit Research in Education that is part of the first year in the undergraduate program in Education Sciences. There were 25 students registered for this course. Ethical protocols for the development of the research, including safeguarding the anonymity of participants, were followed.

The program topics were as follows:

• Main characteristics of qualitative research: tradition and foundations.

• Stages of qualitative research: examples from real-life research.

• The credibility of a qualitative study: issues related to the fidelity and validity of conclusions.

• Analysis of qualitative research designs (case study, participatory research).

• Qualitative research models: naturalistic, ethnographic, case study, action research.

• Analysis of qualitative research techniques (open interview, participant observation, logbook, discourse and content analysis).

• Writing a qualitative research report: possible structures.

• Ethical considerations in the conduct of qualitative research.

The work took place over a semester in a weekly 4-h class. At the beginning, students were invited to study the course syllabus topics through dialogues with free generative AI tools (such as ChatGPT, SCISPACE, BING, PERPLEXITY, CHAT PDF, among others). To support this task, students were also asked to upload texts of the course's bibliographic references to the AI. The students were divided into five groups (group 1 with 6 participants, group 4 with 4 participants and the remaining groups with 5 participants each) based on their relational preferences. Each group recorded the dialogues established with the chatbots and at the end of each class a summary of the work. They sent these records to the Google Forms platform of the ELABORA Project, the research framework for this study. In total we collected 136 student records across 15 weeks (see an example in Annex (1). On average, each students group records were 2,506 words long. A qualitative analysis of the data obtained in the two questions was carried out: 1- Fully transcribe the dialogue held with generative technology with AI (questions and answers). 2- Reflect on how this interaction (AI-human) is promoting your learning.

Data analysis

For data treatment, pattern analysis was used, a powerful methodology for extracting meaningful insights from complex data. In qualitative research, pattern analysis is commonly employed in thematic analysis. Researchers search for patterns and themes within the data (Braun and Clarke, 2022). This process involves coding the data, identifying recurring concepts, and exploring their connections to reveal underlying structures and meanings (Saldaña, 2021).

Qualitative pattern analysis deals with complex data, such as narratives and observations. Miles et al. (2020) assert that this analysis captures nuances and context that may be overlooked in other approaches. By engaging deeply with the data, researchers can generate theoretical insights using techniques like comparison and theoretical sampling (Charmaz, 2014).

However, this type of analysis presents challenges such as subjectivity and issues with generalization. Patton (2015) highlights that qualitative pattern analysis requires reflexivity and methodological rigor to ensure the credibility of the findings. To enhance the robustness of the analysis, Lincoln and Guba (2018) recommend the use of triangulation and reflexivity as essential tools for strengthening the validity and reliability of the research.

The analysis process in this study was conducted collaboratively by the two authors, leveraging their distinct areas of expertise to ensure a thorough and balanced interpretation of the data. The first author focused on the initial stages of data coding, meticulously categorizing the students' questions based on the five levels of critical thinking, while the other concentrated on identifying patterns and drawing connections across categories. Regular discussions between the authors facilitated triangulation, enabling them to cross-verify their interpretations, address potential biases, and refine the emergent themes. This iterative and dialogic approach enhanced the methodological rigor and ensured that the findings accurately reflected the complexity of the data.

Methodological limitations, bias, and reflexive stance

This inquiry is intentionally situated as an in-depth single–course case study. While such a design offers rich contextual insight, it limits transferability beyond the specific institutional culture, disciplinary focus, and technological configuration employed here. Readers should therefore treat the present findings as analytic generalizations rather than statistical ones.

Choice of data source and qualitative procedure introduces several potential biases. First, the dataset consists of student-generated interaction logs. Because each log was authored retrospectively after class, it is vulnerable to selective recall and self-presentation effects: students may omit exchanges they deem trivial or unflattering. Second, the teacher-researcher dual role could lead to halo or expectancy bias when interpreting the sophistication of questions. Third, the pattern-coding framework itself shapes what is visible: by foregrounding the five predefined questioning levels, the analysis may under-represent other meaningful discourse features (e.g., affective tone, epistemic stance).

To manage these risks we adopted a reflexive, multi-layered strategy. (a) Double coding and intercoder agreement: two researchers independently coded 20 % of the corpus; discrepancies were reconciled through negotiated consensus (κ = 0.81). (b) Reflexive memos: after each coding round both coders recorded positionality statements detailing assumptions, emotional reactions, and emerging doubts; these memos were revisited in weekly debriefings to surface blind spots. (c) Triangulation: findings were cross-checked against course artifacts (syllabus discussions, classroom field notes) to corroborate or challenge log-derived interpretations.

Despite these safeguards, two limitations remain salient. The insider perspective, though mitigated, cannot be fully disentangled from interpretation. Furthermore, the enquiry captures only the written layer of student-AI dialogue; multimodal or unrecorded conversational cues are absent. Future research could replicate the protocol with blinded coders, multiple institutions, and complementary quantitative text-mining to test the stability of the questioning patterns reported here.

Analysis of critical questioning

The data from the five groups was organized and aggregated. The process involved coding all the questions based on the five levels of analysis for AI-Human dialogical processes (Elder and Paul, 2002; Facione, 2015; Nosich, 2009; Browne and Keeley, 2004) (Table 1). This coding framework allowed for a structured examination of the critical questioning patterns that emerged from the interactions, offering insights into how students engaged with AI tools at varying levels of depth and complexity. Each question was categorized according to its alignment with foundational, specific, applied, integrative, and critically engaging types of inquiry, reflecting the evolution of critical thinking throughout the study.

Table 1

Table 1. The five levels of analysis for AI-human dialogical processes.

In each data group, patterns of questioning and themes were identified, as well as recurring concepts within the categories. A typology of questioning was established in each group, referred to as a “pattern.” Finally, an interpretation was conducted, analyzing the interconnections within each questioning pattern and their underlying meanings. For this process, the AI tool Claude IA, version 3.5 Sonnet, was utilized to harness its analytical capabilities.

Below is the systematization of this data, presented as the critical questioning pattern observed in each group (see Tables 2–6).

Table 2

Table 2. Critical questioning pattern for group 1.

Table 3

Table 3. Critical questioning pattern for group 2.

Table 4

Table 4. Critical questioning pattern for group 3.

Table 5

Table 5. Critical questioning pattern for group 4.

Table 6

Table 6. Critical questioning pattern for group 5.

The pattern in Group 1 demonstrates a strong focus on specific questions (SQ), seeking guidance on practical aspects of conducting qualitative research. However, it also shows signs of applied (AQ), integrative (IQ), and critical engagement (CEQ) questions, suggesting a desire not only to understand the foundations of qualitative research but also to consider its application in a broader context and the limitations that arise. Group 1 did not develop questions that explicitly challenge the assumptions of qualitative research (CEQ) or explore its connections with other disciplines (IQ). Additionally, it does not consider the broader ethical and social implications of the topics studied (CEQ/AQ). Addressing these areas would foster greater critical and reflective cohesion regarding the topics.

Group 2 demonstrates a primary focus on general and defining questions (GDQ), aiming to establish a conceptual foundation for understanding qualitative research. The group also shows a tendency toward specific (SQ) and integrative (IQ) questions, suggesting a desire for more refined details regarding qualitative research and its relationship to other approaches.

In connecting theory to practice (AQ), Group 2 did not fully consider a holistic view (IQ), an important strategy for critical engagement (CEQ) based on underlying assumptions and implications. This would have complemented the conceptual foundation, enabling a deeper and more multifaceted understanding of the investigation.

Group 2 exhibits a questioning pattern primarily focused on building a conceptual base for understanding qualitative research. The initial question, “What are the main characteristics of qualitative research: tradition and foundations,” is a general and defining question (GDQ) that seeks to clarify the fundamental principles of the approach, indicating a desire to establish a solid foundational knowledge of the subject.

The subsequent questions show a progression toward more specific (SQ) and integrative (IQ) questioning. The question “What is qualitative research, and what are its main characteristics?” includes elements of both GDQ and SQ, seeking both a basic definition and a deeper exploration of its characteristics. The question “What are the differences between qualitative and quantitative research?” is integrative (IQ), inviting a comparison between different research approaches.

Group 2's questioning pattern suggests a focus on building a conceptual understanding of qualitative research, with some specific details related to its relationship with other research methods. This foundation sets the stage for developing a critical perspective on research, helping to understand the conceptual framework of qualitative research and how it is shaped by real-world contexts.

Group 3 focuses on general and defining questions (GDQ), although specific (SQ), applied (AQ), and integrative (IQ) questions are also present. GDQ questions serve as the essential starting point for any investigation, as they establish context and clarify key ideas. However, to achieve truly in-depth and critical inquiry, it is crucial to go beyond this stage. Group 3 appears ready to develop a richer and more multifaceted understanding, although they remained largely within the realm of defining foundations. Balancing all types of questions would foster a more comprehensive view of research, moving beyond concept clarification toward a more integrated approach that connects critique and reflection—an essential condition for a more multifaceted and engaging view of research.

Group 4 demonstrates a strong and balanced approach to critical questioning, encompassing GDQ, AQ, CEQ, and IQ. The questions aim to clarify fundamental concepts, connect theory and practice, engage in reflection, and establish connections between contexts. Group 4 also includes critical engagement (CEQ) and integrative (IQ) questions. They consider the implications of adhering to or neglecting ethical principles and draw connections between ethical considerations in different research approaches. This enhanced focus on SQ and CEQ exemplifies the group's tendency toward multidimensional questioning.

Group 4's inclination toward CEQ reveals their readiness to reflect on the implications of ethical considerations, particularly the consequences of following or disregarding ethical principles. This reflects a more thoughtful and engaged perspective. Their use of IQ indicates integrative thinking that seeks to make connections and comparisons, such as exploring ethical considerations in both qualitative and quantitative research. This suggests an interconnected appreciation of ethics in research.

With a further focus on SQ and CEQ, Group 4 is well-positioned to deepen their understanding and critical engagement with the ethical dimensions of qualitative research.

Group 5 presents a multifaceted questioning approach with various levels of critical development. It demonstrates a primary focus on general and defining questions (GDQ) and applied questions (AQ), aiming to clarify basic concepts and understand how these concepts manifest in the practice of qualitative research. The group also includes integrative questions (IQ) and critical engagement questions (CEQ), making comparisons between research approaches and critically questioning strategies for ensuring reliability. With this profile, Group 5 is well-positioned to develop more specific questions (SQ), which would aid in a more detailed understanding of qualitative research techniques and outcomes.

Group 5's questioning shows a good balance between GDQ, AQ, IQ, and CEQ. The strong focus on understanding the basic concept of qualitative research and its characteristics forms a solid foundation for grasping the principles that inform this methodological approach. This strategy ensures a deep comprehension of qualitative research and its underlying frameworks.

The extensive use of AQ highlights the group's interest in understanding how qualitative research concepts and methods are applied in practice and in exploring issues of reliability. Group 5 also integrates IQ and CEQ questions. The use of IQ, by comparing qualitative and quantitative approaches, demonstrates an effort to connect ideas across different research traditions. The group's tendency toward CEQ is evident in its critical questioning of strategies for establishing reliability, reflecting an appreciation for the complexities and challenges inherent in qualitative research.

Group 5's questioning is robust, indicating that it is well-equipped to integrate CEQ-type questions, which would further promote reflection on the relationship between the researcher's formation and the limitations of their approach.

Discussion

Our findings are best interpreted as a descriptive snapshot of how undergraduate students queried generative AI while studying qualitative-research methods. The frequency analysis (Table 7; Figure 1) confirms that the full range of our five-level questioning framework appeared in the logs, with notable variation across groups. Because the study lacked a control cohort and did not administer pre-/post-assessments of critical-thinking competence, we cannot claim that AI caused an improvement. Instead, the data show what kinds of questions students generated when AI was the principal study resource and illustrate the affordances and constraints of that dialogical setting.

Table 7

Table 7. Frequency of critical questioning levels by student group.

Figure 1

Figure 1. Distribution of critical questioning types across groups: spider chart visualization.

Table 7 presents a concise quantitative snapshot of how frequently each of the five critical-questioning levels emerged in the students' weekly interaction logs. By listing the raw counts for General & Defining (GDQ), Specific (SQ), Applied (AQ), Integrative (IQ), and Critical Engagement (CEQ) questions across the five groups, together with cohort totals, the table complements the earlier qualitative pattern analysis and makes it possible to compare questioning profiles both within and between groups.

Figure 1 visualizes the same distribution with a radar chart, allowing rapid comparison of the five groups against the overall average.

Figure 1 provides a visual counterpart to Table 7, enabling a more immediate and intuitive grasp of the variations in critical-questioning profiles across the five groups. The spider chart highlights distinct group-level tendencies: for example, Group 1 displays a sharp emphasis on Applied Questions (AQ), contrasting with Group 2's higher engagement in Critical Engagement Questions (CEQ) despite low overall question volume. Group 3 shows a more balanced distribution across GDQ, SQ, and IQ, while Groups 4 and 5 reveal modest but differentiated patterns, notably Group 5's concentration in CEQ. This graphical representation reinforces the textual analysis by illustrating not only the frequency but also the relative emphasis each group places on different levels of cognitive engagement.

As shown in Table 7 and Figure 1, Critical Engagement Questions (CEQ) were the most frequent, representing 30% of all questions recorded across the five groups. These were followed by Specific Questions (SQ) at 24% and Applied Questions (AQ) at 22%. General and Defining Questions (GDQ) accounted for 19% of the total, while Integrative Questions (IQ) were the least common, making up only 5% of all instances. This distribution suggests that while students engaged with various levels of critical questioning, integrative thinking remained underrepresented, and critical engagement was more prominent than initially anticipated.

Based on the results presented earlier from the five groups, the students' critical development resulting from interaction with AI was interpreted according to 4 emerging Interconnected Dimensions of Critical Thinking in AI-Supported Educational Dialogues: (A) Levels of Cognitive Engagement, (B) Research and Conceptual Understanding, (C) Analytical Approaches and Thought Structures, and (D) Individual and Group Dynamics in Critical Thinking. These dimensions work together to shape students' capacity to formulate, integrate, and evaluate knowledge during sustained engagement with generative AI tools in higher education.

Levels of cognitive engagement

This category illustrates how students understand and interact with concepts at different cognitive depths:

• Diversity of Questioning Levels: The different types of questions reflect varying levels of cognitive engagement, from basic understanding to critical evaluation. This category highlights the breadth of thinking skills that AI can stimulate.

It was found that students' questions spanned a range of cognitive levels, from basic understanding to deeper analysis and evaluation. For example:

“How to implement a qualitative research approach in practice” reflects a level of comprehension and application, seeking practical guidance.

“How to develop a questionnaire for qualitative research” is also at the application level and focuses on creating a specific research tool.

“How to analyze the results of a qualitative investigation” advances to higher levels of analysis and synthesis, requiring strategies to make sense of the data.

“Advantages and disadvantages of qualitative research” aims to weigh the pros and cons of this approach.

“What factors can influence qualitative research” reflects a level of analysis and considers the variables that shape the research process.

The diversity of questions demonstrates that interaction with AI is stimulating students to engage in thinking skills, from the basic level to the higher levels of Bloom's Taxonomy (Bloom, 1972).

• The Progression of Critical Thinking revealed how students moved from basic to more advanced types of questioning, showing the development and cognitive engagement throughout the work.

This regrouping shows how AI promotes deeper thinking and elevates students' thinking from initial understanding of concepts to more complex analytical skills.

There was progression in the level of critical thinking, demonstrated in the students' questions. They began with more basic questions about how to implement qualitative research, such as how to develop questionnaires, questions that are at the levels of comprehension and application.

They then progressed to questions requiring analysis and synthesis, such as analyzing qualitative results and how to consider factors influencing the research. Finally, they advanced to the evaluative level, weighing the advantages and disadvantages of the qualitative approach. As students interacted with AI, they felt more confident in developing more complex forms of questioning and exploring the topic. AI seems to support and encourage the development of more sophisticated critical thinking skills in students.

Research and conceptual understanding

This category aimed to understand how students process and apply knowledge through their interactions with AI:

• Engagement with Complex Concepts describes students' ability to deal with sophisticated ideas, showing how AI interactions help them handle more complex content. It was found that students' questions demonstrated an ability to engage with sophisticated ideas and concepts related to qualitative research. For example: questions about how to implement qualitative research in practice and how to develop questionnaires revealed that students intended to deal with the technical and methodological aspects of conducting qualitative studies; questions about how to analyze qualitative results indicates that students attempted to understand the processes of interpretation and deriving meaning from qualitative data; questions about the advantages, disadvantages, and factors influencing qualitative research show that students considered the nuances and complexities of using this methodological approach. By asking these questions, students demonstrated an ability to grapple with the complex and multifaceted concepts involved in understanding and applying qualitative research.

• Practical Application highlights the connection between theoretical learning and real-world practice. It reveals how students used AI to bridge the gap between knowledge and its application. This category reflects the way AI helps students transform abstract concepts into actionable knowledge.

Many of the students' questions focused on the practical application of theoretical knowledge about qualitative research. For example: asking how to implement a qualitative approach in practice shows a desire to bridge the gap between conceptual understanding and real-world application; asking how to develop a qualitative questionnaire is directly linked to creating real-life research tools; asking how to analyze qualitative results reflects students' desire to develop practical skills in data interpretation; considering the advantages, disadvantages, and factors influencing qualitative research helps students make informed decisions when designing their studies. Students used AI as a resource to bridge the gap between the theoretical knowledge they acquired and its application in real-life research contexts. AI served as a bridge in transforming concepts into actionable knowledge.

Analytical approaches and structures of thought

The aim of this category was to understand the structures and methodologies that students use to structure their research.

1. Interdisciplinary thinking focuses on connecting students' ideas across different fields and leading them to develop holistic, interdisciplinary approaches.

Although students' questions were primarily focused on the domain of qualitative research, there were some indications of interdisciplinary thinking. For example, considering the advantages, disadvantages, and factors that influence qualitative research requires students to think beyond the technical aspects of the methodology and consider the broader philosophical, ethical, and practical perspectives that shape research; asking when to choose qualitative research encourages students to consider how this methodological approach connects and compares with other approaches in different fields and disciplines.

However, opportunities for deep interdisciplinary connections were limited in this particular set of questions. Students appeared to be focused on mastering the concepts and practices within the domain of qualitative research itself.

2. Reflexivity and metacognition involve students' ability to be self-aware of their thought processes, with AI tools fostering metacognitive reflection.

There was some evidence of reflexivity and metacognitive thinking in the students' questions. For example, asking about the advantages and disadvantages of qualitative research requires students to critically reflect on the methodology, and consider its strengths and weaknesses in a metacognitive way; considering the factors that influence qualitative research involves a type of metacognitive reflection, about how various contexts and variables shape the research process; asking in what situations to choose qualitative methods encourages students to reflect on the conditions and contexts in which a qualitative approach is most appropriate, involving a type of strategic and metacognitive thinking.

While these examples suggest some level of reflexive and metacognitive engagement, the questions did not delve into students' thought processes. The focus was more on the mastery of qualitative research itself, rather than students' self-awareness of their learning and cognition.

Individual and group dynamics in critical thinking

This category illustrates the emerging differences between students' questions and engagement with AI in relation to critical thinking. The variation between different student approaches highlights the varying degrees of interaction with AI across question types. It emphasizes the different outcomes among students, an area where AI-driven inquiry can be refined. By analyzing the interactions between students and AI, it was possible to identify relevant points in terms of variation in approaches:

1. There was a variation in the degree of depth and specificity of the questions asked by students. While some asked broader and more conceptual questions, such as “Advantages and disadvantages of qualitative research,” others addressed more practical and procedural aspects, such as “How to design a questionnaire for qualitative research.”

2. The level of interaction and depth also varied. Some students were content with a more general initial response from the AI. In others, they made a greater effort to obtain more detailed and justified information, requesting references and citations from authors.

3. Identifying Areas for Improvement of gaps or weaknesses in the study on the topic of qualitative research in education. The aim is to identify where further development in critical questioning is needed.

It would be interesting to encourage students to ask more specific and contextualized questions about their own research, rather than very broad questions. This could generate more relevant and applicable insights. We present the following summary (Table 8).

Table 8

Table 8. Summary of critical development in students' interaction with AI.

We contend that there is significant potential for students to engage in more critical questioning and to undertake a more profound reflection on how to effectively utilize the insights generated by AI for their future qualitative research projects. Providing guidance in this direction could assist them in deriving greater benefits from this interaction.

Conclusion

This exploratory case study charted the distribution of critical-questioning levels in 136 student–AI interaction logs. All five levels: General & Defining through to Critical Engagement—were present, indicating that generative chatbots can host questions spanning the critical-thinking spectrum. However, the design does not allow us to conclude that AI enhanced students' critical-thinking skills. Without either (a) a comparison group using non-AI study strategies or (b) a pre-/post-measure of competence, causal inference is unwarranted. What we can assert is that AI provided a flexible conversational space in which such questioning was observable and capturable for subsequent analysis. These descriptive insights lay the groundwork for more future tests of AI's pedagogical impact.

The interaction between students and AI may stimulste critical thinking, as evidenced by the results on the levels of questioning in all groups. The questions elaborated by the students varied between general and critical, showing how AI can stimulate reflective thinking. This confirms what Browne and Keeley (2007) argue when they say that questioning is fundamental to critical thinking, and it applies to student-AI interaction.

The groups presented different questioning patterns, indicating an individualization of learning. AI adapted to learning styles and different needs, as Holmes et al. (2019) discuss the implications of AI for teaching and learning, supporting the idea of individualized learning paths. Students' critical thinking evolved from basic to complex questions, reflecting intellectual maturation. AI may act as cognitive support in this process. According to Elder and Paul (2020), AI tools should interact in learning, particularly in the development of critical thinking. The integration of AI tools into educational contexts requires not only technological innovation but also a robust, evidence-informed framework that aligns research, practice, and policy. This is consistent with the “golden triangle” approach to educational technology proposed by Cukurova et al. (2019), which emphasizes the importance of linking rigorous evidence, practitioner expertise, and industry development to ensure meaningful and sustainable adoption of digital tools in education.

The dialogical processes observed in this study also resonate with the principles of connectivism, which frames learning as the capacity to create and navigate networks of knowledge in a digital era (Siemens, 2005). From this perspective, the interaction between students and AI can be seen as an extension of the learning network, where knowledge is constructed through connections across human and non-human agents.

The identification of areas for improvement in each group encouraged continuous intellectual expansion, corroborating Siemens and Crosslin (2020) on the adequacy of connectivist learning theory in the digital age. Students interconnected complex concepts and practical situations, demonstrating the effectiveness of AI as a bridge between theory and practice. Luckin et al. (2016) argue in favor of AI in education, highlighting its potential to connect abstract concepts to concrete applications.

Interactions with AI can foster reflexivity and enhanced students' metacognitive abilities, enabling them to critically reflect on their thought processes and approach learning with greater self-awareness. This aligns with Facione's (2015) assertion that critical thinking inherently includes metacognitive components, which are essential for cultivating autonomous and critical learners. Additionally, the AI-stimulated interdisciplinary thinking broadened students' conceptual boundaries, encouraging them to integrate knowledge across diverse domains and adopt more holistic perspectives in their inquiry.

Our results align with Zawacki-Richter et al.'s (2019) emphasis on the potential of AI applications in higher education to foster interdisciplinary connections, as evidenced by the integrative questions generated by the students. Similarly, the progression of critical thinking observed in our study reflects the findings of Wu and Yu's (2024) meta-analysis, which highlights the effectiveness of AI chatbots in enhancing cognitive engagement and learning outcomes.

Student-AI interaction, when structured around progressive levels of critical questioning, has the potential to catalyze multidimensional cognitive development. This process not only enriches students' intellectual repertoire but also prepares students to deal with greater dexterity with the complexity of the contemporary world, where the ability to question, integrate, and apply knowledge critically is increasingly valued.

The pattern of critical development observed in the groups suggests that AI interaction with students was effective because it stimulated different levels of critical thinking. This work methodology provided a favorable context for students to explore concepts of progressive criticality through AI.

Other possibilities are also presented to improve critical reflection in interaction with AI:

1. Guide students to request references and sources about the information provided by AI, so that they can verify reliability and deepen concepts.

2. Encourage more critical reflection on how the information obtained in the interaction with AI can be incorporated in a way that impacts each student's research. Go beyond generic comments and analyze the applicability and limitations of that knowledge.

3. Foster more interaction circuits and follow-ups with AI, so that students can explore nuances, clarify doubts to reach a more complete understanding of the topic.

4. Contextualize and encourage students to consider how the information provided by AI specifically relates to the conceptions of their research projects. They should reflect on the relevance and applicability of that knowledge in the research context.

5. Suggest that students record their reflections and insights throughout interactions with AI, so that they can identify areas that require further deepening or clarification.

By developing this type of guidance, teachers actively promote students' critical reflection and help them in analytical thinking skills from the interaction with AI. This enriches the learning experience and better prepares students for robust and well-founded qualitative investigations.

To determine whether AI dialoguing can develop critical thinking, subsequent studies should incorporate: (i) a non-AI control condition or alternative instructional treatment; (ii) standardized pre- and post-tests of critical-thinking disposition and skill; and (iii) larger, multi-institution samples to enhance generalizability. Mixed-methods designs combining automated discourse analytics with human coding may also clarify how specific chatbot features (prompting style, feedback immediacy) relate to movement across questioning levels.

Data availability statement

The raw data supporting the conclusions of this article will be made available by the authors, without undue reservation.

Ethics statement

The studies involving humans were approved by Comissão de Ética (CdE) do Instituto de Educação (IE) da Universidade de Lisboa. The studies were conducted in accordance with the local legislation and institutional requirements. The participants provided their written informed consent to participate in this study.

Author contributions

LT: Funding acquisition, Conceptualization, Writing – original draft, Writing – review & editing. PB: Validation, Writing – review & editing, Funding acquisition, Resources, Formal analysis, Supervision, Project administration, Data curation, Writing – original draft, Visualization, Conceptualization, Methodology, Investigation.

Funding

The author(s) declare that financial support was received for the research and/or publication of this article. This work was supported by National Funds through FCT—Portuguese Foundation for Science and Technology, I.P., under the scope of UIDEF—Unidade de Investigação e Desenvolvimento em Educação e Formação, UIDB/04107/2020 (https://doi.org/10.54499/UIDB/04107/2020).

Conflict of interest

The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Generative AI statement

The author(s) declare that no Gen AI was used in the creation of this manuscript.

Any alternative text (alt text) provided alongside figures in this article has been generated by Frontiers with the support of artificial intelligence and reasonable efforts have been made to ensure accuracy, including review by the authors wherever possible. If you identify any issues, please contact us.

Publisher's note

All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.

Supplementary material

The Supplementary Material for this article can be found online at: https://www.frontiersin.org/articles/10.3389/feduc.2025.1630493/full#supplementary-material

References

Baker, T., Smith, L., and Anissa, N. (2019). Educ-AI-tion Rebooted? Exploring the Future of Artificial Intelligence in Schools and Colleges. London: Nesta Foundation.

Google Scholar

Bloom, B. S., (ed.). (1972). Taxonomia de Objetivos Educacionais: Domínio Cognitivo. São Paulo: Editora Globo.

Google Scholar

Braun, V., and Clarke, V. (2022). Thematic Analysis: A Practical Guide. London: SAGE Publications. doi: 10.53841/bpsqmip.2022.1.33.46

Crossref Full Text | Google Scholar

Browne, M. N., and Keeley, S. M. (2004). Asking the Right Questions: A Guide to Critical Thinking, 7th Edn. Pearson Prentice Hall.

Google Scholar

Browne, M. N., and Keeley, S. M. (2007). Asking the Right Questions: A Guide to Critical Thinking. London: Pearson.

Google Scholar

Charmaz, K. (2014). Constructing Grounded Theory, 2nd Edn. London: SAGE Publications.

Google Scholar

Cukurova, M., Luckin, R., and Clark-Wilson, A. (2019). Creating the golden triangle of evidence-informed education technology with EDUCATE. Br. J. Educ. Technol. 50, 490–504. doi: 10.1111/bjet.12727

Crossref Full Text | Google Scholar

Elder, L., and Paul, R. (2002). Critical Thinking: Tools for Taking Charge of Your Professional and Personal Life. Financial Times Prentice Hall.

Google Scholar

Elder, L., and Paul, R. (2020). Critical Thinking: Tools for Taking Charge of Your Learning and Your Life, 3rd Edn. Santa Barbara, CA: Foundation for Critical Thinking.

Google Scholar

Engeness, I., and Lund, A. (2020). Learning for the future: insights arising from the contributions of Piotr Galperin to the cultural-historical theory. Learn. Cult. Soc. Interact. 25:100257. doi: 10.1016/j.lcsi.2018.11.004

Crossref Full Text | Google Scholar

Facione, P. A. (2015). Critical Thinking: What It Is and Why It Counts. Insight Assessment. Available online at: https://courseware.e-education.psu.edu/downloads/geog882/Critical%20Thinking%20What%20it%20is%20and%20why%20it%20counts.pdf Retrieved January 8, 2024, from https://www.insightassessment.com/

Google Scholar

Goel, A. K., and Polepeddi, L. (2018). Jill Watson: A Virtual Teaching Assistant for Online Education. Atlanta, GA: Georgia Institute of Technology. doi: 10.4324/9781351186193-7

Crossref Full Text | Google Scholar

Holmes, W., Bialik, M., and Fadel, C. (2019). Artificial Intelligence in Education: Promises and Implications for Teaching and Learning. Boston, MA: Center for Curriculum Redesign.

Google Scholar

Johansen, J., Øvergaard, K. R., and Eriksen, S. (2019). Chatbots in Education: A Passing Trend or a Valuable Pedagogical Tool? Oslo: OsloMet Artificial Intelligence Lab.

Google Scholar

Lincoln, Y. S., and Guba, E. G. (2018). Naturalistic Inquiry. London; Thousand Oaks, CA: SAGE Publications.

Google Scholar

Luckin, R., Holmes, W., Griffiths, M., and Forcier, L. B. (2016). Intelligence Unleashed: An Argument for AI in Education. London: Pearson.

Google Scholar

Miles, M. B., Huberman, A. M., and Saldaña, J. (2020). Qualitative Data Analysis: A Methods Sourcebook, 4th Edn. London; Thousand Oaks, CA: SAGE Publications.

Google Scholar

Mollick, E., and Mollick, L. (2022). New modes of learning enabled by AI chatbots: three methods and assignments. SSRN Electr. J. doi: 10.2139/ssrn.4300783

Crossref Full Text | Google Scholar

Nosich, G. M. (2009). Learning to Think Things Through: A Guide to Critical Thinking Across the Curriculum, 5th Ed. London: Pearson.

Google Scholar

Patton, M. Q. (2015). Qualitative Research and Evaluation Methods: Integrating Theory and Practice, 4th Edn. London; Thousand Oaks, CA: SAGE Publications.

Google Scholar

Paul, R., and Elder, L. (2007). The Thinker's Guide to Analytic Thinking. Foundation for Critical Thinking Press.

Google Scholar

Saldaña, J. (2021). The Coding Manual for Qualitative Researchers, 4th Edn. London; Thousand Oaks, CA: SAGE Publications.

Google Scholar

Selwyn, N. (2019). Should Robots Replace Teachers? AI and the Future of Education. Cambridge: Polity.

Google Scholar

Siemens, G. (2005). Connectivism: a learning theory for the digital age. Int. J. Instruct. Technol. Dist. Learn. 2, 3–10. Available online at: http://www.itdl.org/Journal/Jan_05/article01.htm

Google Scholar

Siemens, G., and Crosslin, M. (2020). Connectivism: A Learning Theory for the Digital Age. Cambridge University Press.

Google Scholar

Tegos, S., Demetriadis, S., and Karakostas, A. (2020). Conversational agents for academically productive talk: a review of issues and potential. Int. J. Artif. Intell. Educ. 30, 1–24.

Google Scholar

Wu, R., and Yu, Z. (2024). Do AI chatbots improve students learning outcomes? Evidence from a meta-analysis. Br. J. Educ. Technol. 55, 10–33. doi: 10.1111/bjet.13334

Crossref Full Text | Google Scholar

Zawacki-Richter, O., Marín, V. I., Bond, M., and Gouverneur, F. (2019). Systematic review of research on artificial intelligence applications in higher education – where are the educators? Int. J. Educ. Technol. High. Educ. 16:39. doi: 10.1186/s41239-019-0171-0

Crossref Full Text | Google Scholar