ChatGPT in a programming course: benefits and limitations
Rex Bringula- College of Computer Studies and Systems, University of the East, Manila, Philippines
This paper explores the potential use and limitations of ChatGPT in a programming course, specifically focusing on its evaluation in a Data Analytics course due to its broad applications. The study reveals that ChatGPT offers valuable assistance to teachers in creating class materials, facilitating teaching-learning activities, and designing assessments. Students also benefit from the technology, as it can generate R programming codes and serve as a learning companion. However, limitations were identified, including the generation of incorrect reference materials, the tendency to produce pattern-like content, and potential misuse. The paper encourages replication of the study in other courses to uncover additional strengths and limitations, contributing to the development of ethical guidelines for responsible utilization of this exciting technology.
1 Introduction
Artificial intelligence (AI) tools in education have transformed the educational landscape by providing personalized learning experiences and optimizing administrative tasks. For example, intelligent tutoring systems employ AI algorithms to tailor content to the needs of individual students, providing tailored instruction and feedback (Graesser et al., 2018). chatbots and virtual assistants automate administrative processes by answering routine questions and performing administrative tasks, freeing educators’ time (Mulyana et al., 2018). AI also helps educators assess student performance trends and tailor instructional strategies by facilitating data analysis (Graesser et al., 2018). Overall, AI tools in education contribute to more efficient, adaptive, and inclusive learning experiences (Chen et al., 2020).
Currently, the most popular AI tool is ChatGPT. OpenAI (2021) developed ChatGPT, an AI language model. This technology was introduced in November 2020 (Altmäe et al., 2023). It is a chatbot that was built on the Generative Pre-trained Transformer architecture and can respond to a query in a human-like manner. It can understand and reply to a variety of topics, answer questions, provide explanations, make ideas, and participate in interactive dialogues (Haleem et al., 2022; Stokel-Walker, 2023). It is a versatile tool that may serve as customer support chatbots, virtual assistants, content developers, and educational aids (Kalla and Smith, 2023). Because of its state-of-the-art capabilities, it has gained enormous popularity. ChatGPT had already reached 100 million users as of February 2023, making it the fastest application used by people (Hu, 2023).
Previous studies have indicated that ChatGPT can be effectively employed in programming courses, demonstrating positive impacts on programming learning (Yilmaz and Yilmaz, 2023a, 2023b). The current study seeks to contribute to ongoing discussions regarding the educational value of this technology in programming by highlighting its potential as a versatile educational resource for teachers. Specifically, this research aims to demonstrate that this technology offers valuable support to teachers in various capacities. Teachers can utilize it for generating educational content, addressing student queries, delivering explanations, and facilitating lesson planning. They can employ it in crafting engaging learning materials, worksheets, and quiz questions. By providing a prompt related to the subject matter, they can elicit detailed and informative responses from this chatbot.
Furthermore, teachers can input student questions into ChatGPT, receiving immediate responses or explanations. They can also input a concise description of the topics or concepts they wish to cover, prompting the chatbot to recommend relevant information, examples, or activities (Rahman and Watanobe, 2023; van den Berg and du Plessis, 2023). This proves particularly advantageous for tailoring support to individual student needs beyond the confines of the classroom. This technology serves as a valuable tool for brainstorming lesson plan ideas and content creation. This paper aims to provide a perspective on how teachers can leverage its capabilities, particularly in specialized courses such as data analytics.
2 The Data Analytics course
Data analytics is the application of computer systems to the analysis of large data sets to support decisions. Recognizing the benefits of this field, universities in the Philippines offered the Data Analytics course. For example, at one university in Manila, a Data Analytics course has been offered to prepare students to meet the demands of the industry for data analytics graduates. Due to its extensive content, it is offered as a five-unit subject. Its popularity attracts more than 100 students annually.
The course is generally divided into two general topics: clustering algorithms (e.g., k-means and hierarchical clustering) and classification techniques (e.g., decision trees, simple linear regression, and simple logistic regression). Clustering is the process of grouping similar objects to find possible patterns in a data set. Meanwhile, classification is the process of finding a rule (i.e., model or function) from the existing dataset and then using this rule to categorize an object that was not previously labeled (Han and Kamber, 2006).
Because of the technicalities of the course, teachers need to prepare students in both conceptual (e.g., concepts) and technical skills (e.g., programming). Due to the number of students in the course and the extensive class preparations required by teachers, ChatGPT may serve as an educational resource for teachers. The Data Analytics course could benefit from the capabilities of this chatbot or technology (i.e., ChatGPT). For instance, before this technology, teachers had to read various books to obtain the desired code. With ChatGPT, teachers can instantly acquire the desired code and test it in a programming language. Given its capabilities, this paper intends to determine whether ChatGPT could be utilized as a tool from the perspective of an educator.
However, the use of this technology raises ethical concerns such as inequity, bias, and plagiarism (Huang and Tan, 2023; Ray, 2023). Furthermore, it is crucial to highlight that ChatGPT is still an AI model that may provide wrong, biased, and inaccurate responses. In other words, this technology may still have inherent limitations. Hence, this paper will also highlight the possible limitations of this technology within the context of a Data Analytics course.
3 Potential uses of ChatGPT on data analytics
ChatGPT can be considered a chatbot—a conversational or interactive agent that provides responses and feedback from inquiries (Okonkwo and Ade-Ibijola, 2020; Smutny and Schreiberova, 2020). In addition, it can generate content, define terms, and act as a programming assistant. These functionalities were non-existent or limited in the previous educational chatbots (see Okonkwo and Ade-Ibijola, 2020). To highlight these functionalities in an educational context, I asked ChatGPT to generate content for my instructional resources (e.g., class materials, teaching-learning activities, and assessments) in my Data Analytics course. To test its capabilities, I limited the content to k-means clustering. K-means clustering is the first topic that involves an algorithmic process.
3.1 Class material—handouts, and ChatGPT itself
I instructed ChatGPT to generate a handout on k-means clustering, and it produced an outline-like text. This output organizes the potential points of discussion on the topic in a well-structured manner. The outline format provides teachers with a clear overview of the flow of the discussion, starting from the main sections down to the subtopics. Additionally, the generated output serves as a helpful reminder for teachers to stay aligned with the course objectives and track their progress. As a result, it enhances clarity, keeps the focus intact, and promotes effective communication among learners and teachers.
ChatGPT itself can be considered an educational tool. For instance, if asked “Define k-means clustering,” it provided a very quick response. The response was generated in less than 10 s. As a chatbot, it can provide instant responses to queries and serve as instant educational support. In addition, this support is available anytime, eliminating the barriers of time and location.
3.2 Teaching learning activities—group work, hands-on activity, machine problem, inquiry-based learning, project-based learning, role playing, and debate
For teaching-learning activities (TLA), ChatGPT was given this instruction: “Provide me with a [TLA] for k-means clustering [using R],” where TLA can be replaced by the type of TLA being requested. For a machine problem, the phrase “using R” was added to emphasize the use of the R programming language. The outputs are very promising. For instance, it provided detailed instructions on how to conduct group work. It suggested dividing the class into groups of 3–4 students per group and providing them with a dataset to work with. Then, it was suggested to instruct the students to preprocess the data. Afterward, reconvene the class and ask them to present their processed data as well as the challenges in processing the data.
Overall, the results provide teachers with TLA that may encourage students’ classroom participation. The findings are important for educators because they allow teachers to enable interactive and collaborative problem-solving experiences for students. Furthermore, these outcomes can be adjusted to students’ different learning needs and lecture delivery preferences. In other words, ChatGPT can provide teachers with TLAs that may break the monotony of a lecture-based delivery style.
3.3 Content generation—programming code
Another interesting capability of ChatGPT is its ability to serve as a programming assistant. It can generate programming codes for k-means clustering. First, I requested that it to generate a code for k-means clustering (e.g., “Construct me a k-means in R”). Second, I requested a code where students can upload their datasets and then analyze this data (e.g., “Generate an R code for k-means clustering where users have their own dataset”) (Figure 1). In both instances, it was able to generate the requested codes. These codes were tested, and they worked perfectly. Moreover, it also provided a short annotation explaining the code.
Figure 1. A ChatGPT-generated R code for k-means clustering.
3.4 Learning companion
A learning companion is a supportive resource that can provide guidance, feedback, and personalized assistance both for teachers and students. A learning companion can be a human tutor or a non-human tutor (e.g., a virtual tutor, an intelligent tutoring system, or an interactive software application). ChatGPT fits perfectly with the definition of a learning companion. For example, I asked this chatbot to act as a tutor in data analytics. I asked it to give me a five-item multiple-choice type of quiz, and checked whether my answers were correct. It neatly provided the five questions and checked my answers. It did not only provide me with the correct answers, but it also explained why my answers were correct (or incorrect).
ChatGPT can also facilitate interactive Q&A sessions. It can serve as a virtual assistant where students may ask questions or get explanations about a concept. I asked it to explain the limitations of the k-means algorithm. It provided several limitations (e.g., sensitivity to outliers, determining the number of clusters to retain, etc.), which were all correct. It also has the capability to furnish a dataset suitable for practice exercises. I instructed it to generate a dataset in tabular format with 30 rows of data. As expected, it was able to execute the instructions very well and generate the requested dataset.
Teachers could indirectly benefit from these capabilities. Human-to-human tutoring can be augmented with the help of this chatbot. It can provide individualized tutoring assistance to struggling students based on their specific needs and learning pace. This individualized support complements the teacher’s efforts in the classroom to handle varied learning abilities. Teachers can use it to enhance their teaching materials and provide additional resources to students who require extra assistance. This can include drills, explanations, and study guides. It can offer students with immediate feedback, allowing them to correct errors and learn concepts more quickly. This relieves teachers of some of the challenges of providing one-on-one feedback.
Moreover, while learning a new subject, a teacher can use ChatGPT to learn how to code in R. It can help teachers save time when learning data analytics and the R programming language. Teachers can request it to generate the codes and then run them in the R programming language to validate their accuracy.
3.5 Assessments—written exams and assignments
Perhaps one of the most exciting functionalities of ChatGPT is its ability to generate texts resembling assessments. In this paper, I asked (e.g., “Generate a 5-item multiple-choice type of quiz with 3 options for k-means clustering.”) it to generate five-item written exams in four different forms: multiple-choice, identification, fill-in-the-blanks, and true-or-false (see Figure 2). In almost an instant, it generated the requested content. It not only generated the content but also provided the answers to the questions. It may also recommend a reading assignment and a rubric for rating the activities. This finding implies that teachers can utilize these functionalities and could reduce their workload by constructing quizzes, exams, and rubrics.
Figure 2. A multiple-choice quiz generated by ChatGPT.
However, upon inspection of the recommended reading, it is found that the paper is nonexistent. This issue has been reported in previous studies (Bringula, 2023; Currie, 2023; Kumar, 2023). The recommended reading was searched in Google Scholar, and there were three possible candidates due to their almost similar titles. These articles were written by three different sets of authors: Ahmed et al. (2020), Xu and Tian (2015), and Oti et al. (2021). I also searched the names of the recommended authors and found that the article they wrote was titled “Data Clustering: A Review” (Jain et al., 1999). This limitation is further discussed in the succeeding section.
4 Issues and limitations
ChatGPT is a very promising technology that can aid teachers in the Data Analytics course. However, just like any other technology, it has potential disadvantages. One possibility is the overdependence on this technology. If teachers solely rely on the responses of this chatbot, they may not develop their critical thinking skills and may not consult other educational materials. The developers of this technology acknowledge that it may sometimes provide biased or incorrect responses. Thus, teachers may be misinformed if they do not verify the accuracy of the responses.
The technology may reduce the workload of teachers in terms of creating their TLAs. However, there is a possibility that students may use the tool to replicate the possible items in the TLAs. Therefore, teachers should use this functionality with caution. The results of the test-content generation may serve as a basis, but teachers should not entirely rely on this functionality. Similarly, without a clear policy on the use of this technology for code generation, it will be unclear whether students are cheating in their activities or not. Students’ assignment submissions with AI-generated content may be construed as cheating (e.g., plagiarism) if answers are taken verbatim from ChatGPT. Institutional or classroom-level policies are needed to address these issues.
Meanwhile, it was disclosed that the technology has some limitations in the context of the Data Analytics course. It has been observed that the generated TLAs will eventually form a pattern. For example, once the topic was changed from k-means to hierarchical clustering, almost the same contents were provided except that the topic was changed to hierarchical clustering. While this is technically acceptable and supports topic continuity, it will become monotonous. It is desirable to apply the algorithm to different fields of interest.
Another limitation is the possibility that the generated quiz items might not be discussed in class. Hence, students may not be able to answer these items. Another limitation observed was that when asked to generate a 20-item multiple-choice question, ChatGPT was only able to provide 14 items. This is possibly attributed to the limited concepts involved in the topic.
Program code generation offers an opportunity for students to learn programming. Nevertheless, human teachers are still needed to explain the codes. Human teachers can adapt to the learning needs of students based on observations and feedback from students. They can modify the approach and pace of the discussion based on the comprehension levels of the students. The pedagogical expertise of teachers may allow probing questions, ensuring that learning outcomes will be met.
Aside from the findings of this study, prior works have also addressed the limitations of ChatGPT in educational settings. The study of Tyson (2023) found that it cannot reliably perform mathematical operations, makes conceptual errors, and fabricates plausible-looking citations. Similarly, this issue is also reported in the study of Bringula (2023), which suggests that it could facilitate plagiarism. In another study, the risk of generating inaccurate content was highlighted (Sallam et al., 2023). However, in the field of data analytics, inaccurate content, such as programming codes, can be easily verified since teachers (and students alike) may run the code in a code editor.
Finally, this study has shown that ChatGPT may not be able to provide a correct reference for all occasions. As an AI, its model is constantly learning and can eventually attain the desired accuracy in referencing results. Validation of the results is needed to ensure the authenticity of the recommended readings.
5 Conclusion and future research directions
The concepts discussed in the preceding sections have demonstrated the capabilities of ChatGPT as a valuable tool for assisting teachers in various aspects of their work, including the development of class materials, teaching-learning activities, assessment tools, and programming code generation. This paper also discussed the possible usage of this technology to support students’ learning in the context of the Data Analytics course. However, it is crucial to acknowledge the potential issues and limitations associated with this technology. While it offers valuable support, human judgment remains essential in its utilization. To leverage the advantages of this technology effectively, it is advisable to establish institutional and classroom policies that regulate its usage and define ethical guidelines.
Further research is warranted to comprehensively explore the capabilities and limitations of ChatGPT. Given its continuous nature, we can anticipate increasingly accurate responses over time. Future studies should investigate its impact on teaching effectiveness, examining how it can enhance the learning experience for both teachers and students. Additionally, researchers can delve into the influence of program code generation on students’ programming skills within the domain of data analytics. Lastly, the possible uses and limitations of this technology could be investigated in other courses.
Data availability statement
The original contributions presented in the study are included in the article/supplementary material, further inquiries can be directed to the corresponding author.
Author contributions
The author confirms being the sole contributor of this work and has approved it for publication.
Funding
This study is funded by the University of the East.
Acknowledgments
The author would like to thank University of the East for funding this study.
Conflict of interest
The author declares that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.
Publisher’s note
All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.
References
Ahmed, M., Seraj, R., and Islam, S. M. S. (2020). The k-means algorithm: a comprehensive survey and performance evaluation. Electronics 9:1295. doi: 10.3390/electronics9081295
Altmäe, S., Sola-Leyva, A., and Salumets, A. (2023). Artificial intelligence in scientific writing: a friend or a foe? Reprod. Biomed. Online 47, 3–9. doi: 10.1016/j.rbmo.2023.04.009
Bringula, R. (2023). What do academics have to say about ChatGPT? A text mining analytics on the discussions regarding ChatGPT on research writing. AI Ethics, 1–13. doi: 10.1007/s43681-023-00354-w
Chen, L., Chen, P., and Lin, Z. (2020). Artificial intelligence in education: a review. IEEE Access 8, 75264–75278. doi: 10.1109/ACCESS.2020.2988510
Currie, G. M. (2023). Academic integrity and artificial intelligence: is ChatGPT hype, hero, or heresy? Semin. Nucl. Med. 53, 719–730. doi: 10.1053/j.semnuclmed.2023.04.008
Graesser, A. C., Hu, X., and Sottilare, R. (2018). “Intelligent tutoring systems” In: F Fischer, CE Hmelo-Silver, SR Goldman, and P Reimann, editors. International Handbook of the Learning Sciences (Routledge), 246–255.
Haleem, A., Javaid, M., and Singh, R. P. (2022). An era of ChatGPT as a significant futuristic support tool: a study on features, abilities, and challenges. BenchCouncil Trans. Benchmarks Std. Eval. 2:100089. doi: 10.1016/j.tbench.2023.100089
Han, J., and Kamber, M. (2006). Data Mining: Concepts and Techniques. 2nd Edn. San Francisco, CA, USA: Elsevier Inc.
Hu, K. (2023). ChatGPT Sets Record for Fastest-Growing User Base—Analyst note Reuters. Available at: https://www.reuters.com/technology/chatgpt-sets-record-fastest-growing-user-base-analyst-note-2023-02-01/
Huang, J., and Tan, M. (2023). The role of ChatGPT in scientific communication: writing better scientific review articles. Am. J. Cancer Res. 13, 1148–1154.
Jain, A. K., Murty, M. N., and Flynn, P. J. (1999). Data clustering: a review. ACM Comput. Surv. 31, 264–323. doi: 10.1145/331499.331504
Kalla, D., and Smith, N. (2023). Study and analysis of chat GPT and its impact on different fields of study. Int. J. Innov. Sci. Res. Technol. 8, 827–833.
Kumar, A. H. (2023). Analysis of ChatGPT tool to assess the potential of its utility for academic writing in biomedical domain. Biol. Eng. Med. Sci. Rep. 9, 24–30. doi: 10.5530/bems.9.1.5
Mulyana, E., Hakimi, R., and Hendrawan,. (2018). “Bringing automation to the classroom: a chatOps-based approach” in 2018 4th International Conference on Wireless and Telematics (ICWT). IEEE, pp. 1–6.
Okonkwo, C. W., and Ade-Ibijola, A. (2020). Python-bot: a chatbot for teaching Python programming. Eng. Lett. 29, 25–34.
OpenAI (2021). ChatGPT: Language models are few-shot learners. Available at: https://openai.com/research/chatgpt/
Oti, E. U., Olusola, M. O., Eze, F. C., and Enogwe, S. U. (2021). Comprehensive review of K-means clustering algorithms. Criterion 7, 64–69. doi: 10.31695/IJASRE.2021.34050
Rahman, M. M., and Watanobe, Y. (2023). ChatGPT for education and research: opportunities, threats, and strategies. Appl. Sci. 13:5783. doi: 10.3390/app13095783
Ray, P. P. (2023). ChatGPT: a comprehensive review on background, applications, key challenges, bias, ethics, limitations and future scope. IoT Cyber-Phys. Syst. 3, 121–154. doi: 10.1016/j.iotcps.2023.04.003
Sallam, M., Salim, N., Barakat, M., and Al-Tammemi, A. (2023). ChatGPT applications in medical, dental, pharmacy, and public health education: a descriptive study highlighting the advantages and limitations. Narra J. 3:e103. doi: 10.52225/narra.v3i1.103
Smutny, P., and Schreiberova, P. (2020). Chatbots for learning: a review of educational chatbots for the Facebook messenger. Comput. Educ. 151:103862. doi: 10.1016/j.compedu.2020.103862
Stokel-Walker, C. (2023). ChatGPT listed as author on research papers: many scientists disapprove. Nature 613, 620–621. doi: 10.1038/d41586-023-00107-z
Tyson, J. (2023). Shortcomings of ChatGPT. J. Chem. Educ. 100, 3098–3101. doi: 10.1021/acs.jchemed.3c00361
van den Berg, G., and du Plessis, E. (2023). ChatGPT and generative AI: possibilities for its contribution to lesson planning, critical thinking and openness in teacher education. Educ. Sci. 13:998. doi: 10.3390/educsci13100998
Xu, D., and Tian, Y. (2015). A comprehensive survey of clustering algorithms. Ann. Data Sci. 2, 165–193. doi: 10.1007/s40745-015-0040-1
Yilmaz, R., and Yilmaz, F. G. K. (2023a). Augmented intelligence in programming learning: examining student views on the use of ChatGPT for programming learning. Comput. Hum Behav. Artif. Hum. 1:100005. doi: 10.1016/j.chbah.2023.100005
Keywords: analytics, artificial intelligence, chatbot, education, ChatGPT
Citation: Bringula R (2024) ChatGPT in a programming course: benefits and limitations. Front. Educ. 9:1248705. doi: 10.3389/feduc.2024.1248705
Edited by:
Charoula Angeli, University of Cyprus, CyprusReviewed by:
Stefan Pasterk, University of Klagenfurt, AustriaRamazan Yilmaz, Bartin University, Türkiye
Copyright © 2024 Bringula. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) and the copyright owner(s) are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.
*Correspondence: Rex Bringula, rex_bringula@yahoo.com