Medical knowledge of ChatGPT in public health, infectious diseases, COVID-19 pandemic, and vaccines: multiple choice questions examination based performance

Background At the beginning of the year 2023, the Chatbot Generative Pre-Trained Transformer (ChatGPT) gained remarkable attention from the public. There is a great discussion about ChatGPT and its knowledge in medical sciences, however, literature is lacking to evaluate the ChatGPT knowledge level in public health. Therefore, this study investigates the knowledge of ChatGPT in public health, infectious diseases, the COVID-19 pandemic, and its vaccines. Methods Multiple Choice Questions (MCQs) bank was established. The question’s contents were reviewed and confirmed that the questions were appropriate to the contents. The MCQs were based on the case scenario, with four sub-stems, with a single correct answer. From the MCQs bank, 60 MCQs we selected, 30 MCQs were from public health, and infectious diseases topics, 17 MCQs were from the COVID-19 pandemic, and 13 MCQs were on COVID-19 vaccines. Each MCQ was manually entered, and tasks were given to determine the knowledge level of ChatGPT on MCQs. Results Out of a total of 60 MCQs in public health, infectious diseases, the COVID-19 pandemic, and vaccines, ChatGPT attempted all the MCQs and obtained 17/30 (56.66%) marks in public health, infectious diseases, 15/17 (88.23%) in COVID-19, and 12/13 (92.30%) marks in COVID-19 vaccines MCQs, with an overall score of 44/60 (73.33%). The observed results of the correct answers in each section were significantly higher (p = 0.001). The ChatGPT obtained satisfactory grades in all three domains of public health, infectious diseases, and COVID-19 pandemic-allied examination. Conclusion ChatGPT has satisfactory knowledge of public health, infectious diseases, the COVID-19 pandemic, and its vaccines. In future, ChatGPT may assist medical educators, academicians, and healthcare professionals in providing a better understanding of public health, infectious diseases, the COVID-19 pandemic, and vaccines.


Introduction
The "Chatbot Generative Pre-Trained Transformer (ChatGPT), developed by OpenAI is an Artificial Intelligence (AI)" based tool that generates dialogs with humans.ChatGPT is developed with highly advanced technology to answer users' questions (1), swiftly clarify and provide the pleaded information.It is a valuable tool for getting ideas on any topic, article writing, and information searching for academic execution (2).At the beginning of 2023, ChatGPT received great attention among academicians and researchers as it offered attractive information for students, faculty, and educators.This is also a fact that it poses threats to the traditional structure of research and educational systems, and it may increase the chances of cheating on online exams and may minimize human rational, and cognitive abilities (3).
ChatGPT has shown impressive performance in various domains, including academia, research, and creative writing.However, challenges and concerns related to biases and trust persist (4).The science community and public views exist with some level of limitations about the adequacy and usage of ChatGPT in academia, research, and healthcare (5,6).ChatGPT has the potential to enhance learning and connect the stakeholders in academic and research settings.The potential of ChatGPT is not limited to the development of personalized and complex learning, assessments, and accuracy of the information.Still, there are several challenges that ChatGPT is facing in education.The problems linked with plagiarism, misuse or lack of knowledge, accountability, academic integrity, privacy, ethics, and professionalism (6).The controversies about ChatGPT and its implications in higher education are the major limitations to the implementation of ChatGPT in academia (7).
There is a great discussion about ChatGPT and its knowledge, but the literature is lacking in determining the knowledge level of ChatGPT in public health and infectious disease topics such as the COVID-19 pandemic and its vaccines.The "Severe Acute Respiratory Syndrome Coronavirus 2 (SARS-CoV-2)" also known as the COVID-19 pandemic is a highly challenging issue and a topic of great interest to the public, the science community, and policymakers (8).Worldwide, the pandemic has claimed the lives of over seven million people and has disturbed the economies, education systems, and daily lives (9).While major progress has been made in developing COVID-19 vaccines, but still the threat of new variants and the uneven distribution of these medical resources still pose continued challenges (10).The pandemic has affected marginalized communities, including low-income nations, racial and ethnic minorities, and those with pre-existing health conditions (11).In this situation, it is important to understand the need for equitable distribution of healthcare resources, improved accessibility to testing and treatment, and the prioritization of vulnerable populations in vaccination campaigns (12).The global situation due to the pandemic has exposed the global healthcare systems and the interconnectedness of the world.The COVID-19 pandemic has challenged global societies in unprecedented ways, and its impact on public health is highly essential to understand to combat such a global crisis (13).
The COVID-19 vaccines play a crucial role in controlling the coronavirus's spread and mitigating the pandemic's impact.The COVID-19 vaccines are incredibly important in the prevention of the COVID-19 pandemic, provide protection against the SARS-CoV-2 virus, reduction in hospitalizations and deaths.The people who are vaccinated are less likely to experience severe illness, and need hospitalization, and vaccines have demonstrated a highly effective in preventing severe cases and reducing the risk of mortality.It develops herd immunity, resumption of normalcy, protects vulnerable populations, reduces transmission, vaccines protect these vulnerable groups and decreases the risk of exposure, and limits virus mutations (14).
The MCQ pattern of examinations is important in assessing the knowledge in numerous disciplines of medical sciences (15).MCQs are well-acknowledged and widely used tools in medical education examinations which can assess the higher levels of learning strategies (16,17).Worldwide, various institutes are using the MCQs in medical sciences examinations to assess knowledge and skills.The MCQs help in better conceptual understanding developing clinical reasoning skills and better tools in medical and health care institutes (18,19).The case scenario-based MCQs assess higher-order cognitive skills (20).MCQs are better at assessing higher cognitive skills and critical thinking skills, deeper conceptual understanding, and evaluation of higher cognitive functions (21,22).At the beginning of the year 2023, worldwide ChatGPT received great attention for its role among students, faculty, academicians, and researchers.However, this is also a fact that with any technology including ChatGPT, there are some limitations and challenges to consider while using ChatGPT in educational settings.This study hypothesis proposes that ChatGPT may have the medical knowledge to achieve the appropriate grades in public health, infectious disease, and COVID-19 vaccines MCQ-based assessments.Therefore, the present study aimed to investigate the medical knowledge of ChatGPT in infectious diseases, the COVID-19 pandemic, and vaccines on multiple choice questions (MCQs) examination-based performance.

Research methodology 2.1 Multiple choice questions examination bank
In this study, the research team members established a Multiple-Choice Questions (MCQs) bank based on the subject information from "First Aid USMLE Step1; AMBOSS Step establishment of the MCQs questions bank pool, the investigators examined all the question contents and were satisfied that the MCQs were related to the subject contents.The questions were based on the case scenario, with four sub-stems and had a single correct answer.While establishing the MCQs bank, the types and difficulty levels of MCQs were checked, which is important for confirming the quality and accurate assessments of knowledge.Two senior faculty with a medical education background reviewed all the MCQs and analyzed the clarity, accuracy, and alignment with the contents.Moreover, any distractors and potential biases were also checked.The difficulty level, clarity, and potential test-taking issues were also evaluated.The complex wording, ambiguous language, or technical jargon were minimized.Based on these factors, the difficulty level of MCQs was examined to make the assessment more accurate, reliable, and valid.The research team was satisfied with the required standard of MCQs, and after that, the MCQs were transferred to the MCQs pool.The pattern was based on the questions required in medical education, such as, "The patient's condition is most probably caused by which of the following pathogens?What is the most appropriate dose of the COVID-19 vaccine?Which one of the following temperatures is required for the storage of the vaccine." The questions were scored according to the number of correct answers.The MCQs were cautiously checked for the related content, main stem, sub-stems, and answer keys.Any MCQ that had images were excluded from the study.We also excluded 03 MCQs from the pool which were used for the technical checking.After the establishment of the MCQs bank, we obtained 60 MCQs, 30 MCQs from infectious diseases topics, 17 MCQs from the COVID-19 pandemic, and 13 MCQs on COVID-19 vaccines.The tasks were given to evaluate the knowledge level of ChatGPT on MCQs.

ChatGPT response rate collection
The ChatGPT response rate on the MCQs was obtained between July 2, 2023, to September 20, 2023.The research team members manually entered MCQ one by one MCQ, and a new ChatGPT session was started for each entry to evade information-retaining prejudice.On each entry of MCQ, ChatGPT provided the answers with explanations.The first response of ChatGPT was considered final, and no attempt was made as a choice of "regenerate response." As per the established answer key, grades were calculated on a scale of 0-1, zero representing incorrect and one representing correct answer.

Statistical analysis
The data were analyzed using descriptive statistical tests and expressed as numbers and percentages.The comparison between right and wrong MCQs was assessed.The p-values were calculated using the binomial test on Statistical Package for the Social Sciences (SPSS 29.0.1.0).The binomial test is useful as it deals with categorical data where there are possible outcomes such as one MCQS was right or three were wrong.To calculate a p-value using the binomial test, the steps followed were to formulate hypotheses, choose the significance level, count the number of right and wrong, and calculate the probability.This is done using the binomial probability formula.Since all the MCQs had four options, this made the chance of getting the question right 25%.A calculated p-value of less than 0.05 was considered statistically significant.

Results
The ChatGPT knowledge was assessed on the individual MCQs in public health, infectious diseases, the COVID-19 pandemic, and vaccines.The MCQs were comprehensive, highly standardized questions that covered assorted topics with basic, clinical concepts and facts in the field of public health and infectious diseases, the COVID-19 pandemic, and vaccines.A total of 60 Multiple Choice Questions (MCQs) on infectious diseases (30 MCQs) the COVID-19 pandemic (17 MCQs) and vaccines (13 MCQs) were randomly selected, prepared based on the information available in various national, and international examination pools, textbooks, facts sheets and the task were given to evaluate the ChatGPT knowledge level (Table 1).2; Figure 1).
The analysis further showed that the p-values were less than (p = 0.05), indicating that the observed results of the correct answers in each section were significantly higher than what would be expected by chance alone if the answers were guessed at random (Table 2).
ChatGPT obtained a satisfactory score in all three domains of COVID-19 pandemic exams with no focused instruction or guided support.These results highlight the role of ChatGPT, it may assist health professionals and policymakers in assisting such pandemicallied situations.

Discussion
ChatGPT is a novel accomplishment of this most modern and technology-based highly advanced world.It achieved great attention globally as it rapidly responds to questions and provides appropriate answers on the required topic.It is useful for enhancing knowledge, providing explanations, offering suggestions, creating conversational dialogs, and assisting with multiple tasks (23).Worldwide, people are searching for updated knowledge of the COVID-19 pandemic hence the use of ChatGPT has increased as information informationproviding tool.However, the literature is lacking on whether ChatGPT provides correct information or not.Therefore, the present study investigated the ChatGPT knowledge in public health, infectious diseases, the COVID-19 pandemic, and its vaccines.The present study findings reveal that ChatGPT achieved a good grade in MCQs-based examination in public health, infectious diseases, the COVID-19 pandemic, and its allied vaccines.The analytic thinking approach in medical education is fundamental to producing proficient physicians.Hence, at regional and international levels, medical educationalists and examination bodies introduced MCQ-based examination systems to evaluate knowledge and skills (18).The higher cognitive knowledge in medical education is based on the MCQ examinations (19).MCQs play a pivotal role in assessing thinking capabilities, critical analyzing skills, and problem-solving abilities (21,22).MCQs examine the capability to connect concepts and analyze evidence across copious contexts.MCQs require critical thinking and analytical skills, higherorder thinking skills, which are crucial for success in higher education and professional endeavors (21,22).Since the introduction of ChatGPT, in November 2022, few studies have been published and assessed the ChatGPT knowledge in medical sciences.Passby et al. (24) reported that ChatGPT-3.5 obtained an overall score of 63.0%, answered the clinical questions and achieved passing marks.Duong et al. (25) investigated the capability of ChatGPT in human genetics.The overall performance was almost similar to human performance.The human response was 67.0% and "The ChatGPT response was 68.0%.Similarly, Wang et al. (26) found that the correct response rate of ChatGPT in Chinese and English questions was 54.4 and 56.9% in the first stage, and 53.8 and 67.6% in the second stage." Suchman et al. ( 27) appraised the ChatGPT-3 knowledge and scored 65.0% and GPT-4 scored 62.0%.The authors highlighted its limited role in medical education.In another study, Humar et al. (28) examined the ChatGPT's score in examination compared to working trainee students.It was found that ChatGPT answered 55.8% correctly, and performance was similar to that of firstyear resident trainees.
Gilson et al. (29) found that ChatGPT obtained 44, 42, 64, and 58% scores in United States Medical Licensing Examination Step 1 and 2, respectively.The grades were significantly decreased as questions became more difficult.Similarly, Das et al. (30) observed that ChatGPT scored 80% in microbiology.In Korea, Huh et al. (31) performed an examination and compared the ChatGPT knowledge level with medical students in Korea.The overall performance of ChatGPT was lesser than the medical students.More recently, Meo and Al-Masri (32) reported that ChatGPT obtained 72% marks, a reasonable score in basic and clinical medical sciences MCQs-based examination.However, in another study, Meo and Al-Khlaiwi et al. (33) compared the Bard and ChatGPT knowledge in three different topics including endocrinology, diabetes, and diabetes technology through MCQ examination.The authors found that both ChatGPT and Bard did not achieve appropriate scores in these important subjects.ChatGPT opens multiple paths for developing knowledge with reasonings.ChatGPT provides accurate information, however, it might generate incorrect answer responses.The most probable reason is that the information is an outdated misinterpretation of complex questions, lengthy questions, and formulas (32, 33).
Similarly, Farhat et al. (34) provided valuable insights into the performance of ChatGPT-3.5,ChatGPT-4, and Bard in answering the questions.The authors reported that CHAT GPT-4 is the perfect model, highlighting its potential role in education.The literature established a benchmark for evaluating and enhancing performance in educational tasks, and its use in diverse learning environments (35).Choi (36) conducted a study using about 80 question test items from the Korean Comprehensive Basic Medical Sciences Examination, ChatGPT responded with an accuracy of 76.0%.ChatGPT was able to gather, review, and generate credible text relevant to public health and allied content.In another study, Wang et al. (37) estimated the accuracy of COVID-19 information by ChatGPT 3.5 and 4.0.The authors found that ChatGPT 3.5 and 4.0 can produce correct and appropriate COVID-19 information to a certain extent, but compared to global health organizations' responses, gaps and deficiencies exist.

Implications in educational settings
The importance and implications of AI tools are swiftly increasing in various sections of society (34).In this context, this study's findings are vital for understanding the ChatGPT knowledge for its implementation in educational settings (38).The assessment of ChatGPT knowledge could be instrumental in providing information and shaping people's perceptions about public health allied issues such as the COVID-19 pandemic and vaccines.ChatGPT offers opportunities to enhance learning, research, assessment, and various spheres of educational settings.However, it's essential to address ethical, quality assurance, and training considerations to maximize its benefits while mitigating potential risks (32).

Study strengths and limitations
Similar to other studies, this study has some strengths and limitations.The first strength is that this is a novel and interesting study that evaluates the ChatGPT knowledge on a particularly important topic of public health, infectious diseases, the COVID-19 pandemic, and its vaccines.Second, understanding the role of ChatGPT in public health, the COVID-19 pandemic and vaccine knowledge is the need of time, as ChatGPT in future may provide proper knowledge which is an acute need of the public in various fields of medical sciences including public health sciences.Third, the assessment was based on well-constructed and highly standardized MCQs.This study has some limitations.The analysis was limited by the limited size of MCQs, and the tool must be assessed in both realworld and controlled settings and its comparison with similar levels of students.

Conclusion
The knowledge of ChatGPT in public health, infectious diseases, the COVID-19 pandemic, and vaccines is adequate.The ChatGPT obtained overall satisfactory grades in all the domains of the COVID-19 pandemic-allied MCQs.The findings indicate that ChatGPT may have the potential to assist medical educationists, academicians, healthcare professionals and policymakers in providing a better understanding of infectious diseases, the COVID-19 pandemic and COVID-19 pandemic vaccines.ChatGPT could provide the latest information on public health, disease surveillance, early warnings and identify disease outbreaks.It may minimize misleading information, help to combat the pandemic, improve understanding and adherence to public health strategies and also enhance public trust.There are some challenges and gaps between the accuracy and clarity of the responses generated by ChatGPT.Similar large sample-sized studies are required to further validate the ChatGPT knowledge and effectiveness in public health, infectious diseases, the COVID-19 pandemic, and vaccines.
1, Harrison's infectious disease book, Climate and COVID-19 pandemic book, Vaccines Fact Sheet, World Health Organization, Centers for Disease Control and Prevention, Food and Drug Authorities USA, Elsevier COVID-19 Resource Centre, and the university examination pools." After the

Table 2
demonstrates the knowledge level of ChatGPT on the established set of MCQs.The grades achieved on MCQs are shown in

TABLE 1
Distribution of MCQs about public health, infectious diseases, the COVID-19 pandemic, and vaccines.

TABLE 2
Marks obtained by ChatGPT on MCQs in public health, infectious diseases, COVID-19 pandemic, and its vaccines.Marks obtained by ChatGPT in public health, infectious diseases, and COVID-19 pandemic and vaccines examination.