The availability of medical information on the internet has transformed the dynamic of the patient-physician relationship. Patients are relying more and more on online resources to address their concerns. In November 2022, a dialogue-based artificial intelligence (AI) large language model (ChatGPT) was made accessible to the general public and has since gained millions of daily users (1, 2). Despite its capabilities in handling complex writing tasks (3), the use of chat-based AI in medicine is still in its early stages and is not intended for medical use yet. However, recent studies suggest a heightened potential for such technology to assist in patient education and provide supplementary information on various medical topics (4–10). We aimed to assess the ability of the AI model to provide accurate responses to evidence-based commonly asked patient-centered oncology questions.

We convened a team of three oncologists from different institutions with extensive clinical and patient care experience to develop a set of thirty questions targeting common oncology topics. The questions focused on risk factors, preventive measures, diagnosis, treatment and side effects, and were designed in alignment with guideline-based topics, the oncologists’ clinical experience in both inpatient and outpatient settings and evidence based question prompt lists for patients seeing a medical or radiation oncologist (11, 12). The study was conducted in February 2023 using ChatGPT 3.5 (default model). The same questions were submitted to the online AI model three times to ensure consistency. Each set of 30 answers was assigned to 1 of the 3 oncologists. Each oncologist qualitatively graded each answer based on its relevance to current published preventive and management guidelines (13, 14) and their clinical expertise. They classified each response as either ‘accurate’, ‘inaccurate’, or ‘harmful’. An answer was considered accurate if – according to the oncologist – it contained essential and appropriate information, inaccurate if it provided false or incomplete information, and harmful if it included potentially harmful information for the patient. The oncologists’ assessments were then reviewed, and their evaluations combined. Evidently, to limit subjectivity and selection bias, an answer was deemed ‘accurate’ if all three oncologists judged it to be accurate, ‘inaccurate’ if at least one oncologist deemed it inaccurate, and ‘harmful’ if at least one oncologist deemed it harmful.

AI model responses were considered “accurate” in 26 of 30 questions (86%). 4 responses (14%) were considered “inaccurate”, and none were considered “harmful” ( Table 1 ). For example, when asked about hair loss as a side effect of immunotherapy, the AI model did not provide an adequate response and gave false but non-harmful information. Nonetheless, when asked about preventive strategies and screening recommendations for various types of cancer, such as breast lung cancer and colorectal cancer, the AI was in line with current recommendations. The average response time was approximately 0.18 seconds per word.

We reported that a popular online chat-based AI model provided mostly accurate responses to many oncology-related questions as evaluated by experienced oncologists. These findings suggest the potential of interactive AI in assisting in patient education. This could be particularly beneficial for oncology patients who may have complex and diverse questions regarding their condition. For example, the use of AI in this setting could potentially alleviate the need for patients to search for answers across numerous websites, which can often be overwhelming, time-consuming, and provide conflicting information. Nonetheless, the added value of such AI-based online chat remains heavily debated. With 14% of answers being inaccurate, the risk of receiving incorrect recommendations is significant. While future medically validated AI-based online chat could eventually provide patients with accurate supplementary information and resources, it is important for patients to understand that these technologies cannot replace the essential role of patient-physician interaction in making informed decisions about treatment options and managing care.

This study has several limitations. Firstly, it is important to note that the currently used AI model used is not intended for actual medical purposes. Secondly, oncology being a complex field, it cannot be fully covered by the limited set of questions used in this study. Thirdly, the accuracy and reliability of AI are susceptible to limitations and biases in the training data provided by the developers, which may result in incorrect or outdated information being provided. Fourthly, the evaluation of the responses was qualitative and conducted by a group of oncologists who may not represent the diverse views and opinions of healthcare professionals across different settings. Fifthly, inter-reviewer agreement and heterogeneity between the set of 3 AI responses was not assessed in detail. Finally, the AI responses did not include references to supporting evidence.

Future research should focus on developing a more standardized system for grading responses and assessing accuracy through a clear comparative analysis between physicians’ answers and AI-generated answers, as well as evaluating the usefulness of such tools in supporting physician practice.

JK: Drafted the manuscript. All authors contributed to the editing and writing. All authors have read and approved the manuscript. All authors contributed to the article and approved the submitted version.