AUTHOR=Ye Yi , Zheng En-dian , Lan Qiao-li , Wu Le-can , Sun Hao-yue , Xu Bei-bei , Wang Ying , Teng Miao-miao 

TITLE=Comparative evaluation of the accuracy and reliability of ChatGPT versions in providing information on Helicobacter pylori infection

JOURNAL=Frontiers in Public Health

VOLUME=Volume 13 - 2025

YEAR=2025

URL=https://www.frontiersin.org/journals/public-health/articles/10.3389/fpubh.2025.1566982

DOI=10.3389/fpubh.2025.1566982

ISSN=2296-2565

ABSTRACT=ObjectiveThis study aimed to evaluate the accuracy and reliability of responses provided by three versions of ChatGPT (ChatGPT-3.5, ChatGPT-4, and ChatGPT-4o) to questions related to Helicobacter pylori (Hp) infection, as well as to explore their potential applications within the healthcare domain.MethodsA panel of experts compiled and refined a set of 27 clinical questions related to Hp. These questions were presented to each ChatGPT version, generating three distinct sets of responses. The responses were evaluated and scored by three gastroenterology specialists utilizing a 5-point Likert scale, with an emphasis on accuracy and comprehensiveness. To assess response stability and reliability, each question was submitted three times over three consecutive days.ResultsStatistically significant differences in the Likert scale scores were observed among the three ChatGPT versions (p < 0.0001). ChatGPT-4o demonstrated the best performance, achieving an average score of 4.46 (standard deviation 0.82) points. Despite its high accuracy, ChatGPT-4o exhibited relatively low repeatability. In contrast, ChatGPT-3.5 exhibited the highest stability, although it occasionally provided incorrect answers. In terms of readability, ChatGPT-4 achieved the highest Flesch Reading Ease score of 24.88 (standard deviation 0.44), however, no statistically significant differences in readability were observed among the versions.ConclusionAll three versions of ChatGPT were effective in addressing Hp-related questions, with ChatGPT-4o delivering the most accurate information. These findings suggest that artificial intelligence-driven chat models hold significant potential in healthcare, facilitating improved patient awareness, self-management, and treatment compliance, as well as supporting physicians in making informed medical decisions by providing accurate information and personalized recommendations.