Comparative Performance Evaluation of Large Language Models in Answering Esophageal Cancer-Related Questions: A Multi-Model Assessment Study

Zijie He¹Lilan Zhao¹Genglin Li¹Jintao Wang¹Songyu Cai¹Pengjie Tu¹Jingbo Chen²Jianman Wu²Juan Zhang¹Ruiqi Chen¹Yangyun Huang¹Xiaojie Pan¹Wenshu Chen^1*

• ¹Department of Thoracic Surgery, Fujian Provincial Hospital, Fuzhou, Fujian, China
• ²Fujian Provincial Hospital, Fuzhou, China

Background: Esophageal cancer has high incidence and mortality rates, leading to increased public demand for accurate information. However, the reliability of online medical information is often questionable. This study systematically compares the accuracy, completeness, and comprehensibility of mainstream large language models (LLMs) in answering esophageal cancer-related questions. Methods: 65 questions covering fundamental knowledge, preoperative preparation, surgical treatment, and postoperative management were selected. Each model—ChatGPT 5, Claude Sonnet 4.0, DeepSeek-R1, Gemini 2.5 Pro, and Grok-4—was queried independently using standardized prompts. Five senior clinical experts, including three thoracic surgeons, one radiologist, and one medical oncologist, evaluated responses using a five-point Likert scale. A retesting mechanism was applied for low-scoring responses, and the intraclass correlation coefficient (ICC) assessed rating consistency. Statistical analyses included the Friedman test, Wilcoxon signed-rank test, and Bonferroni correction. Results: All models performed well, with average scores exceeding 4.0. However, significant differences emerged: Gemini excelled in accuracy, while ChatGPT led in completeness, particularly in surgical and postoperative contexts. Minor differences appeared in fundamental knowledge but notable disparities were found in complex areas. Retesting showed improvements in overall quality, yet some responses showed decreased completeness and relevance. Conclusion: Large language models demonstrate considerable potential in esophageal cancer question and answer, with significant differences in completeness. ChatGPT is more comprehensive in complex scenarios, while Gemini excels in accuracy. This study offers guidance for selecting AI tools in clinical settings, advocating for a tiered application strategy tailored to specific scenarios and highlighting the importance of user education to clarify the limitations and applicability of LLMs.