Evaluation and Comparison of Large Language Models' Responses to Questions Related Optic Neuritis

Han-Jie He^1,2Fang-Fang Zhao¹Jia-Jian Liang¹Yun Wang¹Qian- Qian He^1,2Hongjie Lin¹Jingyun Cen³Feifei Chen¹Tai- Ping Li¹Zhanchi Hu⁴Jian-Feng Yang¹Lan Chen¹Carol Y Cheung⁵Yih Chung Tham^6,7,8Ling-Ping Cen^1,2,4,9*

• ¹Joint Shantou International Eye Centre of Shantou University and The Chinese University of Hong Kong, Shantou, China
• ²College of Medicine, Shantou University, Shantou, Guangdong Province, China
• ³Medical College, Shaoguan University, Shaoguan, Guangdong Province, China
• ⁴Guangming Eye Hospital, Dongguan, China
• ⁵Department of Ophthalmology and Visual Sciences, Faculty of Medicine, The Chinese University of Hong Kong, Shatin, Hong Kong Region, China
• ⁶Yong Loo Lin School of Medicine, National University of Singapore, Singapore, Singapore
• ⁷Singapore Eye Research Institute (SERI), Singapore, Singapore
• ⁸Centre of Innovation and Precision Eye Health, Department of Ophthalmology, Yong Loo Lin School of Medicine, National University of Singapore and National University Health System, Singapore, Singapore
• ⁹Guangdong Provincial Key Laboratory of Medical Immunology and Molecular Diagnostics, School of Medical Technology, Guangdong Medical University, Dongguan, China

Objective: Large language models (LLMs) show promise as clinical consultation tools and may assist optic neuritis patients, though research on their performance in this area is limited. Our study aims to assess and compare the performance of four commonly used LLM-Chatbots-Claude-2, ChatGPT-3.5, ChatGPT-4.0, and Google Bard-in addressing questions related to optic neuritis.We curated 24 optic neuritis-related questions and had three ophthalmologists rate the responses on two three-point scales for accuracy and comprehensiveness. We also assessed readability using four scales. The final results showed performance differences among the four LLM-Chatbots.The average total accuracy scores (out of 9): ChatGPT-4.0 (7.62 ± 0.86), Google Bard (7.42 ± 1.20), ChatGPT-3.5 (7.21 ± 0.70), Claude-2 (6.44 ± 1.07). ChatGPT-4.0 (p = 0.0006) and Google Bard (p = 0.0015) were significantly more accurate than Claude-2. Also, 62.5% of ChatGPT-4.0's responses were rated "Excellent", followed by 58.3% for Google Bard, both higher than Claude-2's 29.2% (all p ≤ 0.042) and ChatGPT-3.5's 41.7%. Both Claude-2 and Google Bard had 8.3% "Deficient" responses. The comprehensiveness scores were similar among the four LLMs (p = 0.1531). Note that all responses require at least a university-level reading proficiency.LLM-Chatbots hold immense potential as clinical consultation tools for optic neuritis, but they require further refinement and proper evaluation strategies before deployment to ensure reliable and accurate performance.