ChatGPT-4o, Gemini Advanced and DeepSeek R1 in Preoperative Decision-Making for Thyroid Surgery: A Comparative Assessment with Human Surgeons

Long ZouPeng ZhangYu-Qi JiangXiaowen WangXi-Jing YanJie-Zhong WuJia QiWenchao LiQing-Qing CaiZhi-Rong XuanKunpeng Hu^*

• The Third Affiliated Hospital of Sun Yat-Sen University, Guangzhou, China

The integration of large language models (LLMs) into surgical decision-making is an emerging field with potential clinical value. This study assessed the preoperative decision-making consistency of ChatGPT-4o, Gemini Advanced, and DeepSeek R1 in comparison with expert consensus, using clinical data from 123 patients undergoing thyroid surgery. Overall concordance rates were 47.97% for ChatGPT-4o, 24.39% for Gemini Advanced, and 56.10% for DeepSeek R1. In thyroidectomy extent decisions, all three models showed moderate consistency with the surgical team, with agreement rates of 61.79% (κ=0.484) for ChatGPT-4o, 67.48% (κ=0.548) for Gemini, and 67.48% (κ=0.535) for DeepSeek R1 (all p < 0.001). However, significant divergence was observed in lymph node dissection planning: ChatGPT-4o achieved a high concordance rate of 69.11% (κ =0.616), DeepSeek R1 showed the highest at 79.67% (κ=0.741), while Gemini's performance was relatively poor at 34.96% (κ=0.188). Though our findings demonstrate that ChatGPT-4o and DeepSeek R1 exhibit substantial agreement with experienced surgeons in preoperative planning, overall performance still leaves room for improvement. Nevertheless, model-specific variability — particularly in oncologic decision-making—highlights the need for refinement and robust clinical validation before widespread clinical adoption.