Evaluation of ChatGPT-4o's and DeepSeek R1's responses to urological problems: A comparative study

HANBO LU¹Yusa Zhang²Zhan Wang¹Yang Zhao¹Jiang Liu¹Dongxu Qiu^1*Yu-Shi Zhang^1*

• ¹Department of Urology, Peking Union Medical College Hospital, Peking Union Medical College, Chinese Academy of Medical Sciences, Beijing, China
• ²Eight-year Program of Clinical Medicine, Chinese Academy of Medical Sciences, Peking Union Medical College, Beijing, P.R. China, Beijing, China

Medical professionals are increasingly utilizing artificial intelligence (AI) tools to improve learning and diagnostic processes. However, limited research has compared the performance of AI models in specific clinical fields such as urology. This study aimed to compare the performance of ChatGPT-4o and DeepSeek R1 in solving 809 single-choice urological questions from the Chinese National Qualification Examination for Attending Physicians in Urology. Both models were tested across three configurations, standard, advanced reasoning, and retrieval-augmented generation (RAG). Results showed that ChatGPT-4o achieved accuracy rates of 78.12%, 73.79%, and 78.99%, respectively, while DeepSeek R1 achieved higher rates of 83.19%, 81.46%, and 84.55%, with all differences being statistically significant (p < 0.001, Cohen's h = 0.129, 0.185 and 0.144). DeepSeek R1 also exhibited greater stability across reasoning modes, whereas ChatGPT-4o showed significant variability. Notably, DeepSeek R1 demonstrated superior performance in complex, case-based questions. These findings suggest that DeepSeek R1 outperforms ChatGPT-4o in both accuracy and stability, particularly in handling complex urological cases when evaluated in Chinese. These findings suggest that optimized AI models may support medical education and clinical decision-making, particularly in Chinese-language contexts, while cautioning that their performance in other languages or settings remains to be evaluated.