Diagnostic Efficacy of Large Language Models in The Pediatric Emergency Department: A Pilot Study

Francesco Del Monte¹Roberta Barolo²Maria Circhetta³Angelo Giovanni Delmonaco¹Emanuele Castagno^1*Emanuele Pivetta⁴Letizia Bergamasco⁵Matteo Franco⁶Gabriella Olmo³Claudia Bondone¹

• ¹Department of Pediatric Emergency, Ospedale Infantile Regina Margherita, Turin, Italy
• ²Department of Public Health and Pediatrics, Università degli Studi di Torino, Turin, Italy
• ³Department of Control and Computer Engineering, Politecnico di Torino, Turin, Italy
• ⁴Division of Emergency Medicine and High Dependency Unit, Department of Medical Sciences, Università degli Studi di Torino, Turin, Italy
• ⁵Links Foundation, Turin, Italy
• ⁶Department of Clinical and Biological Sciences, Università degli Studi di Torino, Turin, Italy

The Pediatric Emergency Department (PED) faces significant challenges, such as high patient volumes, time-sensitive decisions, and complex diagnoses. Large Language Models (LLMs) have the potential to enhance patient care; however, their effectiveness in supporting the diagnostic process remains uncertain, with studies showing mixed results regarding their impact on clinical reasoning. We aimed to assess LLM-based chatbots performance in realistic PED scenarios, and to explore their use as diagnosis-making assistants in pediatric emergency. Methods: We evaluated the diagnostic effectiveness of 5 LLMs (ChatGPT-4o, Gemini 1.5 Pro, Gemini 1.5 Flash, Llama-3-8B, and ChatGPT-4o mini) compared to 23 physicians (including 10 PED physicians, 6 PED residents, and 7 Emergency Medicine residents). Both LLMs and physicians had to provide one primary diagnosis and two differential diagnoses for 80 realpractice pediatric clinical cases from the PED of a tertiary care Children's Hospital, with three different levels of diagnostic complexity. The responses from both LLMs and physicians were compared to the final diagnoses assigned upon patient discharge; two independent experts evaluated the answers using a five-level accuracy scale. Each physician or LLM received a total score out of 80, based on the sum of all answer points. Results: The best performing chatbots were ChatGPT-4o (score: 72.5) and Gemini 1.5 Pro (score: 62.75), the first performing better (p<0.05) than PED physicians (score: 61.88). Emergency Medicine residents performed worse (score: 43.75) than both the other physicians and chatbots (p<0.01). Chatbots' performance was inversely proportional to case difficulty, but ChatGPT-4o managed to match the majority of the correct answers even for highly difficult cases. Discussion: ChatGPT-4o and Gemini 1.5 Pro could be a valid tool for ED physicians, supporting clinical decision-making without replacing the physician's judgment. Shared protocols for effective collaboration between AI chatbots and healthcare professionals are needed.