When AI Speaks Like a Specialist: ChatGPT-4 in the Management of Inflammatory Bowel Disease

Elena De Cristofaro¹Francesca Zorzi¹Maria Abreu²Alice Colella¹Giovanna Del Vecchio Blanco³Gionata Fiorino⁴Elisabetta Lolli¹Nurulamin M Noor⁵Loris Riccardo Lopetuso⁶Mathieu Pioche⁷Jean Grimaldi⁷Omero Alessandro Paoluzi¹Joana Roseira⁸Giorgia Sena¹Edoardo Troncone¹Emma Calabrese³Giovanni Monteleone³Irene Marafini^1,3*

• ¹Policlinico Tor Vergata, Rome, Italy
• ²University of Miami Miller School of Medicine, Miami, United States
• ³Universita degli Studi di Roma Tor Vergata, Rome, Italy
• ⁴Azienda Ospedaliera San Camillo Forlanini, Rome, Italy
• ⁵Cambridge University Hospitals NHS Foundation Trust, Cambridge, United Kingdom
• ⁶Fondazione Policlinico Universitario Agostino Gemelli IRCCS CEMAD, Rome, Italy
• ⁷Hopital Edouard Herriot, Lyon, France
• ⁸Unidade Local de Saude do Algarve, Faro, Portugal

Abstract Background: Artificial intelligence (AI) is gaining traction in healthcare, especially for patients' education. Inflammatory bowel diseases (IBD) require continuous engagement, yet the quality of online information accessed by patients is inconsistent. ChatGPT, a generative AI model, has shown promise in medical scenarios, but its role in IBD communication needs further evaluation. The objective of this study was to assess the quality of ChatGPT-4's responses to common patient questions about IBD, compared to those provided by experienced IBD specialists. Methods: Twenty-five frequently asked questions were collected during routine IBD outpatient visits and categorized into five themes: pregnancy/breastfeeding, diet, vaccinations, lifestyle, and medical therapy/surgery. Each question was answered by ChatGPT-4 and by two expert gastroenterologists. Responses were anonymized and evaluated by 12 physicians (6 IBD experts and 6 non-experts) using a 5-point Likert scale across four dimensions: accuracy, reliability, comprehensibility, and actionability. Evaluators also attempted to identify whether responses were AI-or human-generated. Results: ChatGPT-4 responses received significantly higher overall scores than those from human experts (mean 4.28 vs. 4.05; p < 0.001). The best-rated scenarios were medical therapy and surgery; the diet scenario consistently received lower scores. Only 33% of AI-generated responses were correctly identified as such, indicating strong similarity to human-written answers. Both expert and non-expert evaluators rated AI responses highly, though IBD specialists gave higher ratings overall. Conclusion: ChatGPT-4 generated high-quality, clear, and actionable responses to IBD-related patient questions, often outperforming human experts. Its outputs were frequently indistinguishable from those written by physicians, suggesting potential as a supportive tool for patient education. Nonetheless, further studies are needed to assess real-world application and ensure appropriate use in personalized clinical care.