DiaGuide-LLM - Using large language models for patient-specific education and health guidance in diabetes

Kristin Skjervold¹Henriette Nordahl Sævig¹Helge Ræder^2,3Arvid Lundervold^1*Alexander Selvikvåg Lundervold⁴

• ¹Universitetet i Bergen Institutt for Biomedisin, Bergen, Norway
• ²Universitetet i Bergen, Bergen, Norway
• ³Haukeland Universitetssjukehus, Bergen, Norway
• ⁴Norges Handelshoyskole, Bergen, Norway

Effective diabetes care relies on communication, patient empowerment, and lifestyle management. However, rising prevalence and workforce shortages challenge current care models. Large language models (LLMs) have the potential to support healthcare delivery by providing personalized health information. While prior studies show promising results, few have compared LLM-generated responses with those from healthcare professionals in chronic disease contexts, particularly from end-users´ perspectives. This study compared GPT-4o and healthcare professional responses to diabetes-related questions, evaluating them on knowledge, helpfulness, and empathy. It also explored correlations between these qualities and differences based on participants' educational background. Using a cross-sectional experimental design, 1810 evaluations were collected through an online questionnaire (November 2024 – January 2025). Participants rated responses on 5-point Likert scales for knowledge, helpfulness, and empathy. For all metrics combined, GPT-4o received higher ratings in 46.7% of evaluations [95% CI: 28.8%–64.5%], while healthcare professionals were preferred in 23.3% [95% CI: 8.2%–38.5%]. Participants with lower education levels rated GPT-4o significantly higher across all dimensions, while those with ≥4 years of higher education rated it higher for empathy and helpfulness. Quality measures were strongly correlated. Although differences were statistically significant, the observed effect sizes were small and should be interpreted as modest in practical terms. These findings assess perceived quality and accessibility of healthcare communication from end-user perspectives and suggest that LLMs may enhance the perceived quality and accessibility of healthcare communication, particularly among individuals with lower educational attainment. Skjervold et al. DiaGuide-LLM Further research is needed to determine their appropriate role in clinical practice, including objective assessment of clinical accuracy.