Assessing ChatGPT versus evidence-based online responses for polycystic ovary syndrome (PCOS) self-management and education: an international cross-sectional blinded survey of healthcare professionals

Sandro Graca^1*Alexander Dallaway^2,3Folashade Alloh²Harpal S. Randeva^3,4,5,6Chris Kite^2,3,7Ioannis Kyrou^3,4,5,6,8,9

• ¹University of Wolverhampton, School of Health and Wellbeing, Faculty of Education, Health and Wellbeing, Wolverhampton, United Kingdom
• ²School of Health and Wellbeing, Faculty of Education, Health and Wellbeing, University of Wolverhampton, Wolverhampton, United Kingdom
• ³Warwickshire Institute for the Study of Diabetes, Endocrinology and Metabolism (WISDEM), University Hospitals Coventry and Warwickshire NHS Trust, Coventry, United Kingdom
• ⁴Warwick Medical School, University of Warwick, Coventry, United Kingdom
• ⁵Centre for Sport, Exercise and Life Sciences, Research Institute for Health & Wellbeing, Coventry University, Coventry, United Kingdom
• ⁶Institute for Cardiometabolic Medicine, University Hospitals Coventry and Warwickshire NHS Trust, Coventry, United Kingdom
• ⁷Division of Public Health, Sport and Wellbeing, Faculty of Health, Medicine and Society, University of Chester, Chester, United Kingdom
• ⁸Aston Medical School, College of Health and Life Sciences, Aston University, Birmingham, United Kingdom
• ⁹College of Health, Psychology and Social Care, University of Derby, Derby, United Kingdom

Artificial intelligence (AI)-powered large language models, such as ChatGPT, are increasingly used by the public for health information. The reliability of such novel AI-tools in providing credible polycystic ovary syndrome (PCOS) information/advice requires investigation. Healthcare professionals involved in PCOS care (n=43 from 14 countries) used a 5-point Likert scale to evaluate ChatGPT-generated responses to frequently asked questions about PCOS against the corresponding patient-orientated, evidence-based recommendations/responses from AskPCOS. ChatGPT responses were rated significantly higher than the evidence-based responses for 11 of the 12 study questions, with moderate to large effect sizes (rrb = -0.46 to - 1.00; all p-values <0.05), with ChatGPT answers being rated on average 0.824 units higher. Scoring agreement varied (poor to fair), with seven questions showing statistically fair agreement (κ=0.24-0.37, p<0.05). Readability analyses found no statistically significant differences between ChatGPT and evidence-based responses. However, using ChatGPT for simplifying the responses resulted in significant improvement. ChatGPT holds potential as a complementary patient self-education tool in PCOS, capable of interactive engagement and simplifying medical language. Further research is needed to identity optimal integration of AI tools and validate their clinical applicability for PCOS self-education/management.