Department of Cardiology, Shanghai General Hospital, Shanghai Jiao Tong University School of Medicine, China
YH
Yuhui Huang
Department of Cardiology, Shanghai General Hospital, Shanghai Jiao Tong University School of Medicine, China
CH
Chen He
Cardiac Arrhythmia Center, Fuwai Hospital, National Center for Cardiovascular Diseases, Peking Union Medical College and Chinese Academy of Medical Sciences, China
Cardiac Arrhythmia Center, Fuwai Hospital, National Center for Cardiovascular Diseases, Peking Union Medical College and Chinese Academy of Medical Sciences, China
Guangdong Provincial Key Laboratory of Autophagy and Major Chronic Non-communicable Diseases, Key Laboratory of Prevention and Management of Chronic Kidney Disease of Zhanjiang City, Department of Nephrology, National Clinical Key Specialty Construction Program (2023), Institute of Nephrology, Affiliated Hospital of Guangdong Medical University, China
ZH
Zhiqing Huang
Guangdong Provincial Key Laboratory of Autophagy and Major Chronic Non-communicable Diseases, Key Laboratory of Prevention and Management of Chronic Kidney Disease of Zhanjiang City, Department of Nephrology, National Clinical Key Specialty Construction Program (2023), Institute of Nephrology, Affiliated Hospital of Guangdong Medical University, China
Guangdong Provincial Key Laboratory of Autophagy and Major Chronic Non-communicable Diseases, Key Laboratory of Prevention and Management of Chronic Kidney Disease of Zhanjiang City, Department of Nephrology, National Clinical Key Specialty Construction Program (2023), Institute of Nephrology, Affiliated Hospital of Guangdong Medical University, China
HZ
Hongjiu Zhou
Guangdong Provincial Key Laboratory of Autophagy and Major Chronic Non-communicable Diseases, Key Laboratory of Prevention and Management of Chronic Kidney Disease of Zhanjiang City, Department of Nephrology, National Clinical Key Specialty Construction Program (2023), Institute of Nephrology, Affiliated Hospital of Guangdong Medical University, China
Guangdong Provincial Key Laboratory of Autophagy and Major Chronic Non-communicable Diseases, Key Laboratory of Prevention and Management of Chronic Kidney Disease of Zhanjiang City, Department of Nephrology, National Clinical Key Specialty Construction Program (2023), Institute of Nephrology, Affiliated Hospital of Guangdong Medical University, China
Guangdong Provincial Key Laboratory of Autophagy and Major Chronic Non-communicable Diseases, Key Laboratory of Prevention and Management of Chronic Kidney Disease of Zhanjiang City, Department of Nephrology, National Clinical Key Specialty Construction Program (2023), Institute of Nephrology, Affiliated Hospital of Guangdong Medical University, China
Guangdong Provincial Key Laboratory of Autophagy and Major Chronic Non-communicable Diseases, Key Laboratory of Prevention and Management of Chronic Kidney Disease of Zhanjiang City, Department of Nephrology, National Clinical Key Specialty Construction Program (2023), Institute of Nephrology, Affiliated Hospital of Guangdong Medical University, China
BackgroundGenerative artificial intelligence (AI) tools are increasingly being used as “ambient scribes” to generate drafts for clinical notes from patient encounters. Despite rapid adoption, few studies have systematically evaluated the quality of AI-generated documentation against physician standards using validated frameworks.ObjectiveThis study aimed to compare the quality of large language model (LLM)-generated clinical notes (“Ambient”) with physician-authored reference (“Gold”) notes across five clinical specialties using the Physician Documentation Quality Instrument (PDQI-9) as a validated framework to assess document quality.MethodsWe pooled 97 de-identified audio recordings of outpatient clinical encounters across general medicine, pediatrics, obstetrics/gynecology, orthopedics, and adult cardiology. For each encounter, clinical notes were generated using both LLM-optimized “Ambient” and blinded physician-drafted “Gold” notes, based solely on audio recording and corresponding transcripts. Two blinded specialty reviewers independently evaluated each note using the modified PDQI-9, which includes 11 criteria rated on a Likert-scale, along with binary hallucination detection. Interrater reliability was assessed using within-group interrater agreement coefficient (RWG) statistics. Paired comparisons were performed using t-tests or Mann–Whitney tests.ResultsPaired analysis of 97 clinical encounters yielded 194 notes (2 per encounter) and 388 paired reviews. Overall, high interrater agreement was observed (RWG > 0.7), with moderate concordance noted in pediatrics and cardiology. Gold notes achieved higher overall quality scores (4.25/5 vs. 4.20/5, p = 0.04), as well as superior accuracy (p = 0.05), succinctness (p < 0.001), and internal consistency (p = 0.004) compared to ambient notes. In contrast, ambient notes scored higher in thoroughness (p < 0.001) and organization (p = 0.03). Hallucinations were detected in 20% of gold notes and 31% of ambient notes (p = 0.01). Despite these limitations, reviewers overall preferred ambient notes (47% vs. 39% for gold).ConclusionLLM-generated Ambient notes demonstrated quality comparable to physician-authored notes across multiple specialties. While Ambient notes were more thorough and better organized, they were also less succinct and more prone to hallucination. The PDQI-9 provides a validated, practical framework for evaluating AI-generated clinical documentation. This quality assessment methodology can inform iterative quality optimization and support the standardization of ambient AI scribes in clinical practice.
