Large Language Models Can Extract Metadata for Annotation of Human Neuroimaging Publications

Matthew D Turner^1*Abhishek Mahesh Appaji²Nibras Ar Rakib³Pedram Golnari⁴Arcot K Rajasekar⁵Anitha Rathnam K V⁶Satya S Sahoo⁴Yue Wang^5,7Lei Wang¹Jessica A Turner¹

• ¹Department of Psychiatry and Behavioral Health, College of Medicine, The Ohio State University, Columbus, Ohio, United States
• ²Department of Medical Electronics Engineering, B.M.S. College of Engineering, Bengaluru, India, Bengaluru, India
• ³Faculty of Information, University of Toronto, Toronto, Ontario, Canada, Toronto, Canada
• ⁴Department of Population and Quantitative Health Sciences, School of Medicine, Case Western Reserve University, Cleveland, Ohio, United States
• ⁵School of Information and Library Science, University of North Carolina at Chapel Hill, Chapel Hill, North Carolina, United States
• ⁶Department of Computer Science and Engineering, University Visvesvaraya College of Engineering, Bangalore University, Bengaluru, India, Bengaluru, India
• ⁷Carolina Health Informatics Program, University of North Carolina, Chapel Hill, NC, USA, Chapel Hill, United States

We show that recent (mid-to-late 2024) commercial large language models (LLMs) are capable of good quality metadata extraction and annotation with very little work on the part of investigators for several exemplar real-world annotation tasks in the neuroimaging literature. We investigated the GPT-4o LLM from OpenAI which performed comparably with several groups of specially trained and supervised human annotators. The LLM achieves similar performance to humans, between 0.91 and 0.97 on zero-shot prompts without feedback to the LLM. Reviewing the disagreements between LLM and gold standard human annotations we note that actual LLM errors are comparable to human errors in most cases, and in many cases these disagreements are not errors. Based on the specific types of annotations we tested, with exceptionally reviewed gold-standard correct values, the LLM performance is usable for metadata annotation at scale. We encourage other research groups to develop and make available more specialized "micro-benchmarks," like the ones we provide here, for testing both LLMs, and more complex agent systems annotation performance in real-world metadata annotation tasks.