AI powered Data Curation & Publishing Virtual Assistant (AIDAVA): Usability, explainability/causability and patient interest in the first generation (G1) prototype

Rutger Van Mierlo^1,2,3,4*Wenjie Liang²Kerli Norak⁵Michaela Kargl⁶Mall Maasik⁵Anne-Lore Bynens⁷Markus Plass⁶Markus Kreuzthaler⁸Martin Benedikt⁹Laura Hochstenbach¹⁰Arnoud van 't Hof^1,3,4Remzi Celebi¹¹Andre Dekker²Isabelle DE ZEGHER¹²Petros Kalendralis²

• ¹Department of Cardiology, Cardiovascular Research Institute Maastricht (CARIM), Maastricht University, Maastricht, Netherlands
• ²Department of Radiation Oncology (Maastro), GROW Research Institute for Oncology and Reproduction, Maastricht University Medical Centre+ (MUMC+), Maastricht, Netherlands
• ³Department of Cardiology, Maastricht University Medical Centre+ (MUMC+), Maastricht, Netherlands
• ⁴Department of Cardiology, Zuyderland Medical Centre, Heerlen, Netherlands
• ⁵Department of Health Technologies, Tallinn University of Technology, Tallinn, Estonia
• ⁶Diagnostic and Research Institute of Pathology, Medical University of Graz, Graz, Austria
• ⁷Clinical Data Science, Maastricht University Medical Centre (MUMC+), Maastricht, Netherlands
• ⁸Institute for Medical Informatics, Statistics and Documentation, Medical University of Graz, Graz, Austria
• ⁹Department of Internal Medicine, Division of Cardiology, Medical University of Graz, Graz, Austria
• ¹⁰Department of Health Services Research, Care and Public Health Research Institute (CAPHRI), Faculty of Health Medicine and Life Sciences, Maastricht University, Maastricht, Netherlands
• ¹¹Department of Advanced Computing Sciences, Maastricht University, Maastricht, Netherlands
• ¹²B!loba, Tervuren, Belgium

Introduction Ensuring high quality and reusability of personal health data is costly and time-consuming. An AI powered virtual assistant for health data curation and publishing could support patients to ensure harmonization and data quality enhancement, which improves interoperability and reusability. This formative evaluation study aimed to assess the usability of the first-generation (G1) prototype developed during the AI powered data curation & publishing virtual assistant (AIDAVA) Horizon Europe project. Methods In this formative evaluation study, we planned to recruit 45 breast cancer patients and 45 cardiovascular disease patients in three European countries. An intuitive front-end, supported by AI and non-AI data curation tools, is being developed across two generations. G1 was based on existing curation tools and early prototypes of tools being developed. Patients were tasked to ingest and curate their personal health data, creating a personal health knowledge graph that represents their integrated, high quality medical records. Usability of G1 was assessed using the system usability scale. Subjective importance of explainability/causability and perceived fulfilment of these needs by G1 as well as interest in AIDAVA-like technology was explored using study specific questionnaires. Results A total of 83 patients were recruited; 70 patients completed the study, of whom 19 were unable to successfully curate their health data due to configuration issues when deploying the curation tools. Patients rated G1 as marginally acceptable on the system usability scale (59.10 ± 19.75/100), moderately positive on explainability/causability (3.3-3.8/5) and moderately positive to positive regarding their interest in AIDAVA-like technology (3.4-4.4/5). Discussion Despite its marginal acceptability, G1 shows potential for automating data curation into a personal health knowledge graph, but it has not reached full maturity yet. G1 deployed very early prototypes of tools planned for the second generation (G2) prototype, which might have contributed to lower usability and explainability/causability scores. Conversely, patient interest in AIDAVA-like technology seems quite high at this stage of development, likely due to the promising potential of data curation and data publication technology. Improvements in the library of data curation and publishing tools are planned for G2 and are necessary to fully realize the value of the AIDAVA solution.