Inter-Observer Agreement in Interpreting Intraoperative Ultrasonography during Brain Tumour Surgery

Aimun A.B. Jamjoom^1,2Olivier J.J. Sluijters³Jack Wildman³Davide Giampiccolo^4*Constantinos Charalambides³Neil U. Barua³

• ¹Department of Neurosurgery, Barking, Havering And Redbridge University Hospitals NHS Trust, Romford, United Kingdom
• ²Queen Mary University of London Blizard Institute, London, United Kingdom
• ³Department of Neurosurgery, NHS North Bristol NHS Trust, Bristol, United Kingdom
• ⁴Department of Clinical and Experimental Epilepsy, University College London Queen Square Institute of Neurology, London, United Kingdom

Background Intraoperative ultrasonography (iUS) is a powerful technology that is being increasingly utilized for brain tumour surgery. However, ultrasonography has been documented to be observer dependent in a range of healthcare settings. Here we objectively assess the degree of inter-observer variability in iUS for brain tumour surgery. Methods Nine images taken from routinely collected iUS videos from brain tumour surgery were presented to 18 neurosurgeons (5 consultants, 7 senior fellows, 6 residents). This included three tumour types (metastasis, high-grade (HGG) and low-grade glioma (LGG)) at three operative stages (before, during and near resection completion). Using 3D Slicer, observers segmented what they deemed to be tumour. Digital Image Correlation Engine Similarity Coefficients (DSC) were calculated to examine inter-observer variability. Results A total of 1377 DSCs were calculated between 18 observers across 9 images. Metastasis had the highest DSC (0.72±0.32), followed by HGG (0.64±0.33) and LGG (0.58±0.25; p<0.00001). As the resection progressed, the degree of inter-observer agreement broke down. Before resection the DSC was 0.87±0.11; during resection (0.74±0.17) and at completion (0.32±0.27; p<0.00001). The trend of decreasing agreement as the resection progressed held across tumour types. Observers reported increasing difficulty with iUS interpretation as the resection proceeded and there was statistically significant (p=0.014) negative correlation (-0.775) between DSC and difficulty rating of the segmentation. Conclusion Here we demonstrate significant inter-observer variability in iUS for brain tumour surgery. The degree of variability is tumour-type and operative stage dependent. This work adds weight to the value of building machine learning augmented iUS for brain tumour surgery.