For the evaluation, we used previously acquired data comprising healthy controls and patients with epilepsy. Included high-resolution T1-weighted MR images were acquired at the Bern University Hospital (Inselspital) on 3T scanners from Siemens (Magnetom Trio and Verio, Siemens, Erlangen, Germany) with 1 mm isotropic resolution. MR protocols were either MDEFT (31), standard 3D MP-RAGE (32), MP-RAGE according to the recommendations of the Alzheimer's Disease Neuroimaging Initiative (33), or MP-RAGE optimized for gray-white contrast (34) with sequence parameters as listed in the Supplementary Material. More than one scan was available for some subjects, resulting in a total of 126 MRIs from 105 patients and 406 MRIs from 354 healthy controls, as listed in Table 1 and Supplementary Table S1.

The assignment of the patients' MRI to the epilepsy sub-group is based on information extracted from the radiological report of the examination (i.e., corresponds to the assessment of the neuroradiologist with the clinical and imaging information available at that time point). In particular, the initial assessment of whether HS is present was based entirely on the radiological finding. Patients without reported HS (IGE/unknown, TLE HS negative) are referred to as the “all-other-epilepsies” sub-group in the text. Where available, additional clinical and radiological information from follow-up examinations was used for a separate outlier review (cf. Section 2.5). The age of onset of the disease is known from 52 patients (with 67 MRI) with an average age of 18.4 (±14.7) years and a duration of the disease at the time of the MRI of 17.7 (±14.9) years.

The segmentation of the hippocampi from the four investigated methods (FS, FS-SF, FSL, and DL) were analyzed in various steps, individually per hemisphere and using the asymmetry between the hemispheres. Asymmetry indices (AI) (28) between the left (lh) and right hemisphere (rh) were calculated as follows:

This quantity is zero for completely symmetric hippocampi and ranges between +1 and −1 otherwise.

Hippocampal volumes corrected for brain size and age were calculated for each method by fitting a linear model (lm) to the volumes of the healthy controls with the normalized (zero-mean, unit SD) co-variates estimated total intracranial volume (eTIV (37) from FreeSurfer) and age. In agreement with the literature (38, 39), the initially included co-variate sex was not significantly related to volumes and was subsequently removed from the model. The resulting lm(vol~eTIV+age) was then applied to all subjects. We then calculated effect sizes between healthy controls and left/right-sided HS for the corrected volumes.

Besides the hippocampal volumes, further metrics were extracted from the binary segmentation using pyradiomics (40), resulting in 14 shape features. Internally, pyradiomics calculated these features on a triangulated mesh generated using marching cubes (41). To identify the most important feature for further analysis, these shape features served as input to train a support vector machine (SVM) (42) to classify HS vs. all-other-epilepsies. A linear SVM (43) with default hyperparameters was trained using 5-fold cross-validation and 20 repeats. The samples were stratified by patients to ensure all MRIs from the same subject were in the same fold. Relative feature importance was aggregated across all runs to determine a feature ranking. Subsequent experiments were performed using the volume and the best-ranked shape feature.

An estimation of the discriminative power of these two metrics was determined by means of the area under the curve (AUC). Using the absolute AI to distinguish between HS (n = 37) and all-other-epilepsies (n = 86), AUC were calculated from an ROC curve (44).

Finally, we used a quantitative report as outlined in Figure 1 to display the ratios of both hippocampi simultaneously. Exact symmetries would appear along the diagonal line. Standard deviations (SD) of the asymmetry indices (AI) in the healthy controls (n=406) were calculated to demarcate limits of two and three SD from the expected norm. Based on these limits, accuracy metrics (sensitivity, specificity, and F1-score) were determined for classifying unilateral HS vs. all-other-epilepsies.

Statistical analyses were performed using R with the stats package version 3.6.2 (45). Effect sizes were reported using Cohen's d (46). A significance level of α = 0.05 was set.

To assess the robustness of the methods, we have used the same-day repeated scans in the dataset and determined a reproducibility error. For each metric m, we calculated the mean absolute percentage error (MAPE) as follows:

where N is the number of sessions with re-scans, n(i) the number of re-scans in the session i for a subject, m_{(i, t)} the measurement at timepoint t, and μ(i)=1n(i)∑t=1n(i)m(i,t) the within-session mean. A session comprises the scans acquired on the same day for the patients and all scans within 1 year for the healthy controls, resulting in 41 sessions. Statistical significance of the differences between the four methods was determined using paired t-tests.

Additionally, we have calculated intraclass correlation coefficients (ICC) with the first two MRIs from every session. The random effects of repeated acquisitions are reflected in a two-way random-effect model with an absolute agreement, also known as ICC(2, 1) (47, 48), implemented in the R-package irr (49).

As outlined above (Section 2.1), the patients' MRIs were initially assigned to epilepsy sub-groups entirely based on information from the radiological report of the corresponding image. Therefore, we performed an additional sensitivity analysis. An experienced radiologist (co-author PR) reviewed all ‘wrongly' classified cases, i.e., putative false negatives (HS appearing inside the limits of 3 SD relative to healthy controls) and false positives (all-other-epilepsies appearing outside the limits). For the review, all available clinical information was taken into account including patient history, follow-up assessments by epileptologists, further diagnostics like EEG, additional MRI examinations, and all medical reports. Results after correcting the assigned sub-groups are reported separately.

In a supplementary subanalysis, we compared the results from DL+DiReCT to two other DL based segmentation methods: the whole-brain neuroanatomy segmentation method FastSurfer (19)¹ and HippoDeep (21)², which specifically segments the hippocampi only.

Additionally, we report surface-to-volume ratios derived from the manual tracing of the hippocampi by experts as provided in the OASIS TRT-20 dataset (50, 51) of twenty healthy individuals.

Effect sizes of hippocamal volumes (after correction for brain size and age) between healthy controls and HS were larger for DL (left hippocampus = −2.968, right= −1.904) than for FS (left= −2.462, right= −1.624), FS-SF (left= −2.376, right= −1.661), and FSL (left= −2.818, right= −1.826) with detailed distributions reported in Supplementary Figures S1,S2. The asymmetry index (AI) of the volumes (uncorrected as the contralateral side serves as internal reference for every MRI) were generally more sensitive, again with the effect sizes for DL being larger (HS left= −4.165, right = 4.203) than for FS (left= −3.085, right= 3.695), FS-SF (left= −3.697, right = 4.080), and FSL (left= −2.301, right = 2.544) as shown in Figure 2. In healthy controls, a statistically significant (p <10⁻⁵) negative mean AI was observed from all four methods (FS= −0.010, FS-SF= −0.008, FSL= −0.011, and DL = −0.007), indicating a slightly larger right hippocampal volume.

A qualitative example of a patient with left-sided HS is shown in Figure 3 with automated segmentation of the hippocampi and ventricles outlined (as shown in Supplementary Figure S3 for additional examples). Qualitatively inspecting the results, mis-segmentation by the atlas-based methods was most frequently observed on the lateral side of the body of atrophic hippocampi.

The surface-to-volume ratio was identified by the SVM as the most important shape feature from the DL-based segmentation (Supplementary Figure S5).

By plotting the left vs. the right surface-to-volume ratios in Figure 4, we can observe the healthy controls and all-other-epilepsies clustering along the diagonal while left-sided HS appear on the lower right triangle, right-sided HS on the upper left, and the bilateral cases toward the upper right corner, indicating that in contrast to the volume, the surface-to-volume ratio increases in the presence of HS.

We have observed a symmetric ratio in the healthy controls only from the DL-based segmentations (p = 0.24, two-sided t-test for asymmetry), whereas the other three methods had either a significantly (p <10⁻⁸) positive (FS, FSL) or negative (FS-SF) ratio. Separation of point clouds (Figure 4) and effect sizes between healthy controls and HS, as reported in Figure 5, were generally larger for DL than the other three methods.

When classifying MRIs using 3 SD of AI on the healthy controls as a threshold, DL reached the highest accuracy in terms of F1-score (Supplementary Table S2) both for the volume (F1 = 70.0) and for the surface-to-volume ratio (F1 = 71.2) which is consistent with the highest AUC observed (Figure 6) for the DL-based segmentation.

A comparison of the robustness evaluation from the re-scan sessions (n = 41) is listed in Table 2. The surface-to-volume ratio derived by the DL-based segmentation was statistically significantly more robust (lower MAPE) than the other three methods (Supplementary Figure S8). For the volume, only FSL was comparably robust to DL. For FS and DL, reproducibility by means of ICC was generally higher for the surface-to-volume ratio than for the volume.

When classifying MRIs like described above with the DL-based method, 21 cases (16.7% of all MRI, from 17 patients) were putative false negatives or false positives in relation to the initial radiological assessment serving as ground truth. MRIs stemming from the same patient (7 MRIs from three patients, as shown in Supplementary Table S3) appeared in close vicinity in the quantitative report (Supplementary Figure S6). After reviewing all medical records by an expert, this diagnosis was confirmed in 8 of these 17 patients. However, the other nine patients were classified differently after considering all follow-up clinical information. These outliers are listed in Supplementary Table S3 and highlighted in Supplementary Figure S6 and the resulting plot with the corrections is shown in (Figure 1). AUC for the classification with the adjusted classes was accordingly higher (see Supplementary Figure S7), while only minimally disturbing the order of the effect sizes.

The supplementary comparison of results from three different DL-based methods can be found in Supplementary Section 6. In the surface-to-volume plots (Supplementary Figure S9), the healthy individuals from the OASIS dataset cluster around the healthy controls for the DL-based methods whereas the ratios are significantly higher from the manual tracing.

As we consider manual segmentation not a viable option for a potential future clinical application, the aim of the study was designed to compare an efficient DL-based method against commonly used atlas-based methods without comparison against a manually derived ground truth. To account for this limitation, we demonstrated the influence of manual labels on the proposed surface-to-volume ratio with data from Mindboggle (51), a frequently used publicly available dataset with manually annotated neuroanatomy labels. Although manual tracings are generally looking good on the coronal view, we confirm the earlier observation of disturbing staircaise effects by Coupé et al. (67) in the axial and sagittal view of manually traced hippocampi (cf. Supplementary Figures S10,S11). This demonstrates the difficulty of manual tracing and the challenge for humans to label 3D structures in 2D views. A remedy would require correcting tracings from all three directions iteratively until complete consistency, which is difficult for the hippocampus as tracing protocols for the hippocampus are predominantly defined in the coronal direction (68). While these slice inconsistencies probably have a less pronounced effect on the calculated volumes due to averaging effects, the surface area is particularly prone to such artifacts. Consequently, manual tracing is not an option for this type of shape analysis.

While the primary analysis aimed to compare DL vs. atlas-based methods and not necessarily find the best DL-based algorithm, we replicated key figures and metrics with two other popular deep learning methods in a supplementary analysis. Overall, the DL-based methods yielded comparable results, outperforming atlas-based methods.

All segmentation methods were used with their default settings (without hyperparameter tuning on the dataset), recognizing this might have caused FSL-FIRST to underperform in this comparison due to the low number of modes. Results from all methods were used as is without manual corrections. The dataset contained T1w images with minor variations in the MR protocol which might influence the segmentations.

We have not performed an in-depth shape analysis of the segmented hippocampi but rather used simple metrics to demonstrate that an improved segmentation leads to better discrimination of abnormal hippocampi. We speculate that advanced shape analysis techniques (55, 69, 70) would benefit from the improved DL-based segmentation.

A varying amount of information was available for assigning the patients' MRIs to the epilepsy sub-groups. Therefore, we have deliberately used the initial radiological diagnosis as ground truth for the primary analysis. To account for uncertainty in the diagnosis, we performed an outlier review by an imaging expert and reported these results separately. Some cases changed diagnosis after reviewing all follow-up clinical information, confirming the challenge of diagnosis HS from MRI.

In visual assessments of suspected HS, the T2w image contains important additional information for the reader. It remains to be investigated whether supplying the corresponding T2w as an additional input to the model can help to further improve the segmentation. Quantitative methods analyzing T2 or FLAIR intensities in a region of interest (30, 71) might also benefit from an improved segmentation of the hippocampi.

The data in this evaluation were predominantly of patients with longer disease duration. We will subsequently apply the method in a multi-center prospective study of first-seizure patients (72) to assess its utility in early-onset epilepsies. Providing the proposed metrics together with the MRI to neuroradiologists could be useful in the clinical routine.