Label tree semantic losses for rich multi-class medical image segmentation

Abstract

Rich and accurate medical image segmentation is poised to underpin the next generation of AI-defined clinical practice by delineating critical anatomy for pre-operative planning, guiding real-time intra-operative navigation, and supporting precise post-operative assessment. However, commonly used learning methods for medical and surgical imaging segmentation tasks penalize all errors equivalently and thus fail to exploit any inter-class semantics in the label space. This becomes particularly problematic as the cardinality and richness of labels increases to include subtly different classes. In this work, we propose two tree-based semantic loss functions which take advantage of a hierarchical organization of the labels. We further incorporate our losses in a recently proposed approach for training with sparse, background-free annotations to extend the applicability of our proposed losses. Extensive experiments are reported on two medical and surgical imaging segmentation tasks, namely head MRI for whole brain parcellation with full supervision and neurosurgical hyperspectral imaging for scene understanding with sparse annotations. Results demonstrate consistent improvements over the evaluated task-specific baselines, with the strongest support for the Wasserstein-based compound loss in whole-brain parcellation and for hierarchy-weighted top-level supervision in the sparse hyperspectral imaging (HSI) setting.

1 Introduction

Segmentation plays a crucial role in medical and surgical image analysis by locating and precisely outlining regions of interest such as organs, lesions, and tissues across a variety of imaging modalities. Two particularly important brain imaging applications that rely on rich and accurate segmentation are head MRI for whole brain parcellation (WBP) and interventional hyperspectral imaging (iHSI) for scene understanding and tissue characterization. WBP divides an MRI volume into spatially coherent, anatomically or functionally meaningful brain regions (Eickhoff et al., 2018). Hyperspectral imaging (HSI) captures wide-field views across dozens to hundreds of optical spectral bands, and can be used intra-operatively to reveal biochemical contrasts invisible to the naked eye (Shapey et al., 2019).

A major challenge with such rich segmentation tasks relates to the granularity at which the data is annotated and that at which segmentation models should operate. Recent works have examined the impact of training models with various levels of labeling granularity such as pixels, patches, and entire images (Seidlitz et al., 2022; Garcia Peraza Herrera et al., 2023). Other studies have provided comparative analyses of different pixel-level algorithms for brain tissue differentiation (Martín-Pérez et al., 2024). Underpinning these questions is the drive for a holistic and refined understanding of the images and surgical scenes. Annotation efforts are ongoing to provide training data across large number of potentially subtly varying classes. Combined with sparse annotations processes that may be employed to label data at scale, many of these classes may only have small amounts of training samples. It is thus important to take advantage of the semantics of the labels and realize that some types of errors are more acceptable than others. However, there is only limited previous work in medical imaging that has leveraged the structure of the label space as a source of information. In surgical imaging, to our knowledge, no such efforts have been published. By contrast, the importance of label semantics is getting recognized in the general field of computer vision (Frogner et al., 2015; Le et al., 2019; Bertinetto et al., 2020). Adopting these concepts to WBP and iHSI is expected to produce more robust models for both dense and sparse annotation setting.

Additional challenges arises when labeling needs to be performed at scale. Leveraging sparsely annotated datasets becomes an effective strategy that has been widely adopted already in iHSI segmentation and surgical imaging (Studier-Fischer et al., 2023; Hyttinen et al., 2020; Carstens et al., 2023). For instance, a recent general surgery iHSI dataset adopted a sparse annotation protocol by labeling only representative image regions, omitting marginal areas, superficial blood vessels, adipose tissue, and other artifacts (Studier-Fischer et al., 2023). Within the neurosurgical HSI dataset used in this study, representative examples of this labeling strategy are shown in the qualitative results figures presented later in the article. In sparsely annotated medical image segmentation, the absence of a label cannot be taken as evidence that a region is negative. A truly positive pixel may go unmarked for two reasons: (i) the annotator finds the region ambiguous or (ii) it is skipped due to time constraints. The most straightforward, albeit wrong, approach would be to presume that every unlabeled pixel belongs to the negative background class. To appropriately address such partial supervision, Wang et al. (2026) recently proposed a framework that learns from background-free, positive-only sparse label masks. Pixel-wise out-of-distribution (OOD) detection methodology is used at inference time to flag as background any tissue or object type that has not been annotated in the training data.

In this work, we exploit semantic relationships in the label structure through a general tree-based semantic supervision framework, instantiated by two complementary loss functions for supervised segmentation. Figure 1 provides an overview of the proposed framework for medical image segmentation. A hierarchical label tree encodes the semantic relationships among classes and serves as the basis for two complementary loss functions. For WBP, the hierarchy is derived from established anatomical guidelines based on the Desikan–Killiany–Tourville (DKT) protocol (Klein and Tourville, 2012) (Figure 2, with a larger version available in Supplementary material and online via an interactive hierarchy visualization); for surgical HSI, the hierarchy is defined by expert consensus, with the full structure provided online via an interactive hierarchy visualization and in Figure A1 in Appendix. We encode this structure directly into optimization via a Wasserstein distance-based segmentation loss, which penalizes errors according to path length in the label tree, and a tree-weighted semantic cross-entropy loss, which extends weighted cross-entropy to all nodes in the hierarchy. Both losses incorporate semantic information into training, but they use the hierarchy differently: the Wasserstein formulation changes the cost of leaf-label mistakes through pairwise tree distances, whereas tree-weighted cross-entropy supervises aggregated probabilities at internal as well as leaf nodes. We further integrate these losses into the positive-only sparse supervision framework of Wang et al. (2026) to support background detection as OOD without degrading performance on positive (ID) classes. Across rich segmentation tasks in brain MRI and neurosurgical iHSI, the proposed losses improve over the evaluated baselines in several clinically relevant settings. In particular, experiments on three MRI WBP datasets show that the Wasserstein-based loss provides the strongest support among the evaluated losses, especially for NSD and small-structure metrics, while experiments spanning two tasks and four datasets demonstrate that the proposed framework can be incorporated into standard segmentation models without architectural modification in both sparse and dense annotation settings. The code is available at https://github.com/cai4cai/nnunet-tree-semantic-extension.

Figure 1

Wang et al. (2025a). Creative Commons Attribution License (CC BY).

Figure 2

The main contributions of this paper can be summarized as follows:

• We introduce two tree-based semantic losses for medical image segmentation, namely a Wasserstein distance-based segmentation loss and a tree-weighted semantic cross-entropy loss, which explicitly encode hierarchical relationships between labels during training.

• We show how these losses can be applied to both densely supervised whole-brain parcellation and sparsely supervised neurosurgical HSI segmentation, including integration with a positive-only OOD detection framework for background identification.

• We demonstrate that incorporating expert-defined or anatomically derived label hierarchies leads to more semantically meaningful supervision and improved segmentation performance relative to standard baselines.

• We provide empirical evidence across two segmentation tasks and four datasets, with the Wasserstein-based loss consistently improving over the CE + Dice nnU-Net baseline for WBP, evaluated using manual references on MB42 and automated GIF-derived pseudo-ground-truth masks on AOMIC/IXI, particularly on small structures with limited annotations.

2 Related work

2.1 Semantic segmentation in WBP

Although the desired granularity varies across protocols, whole-brain parcellation (WBP) typically entails segmenting hundreds of distinct classes (Klein and Tourville, 2012). Classical tools such as FreeSurfer (Fischl et al., 2002) and GIF (Geodesic Information Flows) (Cardoso et al., 2015) automatically deliver robust and accurate results, but processing a single scan can take from several hours to an entire day. In particular, GIF is a well-established multi-atlas segmentation and label-fusion method for brain parcellation that has been used successfully in recent clinical neuroimaging studies (Bocchetta et al., 2023; Young et al., 2021). Deep-learning approaches have been proposed to shorten inference time (Henschel et al., 2020; Guha Roy et al., 2019; Roy et al., 2017) at the cost of increased GPU memory requirements. Recently, Kujawa et al. (2024) introduced a label merge-and-split framework that clusters spatially disjoint regions with a greedy graph-coloring algorithm, allowing the network to predict a much smaller set of merged labels and then restores the original labels at inference via atlas-derived influence regions. However, with the exception of Graham et al. (2020) discussed in Section 2.3, previous learning-based WBP approaches have not exploited the semantic relationships within the label classes.

2.2 Semantic segmentation in hyperspectral imaging

Semantic segmentation works have initially relied on classical machine learning pipelines (Ravì et al., 2017; Fabelo et al., 2018; Moccia et al., 2018) and more recently adopted deep learning (Khan et al., 2021). Many studies (Seidlitz et al., 2022; Trajanovski et al., 2021) adopt a U-Net type architecture (Ronneberger et al., 2015; Jegou et al., 2017). Previous work also examined the impact of training models with various levels of input granularity such as spectral pixels (1D CNNs), image patches, and entire HSI images (2D CNNs) (Seidlitz et al., 2022; Garcia Peraza Herrera et al., 2023; Martín-Pérez et al., 2024) concluding that providing both spectral and spatial context is beneficial. Seidlitz et al. (2022) segmented 20 organ types from 506 hypercubes taken from 20 pigs, demonstrating best performance achieved when models were trained on full images rather than on pixels or patches (Seidlitz et al., 2022). Similar results were obtained in Garcia Peraza Herrera et al. (2023) for HSI-based segmentation of dental tissues from the ODSI-DB dataset (Hyttinen et al., 2020), which includes data from 30 human subjects annotated with 35 tissue types. These works have employed sparse annotations but have not exploited dedicated learning approaches for this, leading to suboptimal results, as discussed in Section 2.4. Furthermore, only small label sets were used with no insight into the semantics relationships between labels.

2.3 Hierarchical loss functions

Hierarchical loss functions encode class-taxonomy information directly in optimization, so prediction errors are penalized by semantic severity rather than uniformly. Early work in large-scale visual recognition established this principle: hierarchy-aware objectives based on WordNet and lowest-common-ancestor structure (Deng et al., 2010), class-similarity-aware multiclass learning with shared regularization (Zhao et al., 2011), and taxonomy-conditioned metric learning (Verma et al., 2012).

Outside medical imaging, deep-learning formulations made hierarchy-aware supervision more practical and expressive. “Better-mistake” cross-entropy variants in Bertinetto et al. (2020) redistributed penalties toward semantically nearby classes, while optimal-transport formulations (Frogner et al., 2015) and later tree-based Wasserstein variants (Le et al., 2019; Takezawa et al., 2021) provided principled distance-based objectives. For dense prediction, Li et al. (2024a) further showed that hierarchy-constrained learning can be applied to semantic segmentation through pixel-wise multi-label formulations.

Within medical imaging, earlier hierarchy-aware work was comparatively limited. A brain-parcellation approach in Graham et al. (2020) predicted along the hierarchy to model uncertainty, but with increased output complexity and without clear segmentation-accuracy gains over leaf-only training. Wasserstein-based Dice objectives (Fidon et al., 2018, 2022) introduced semantically meaningful penalties for multi-class segmentation, though evaluations were mostly in lower-cardinality BraTS-style settings (Menze et al., 2015). In Deng et al. (2024), hierarchical supervision is achieved by encoding anatomical taxonomy and scale relations in a unified matrix and optimizing a taxonomy-consistent loss. By contrast, Lu et al. (2024) introduces multi-level cross-modal interaction with target-aware modality weighting to hierarchically refine PET/CT tumor features.

Taken together, these studies support hierarchy-aware supervision, but gaps remain for our target setting: rich medical label spaces, compatibility with standard segmentation architectures, and training under sparse positive-only annotations where background labels are absent.

2.4 Medical image segmentation with sparse annotation

Pixel- or voxel-level annotation of medical images is time-consuming and costly. Early work showed that accurate segmentation can be achieved from only a handful of labeled regions when unlabeled pixels or voxels are ignored in the loss (Çiçek et al., 2016). The finding inspired much of the subsequent weakly supervised learning (WSL) literature.

Existing WSL methods utilize various modes of sparse annotations, including image-level annotation (Kuang et al., 2024), bounding box (Wang and Xia, 2021; Wang et al., 2018; Wei et al., 2023), scribbles (Wang et al., 2019, 2025b), points (Glocker et al., 2013; Dorent et al., 2021; En and Guo, 2024), and 2D slices within a 3D structure (Cai et al., 2023). These methods all focus on using the sparse annotations at training time to produce full segmentation mask at test time. As an example, in Glocker et al. (2013), the authors introduced a semi-automatic labeling strategy that transforms sparse point-wise annotations into dense probabilistic labels for vertebrae localization and identification. In Xu et al. (2014), the authors propose to segment both healthy and cancerous tissue from colorectal histopathological biopsies using bounding boxes. In Wang et al. (2018), the authors reported improved CNN performance on sparse annotated input through image-specific fine-tuning. In Wang et al. (2019), the authors combined sparsely annotated input with a CNN through geodesic distance transforms, followed by a resolution-preserving network resulting in better dense prediction. However, all of these methods largely ignore the harder setting, which we face here, in which the model does not have guidance on what should be treated as non-object context. To address this limitation, Wang et al. (2026), which we detail in Section 2.5, proposed a segmentation framework that allows for positive-only learning by exploiting out-of-distribution (OOD) detection mechanisms.

2.5 Out-of-distribution detection and positive-only learning

Several studies have explored OOD detection within the context of image classification (Hendrycks and Gimpel, 2017; Liang et al., 2018; Lee et al., 2018; Hsu et al., 2020). As an early example exploiting deep learning, Hendrycks and Gimpel (2017) proposed using the maximum softmax score as a baseline for OOD detection based on an observation that correctly classified images tend to have higher softmax probabilities than erroneously classified examples. Liang et al. (2018) found that applying confidence calibration through temperature scaling (Guo et al., 2017) effectively separates ID and OOD images. Lee et al. (2018) suggested measuring the Mahalanobis distance between test image features and the training distribution from the penultimate convolutional layer of the model. Hsu et al. (2020) proposed decomposing the confidence score to adapt the temperature during training.

Despite methodological advances and positive demonstration for image classification purposes, application of OOD detection in medical image segmentation is uncommon. Some studies hypothesize that this may be due to the lack of OOD-based evaluation protocols and the difficulty in gathering relevant data for it (Lambert et al., 2024; Bulusu et al., 2020). Recent research has attempted to address this issue by using other datasets as OOD examples. Karimi and Gholipour (2023) used two separate datasets: one for training the neural network and evaluating its performance on ID data, and another for testing specifically for OOD detection. González et al. (2022) collected four types of OOD datasets to account for different distribution shifts from ID data for COVID-19 lung lesion segmentation task. However, acquiring an additional dataset that can be considered OOD is a difficult and time-consuming process. Therefore, a more scalable approach would be to establish both training and evaluation within a single dataset.

Recently, Wang et al. (2026) introduced a framework that addresses sparse multi-class positive-only segmentation learning by employing pixel-level out-of-distribution (OOD) detection to detect regions that do not correspond to any of the annotated classes in the training set. The model output probabilities are treated as a reliable signal for pixel-level out-of-distribution (OOD) detection. The network is trained solely on positive (in-distribution) classes, and a pixel is flagged as background (OOD) when the maximum predicted probability across these classes falls below a threshold. To remedy the absence of dedicated sparse positive-only segmentation benchmarks, the authors further devised a two-level cross-validation scheme. By iterating not only over subjects but also over subsets of the label space, this enables a rigorous and comprehensive evaluation using existing annotations only.

3 Methodology

This section provides detailed methodology for our proposed loss functions (Sections 3.2 and 3.2). Furthermore, we demonstrate how these can be integrated in the approach of Wang et al. (2026) for segmenting background pixels from positive-only annotations (Section 3.3).

3.1 Wasserstein distance in label space

Let L be the label space with C leaf nodes, where L = {1, 2, …, C}. Let p, q∈P(L) be probability vectors on L. The Wasserstein distance between p and q is the minimal cost to transform p into q given the ground distances between any two labels l and l′. The ground distance is represented as a matrix M and the associated Wasserstein distance W^M(p, q) is defined through an optimal transport problem:

By leveraging the distance matrix M on L, the Wasserstein distance yields a semantically-meaningful way of comparing two label probability vectors. Given a tree structure with weights associated to the edges, a semantic ground distance can be induced by the path lengths between the leaf nodes. If q = g is a crisp ground truth, a closed-form expression of Equation 1 is given in Fidon et al. (2018):

While Equation 2 can be used directly as the loss for training a segmentation model, prior work has shown benefits in combining generic and task-specific losses (Ma et al., 2021). Segmentation frameworks such as the nnU-Net (Isensee et al., 2021) have also been optimized for working with combined losses such a weighted sum of Dice and CE. We generalize the formulation in our preliminary work (Wang et al., 2025a) and propose combining Equation 2 with a generic segmentation loss to obtain a compound Wasserstein distance based segmentation loss:

3.2 Tree-weighted semantic cross-entropy loss

We also propose another approach to building semantic loss functions by computing the aggregated probabilities across all the nodes in the tree hierarchy, not just the leaf nodes. A segmentation loss such as CE can then be evaluated across all node probabilities.

Let the label tree be composed of K levels, with level 0 corresponding to the deepest level, i.e., the leaf nodes. Let A be the adjacency matrix associated with , which encodes the parent–child links in the tree: A_{u, v} = 1 if node u is the parent of node v, and A_{u, v} = 0 otherwise. Let be a zero-padding of p to initially associate non-leaf nodes with a zero mass, and p^† be the vector collecting all the probabilities:

For a finite tree, A is nilpotent: if the longest leaf-to-root path has K edges, then A^k = 0 for k > K. Hence (I−A)⁻¹ = I+A+⋯+A^K is a finite sum. For a three-level tree ordered as (r, a, b, c, d, e, f), with p = (0.1, 0.1, 0.3, 0.5)^T on the leaves, and p^† = (1, 0.2, 0.8, 0.1, 0.1, 0.3, 0.5)^T. Figure 3 shows the tree and the corresponding adjacency matrix.

Figure 3

Here, we focus on an extended CE weighted according to domain specific insight:

where w_v is the weight of the edge associated with v as a child node. We note that if w_v = 1 for all leaf nodes and w_v = 0 otherwise, Equation 5 reduces to the standard CE.

Similar to the Wasserstein case in Section 3.1, Equation 5 is combined with a generic segmentation loss . We refer to our semantically-informed variant of the segmentation loss as the tree-weighted semantic segmentation loss:

The coefficients α and β in Equations 3 and 6 are loss-specific hyperparameters; we reuse the symbols only to indicate their shared role in balancing the semantic and segmentation terms.

3.3 Learning from sparse positive-only annotations

We build on the recent work (Wang et al., 2026) to learn segmentation from sparse multi-class positive-only annotations. We extend their approach based on pixel-wise OOD detection methodology to benefit from our tree-based semantic losses. Given an image x, each spatial location i is associated with a class label y_i. An annotated pixel i in the sparse positive-only training set is such that y_i∈{c} = {1, 2, …, C} and C is the number of positive classes. c = 0 is retained to denote background pixels. By construction, no background annotation is available at training time but OOD detection can be used to differentiate positive classes from the background at inference time.

The framework starts by training a segmentation model using only the positive classes. For clarity, the background class is not a possible output of the network but OOD-tailored training may be used to improve the performance of the next step. The framework then employs a confidence score from an OOD detection mechanism with a threshold τ to flag background pixels at inference time. The model prediction becomes:

where is a scoring function that captures the probability of pixel i belonging to the positive class c, while acknowledging the possibility of it being background/OOD. can be selected from various OOD detection methods used in image classification tasks (Hendrycks and Gimpel, 2017; Liang et al., 2018; Lee et al., 2018; Hsu et al., 2020).

To use the hierarchy for OOD detection, we aggregate probabilities using Equation 4 and threshold scores at a chosen hierarchy level h instead of at the leaf level. Let be the nodes at level h, with h = 0 for leaves. We use the coarsest non-root level, h = K−1. For pixel i and node , the score in Equation 7 is ; the maximum and argmax are then taken over .

4 Experimental setup

This section details the configuration used to evaluate our tree-based loss functions. We consider two segmentation tasks: 1) 3D MRI based WBP with full supervision; and 2) 2D hyperspectral surgical scene segmentation with sparse positive-only annotations. All methods share identical hyper-parameters unless otherwise noted, ensuring that performance differences arise solely from the loss functions themselves.

4.1 Dataset

4.1.1 WBP datasets with dense reference masks

We evaluated our approach on 3D T1-weighted MR images drawn from three openly available datasets. The Mindboggle101 collection provides 101 manually annotated scans aggregated from multiple public sources (Klein and Tourville, 2012). In Mindboggle101, two constituent subsets (NKI-RS-22 and NKI-TRT-20) were combined to create a 42-image test cohort, hereafter termed Mindboggle42 or MB42 for short. The remaining 59 scans, referred to as Mindboggle59 or MB59, served as multi-atlas data for the classical GIF algorithm (Cardoso et al., 2015). Because manual WBP annotation is often difficult due to single manual annotations may contain local inconsistencies for fine anatomical structures. Automated GIF-derived masks can therefore be qualitatively more anatomically coherent in some regions.

GIF with Mindboggle59 was used to generate GIF-derived pseudo-ground-truth WBP masks for two other datasets which otherwise do not provide manual WBP annotations. Accordingly, AOMIC and IXI metrics quantify agreement with automated GIF-derived pseudo-ground-truth masks rather than direct accuracy against manual annotations, and these experiments cannot quantify GIF label error without additional manual AOMIC/IXI WBP labels. The AOMIC PIOP2 dataset (Snoek et al., 2021) contributes 226 MRI scans (180 training, 46 testing) acquired from healthy participants aged 18–25 years. Finally, the IXI dataset¹ comprises 581 scans (464 training, 117 testing) collected from healthy individuals aged 20–86 years. By construction, all three datasets share the label space arising from the DKT protocol (Klein and Tourville, 2012). The DKT protocol does not directly provide a label hierarchy but its label space is very similar to that used in the original GIF implementation for which a label hierarchy is provided in Graham et al. (2020). Manual matching was performed to adapt the hierarchy in Graham et al. (2020) to the DKT label set. The resulting hierarchy is shown in Figure 2 and an interactive hierarchy visualization.

4.1.2 Surgical HSI dataset with sparse positive-only annotations

The data was obtained from patients undergoing microscopic cranial neurosurgery as part of an ethically approved single-center, prospective clinical observational investigation employing a prototype hyperspectral imaging system (NeuroHSI study: REC reference 22/LO/0046, ClinicalTrials.gov ID NCT05294185). Informed consent was obtained from all participants. The primary objective was to evaluate the intra-operative utility of a 4 × 4, 16-band visible-range snapshot mosaic camera (IMEC CMV2K-SSM4X4-VIS) mounted on a surgical microscope.

The dataset comprises 22,829 annotated frames derived from 45 distinct patients, encompassing both neuro-oncological and neurovascular pathologies. Multiple videos were acquired throughout each case, with each recording representing a specific surgical phase intended to capture relevant intra-operative details. The data includes varying visual perspectives due to changes in surgical microscope positioning. Training snapshot data are first processed with a demosaicking pipeline (Li et al., 2022, 2024b), resulting in 1080p frames with 16 channels (hypercubes). As in Li et al. (2024b), these hypercubes can be converted to synthetic standard RGB (sRGB) for visualization purposes.

The sparse, background-free annotations encompass 107 subclasses organized into a hierarchical structure defined by neurosurgeons. Due to space constraints, the full label hierarchy is provided in Supplementary material and online via an interactive hierarchy visualization, with corresponding color references displayed at each node in the label hierarchy. For each video a representative subset of frames was selected by an experienced neurosurgeon to minimize motion blur, maximize the number of tissue classes included and ensure key surgical phases were represented. Selected frames were manually annotated by two neurosurgeons who had also been present during the surgical procedure. In cases where tissue class was ambiguous based on HSI-derived sRGB images alone, corresponding high resolution snapshot pictures taken using the integrated surgical microscope camera were correlated with hyperspectral data to determine tissue class, and if necessary discussed with the operating surgeon. Where definitive identification of tissue class was not possible, the area was left unlabeled. Where feasible, these manual annotations were then propagated across subsequent frames algorithmically using the registration-based propagation feature of the ImFusion Labels software. Each propagated annotation was verified and corrected (when needed) by a neurosurgeon prior to final submission.

We also note that the coupling of the camera onto the surgical microscope induces a partial masking of the sensor on the outside of the circular field of view of the microscope. In addition to human-labeled categories, an additional label was generated by a content area estimation algorithm (Budd et al., 2023). This algorithm provides a robust estimation of the location of the non-informative areas on the sensor. By treating these areas as part of our label hierarchy, the model can more effectively discriminate content regions. Figure 4 presents example sRGB image plus annotation overlays.

Figure 4

Wang et al. (2025a). Creative Commons Attribution License (CC BY).

4.2 Implementation details

There are two distinct experimental configurations, depending on the dataset and task. Across all experiments, when training with our tree semantic losses, we used the same hyperparameters as those used in the baseline approach to ensure a fair comparison. Similar to Wang et al. (2025a), for all compound losses, we set α = β = 0.5. Note that this shared numerical setting does not imply that the two losses share same coefficients. All experiments ran on an NVidia DGX cluster with V100 (32GB) and A100 (40GB) GPUs.

4.2.1 WBP

For the WBP task, we use the nnU-Net framework (Isensee et al., 2021) and only modify the loss function. This framework employs a built-in empirical rule to automatically decide all hyperparameters based on statistics extracted from the training set. As per the nnU-Net default, the generic segmentation loss is set to .

4.2.2 HSI

For the neurosurgical HSI dataset, we adopted a similar training pipeline as described in Wang et al. (2026). Specifically, we used a U-Net architecture with an EfficientNet-b5 encoder (Tan and Le, 2019), pre-trained on ImageNet (Deng et al., 2009) for all experiments². Given the sparsely annotated nature of the training data, the Dice loss is not applicable. We thus opt for for Wasserstein based loss and for tree-weighted CE loss to prevent computing CE on leaf node classes twice. We employed the Adam optimizer (Kingma and Ba, 2017) with β₁ = 0.9 and β₂ = 0.999, together with an exponential learning rate scheme (γ = 0.999). We set the initial learning rate to 0.001, used a mini-batch size of 5, and trained for a total of 50 epochs. For data augmentation, we adopted a similar setup to that reported in Wang et al. (2026): random rotation (rotation angle limit: 45°), random flipping, random scaling (scaling factor limit: 0.1), and random shifting (shift factor limit: 0.0625). All transformations were applied with a probability of 0.5. In addition, we apply ℓ¹-normalization at each spatial location to account for the non-uniform illumination of the tissue surface. This is routinely applied in HSI because of the dependency of the signal on the distance between the camera and the tissue (Bahl et al., 2023; Studier-Fischer et al., 2023).

4.2.3 Choice of tree weights/ground distances

Edge weights can be defined across hierarchical levels, producing distinct cost matrices M that affect overall performance. We consider four edge weight setups. A first simple case is M_t that assigns a weight of 1 only to edges at the top-level and 0 elsewhere. A second simple case is M_ℓ that assigns a weight of 1 only to leaf nodes and 0 elsewhere. Neither M_t nor M_ℓ effectively take advantage of the label hierarchy and these cases therefore represent baselines. By contrast, M_e and M_h place non-zero weights on every edge. M_e sets all edge weights to 1, whereas M_h imposes a scaling parameter κ such that a parent-level edge weight is κ times larger than its children.

To focus the number of experiments reported here, for the WBP task, we reuse the best configuration identified in Wang et al. (2025a). We thus only report results with the M_h configuration combined with κ = 10.

For the HSI experiments, we report all four edge weight configuration. Through preliminary experiment not reported here, we found that the following choice of κ yields appropriate performance for the M_h configuration: κ = 10 for based experiments and κ = 2 for based experiments.

5 Results

This section presents quantitative (Sections 5.1 and 5.2) and qualitative (Section 5.3) results. In addition, we conducted error analysis by plotting confusion matrix to investigate the effectiveness which model can better infer relationships among different tissue types (Section 5.4).

5.1 WBP with dense reference masks

Table 1 summarizes the cross-dataset evaluation on the three WBP sub-datasets (MB42, AOMIC, and IXI). The MB42 test set supports evaluation against manual WBP annotations, while the AOMIC and IXI test sets support fair relative comparison against the same automated GIF-derived pseudo-ground-truth masks for all losses. Our baseline is the standard nnU-Net, trained with its default loss and hyper-parameters. We also include the Generalized Wasserstein Dice loss (Fidon et al., 2018) as an additional baseline. Replacing these baseline losses with one of our proposed compound losses () yields consistent gains in manual-reference performance on MB42 and in pseudo-ground-truth agreement on AOMIC/IXI. We report the mean Dice score and the mean Normalized Surface Dice (NSD) metric (Seidlitz et al., 2022), where the latter measures the overlap of two volume surface. The surface element is counted as overlapping when the closest distance to other surface is less to 3mm tolerance. The benefits are particularly pronounced for classes with fewer annotations. We performed pixel-level counting across all classes and selected the 10 classes with the fewest annotated pixels. We report the mean Dice and NSD scores of these classes as Dicesmall and NSDsmall, respectively.

For WBP comparisons, subject-level Dice and NSD values were generated from retained predictions and labels for identical test cases. Statistical comparisons used two-sided paired-sample tests on matched subjects for each predefined train–test dataset pair, metric, and proposed loss against . Table A1 in Appendix reports the number of paired subjects, paired differences, and raw and Holm-Bonferroni-adjusted significance markers across the 72 predefined tests. Statistical support is interpreted using the adjusted p-values. Under this criterion, shows the strongest support, with adjusted significance for NSD and small-structure metrics in nearly all train–test settings, while Dice gains are not uniformly significant for every dataset pair.

We further quantify the GIF performance for the dataset with manual labels. We applied GIF to the MB42 images using the MB59 manual labels as the multi-atlas set, and compared the resulting pseudo-labels with the manual MB42 annotations. This yielded a mean label-wise Dice of 78.6 ± 15.4. However, direct quantification of GIF error on AOMIC and IXI is not possible because manual WBP labels are not available for these datasets.

5.2 Surgical HSI with sparse positive-only annotations

Table 2 presents the cross-validation results on top-level classes by selecting the output probability at the top-level only (that is, p^{†, K−1}). We report the results with different confidence thresholds τ. Where τ₀ = 0 represents no background (OOD) detection and τ_m represents the threshold which maximizes scores across the positive annotations (ID data). To better capture performance on the background, τ_m could be also computed by incorporating held-out classes for background / OOD performance monitoring during validation in the two-level cross-validation (Wang et al., 2026). However, this is not presented here due to time constraints.

For all loss functions we evaluate the True Positive Rate (TPR), Balanced Accuracy (BACC), and F1 scores. Results are reported by averaging across classes under one-vs-rest strategy, where the positives are pixels of such class, and the negatives are the pixels of all the other classes. We do not report IoU for the HSI task because the training and evaluation setting is based on sparse positive-only annotations, for which unlabeled pixels cannot be interpreted as reliable negatives and IoU therefore becomes difficult to interpret as a primary metric. We also do not report FPR separately, as it is already reflected in BACC through the class-wise true negative rate / false positive behavior. Paired-sample t-tests on test-image F1 scores showed that the final tree-based configurations using M_h significantly improved over their corresponding M_ℓ baselines for both and (p < 0.0001). For model performance on leaf node classes, we report F1 scores at τ_m for both losses. For , the mean of F1 scores at τ_m based on M_ℓ and M_h are 0.069 and 0.073, respectively. For , the results are 0.068 and 0.037, respectively.

While both loss outperforms the baseline, our results shows that M_t performs similarly to M_e, suggesting that top-level edge weights have the greatest impact on performance when evaluating accumulated probabilities on corresponding nodes. For , employing M_h yields the best performance, surpassing the strong baseline that adapts CE loss to train only on the top-level node. These results indicate that an appropriate choice of M can improve performance over the evaluated baselines on both top-level and leaf nodes. For background / OOD detection, our findings exhibit trends similar to those reported in previous work on three medical image datasets (Wang et al., 2026). By removing pixels considered outliers, all methods gain further improvements.

Furthermore, it is possible to train the model using the pure Wasserstein distance defined in Equation 2. We refer to this loss as the Wasserstein loss, denoted by . The results for based on M_t are reported in Table 2, where we observe that it underperforms the baseline. Based on this observation, further investigation of the distance matrix M was not conducted for .

5.3 Qualitative results

5.3.1 WBP

Figure 5 qualitatively compares the baseline model trained with (CE + Dice) against our Wasserstein compound loss on the AOMIC dataset. Each column displays the predicted mask at an increasingly fine level of the label hierarchy using the color scheme and dendrogram from Figure 2. The white arrow marks the challenging class “non-WM-hypointensities,” which the baseline fails to detect but our method segments correctly across all hierarchical levels.

Figure 5

5.3.2 HSI

Figure 4 presents qualitative results comparing different loss functions on some challenging cases. Although all models have the capacity to differentiate leaf node classes, for simplicity, we visualize predictions at the top-level nodes. We present results at τ₀ = 0 to illustrate the outcome without background/OOD segmentation for the CE baseline (). For the Wasserstein-based loss, we include results using ground distance matrices M_t and M_h for comparison. Comparing against M_ℓ baseline, and show qualitative results that are more semantically plausible in terms of differentiating normal and abnormal tissues. For other classes such as vascular ones, they also show improved segmentation performance by reducing false positive prediction.

5.4 Analysis of error types (confusion matrices)

Evaluation relying solely on overall segmentation performance is insufficient to capture how effectively a model infers relationships among different tissue types. It neglects the possibility that the model may choose incorrect but semantically meaningful labels. To explore these aspects, we plot a multi-class confusion matrix for the top-level nodes p^{†, K−1}, as shown in Figure 6. Because class distributions vary across folds, we average the results of each cross-validation fold to ensure a fair comparison.

Figure 6

For CE baseline on HSI task, the model struggles to distinguish normal from abnormal tissues, which constitute a semantically important distinction. By contrast, the tree-semantic losses () exhibit a more meaningful confusion pattern between normal and abnormal tissues. Similarly for WBP task, baseline model struggle to differentiate between hypointensity and normal regions. These findings suggest that the proposed method successfully exploits hierarchical relationships within the label space.

5.5 Hyperparameter tuning result

Our method relies heavily on the distance matrix M, which encodes prior knowledge of inter-class relationships. In practice, exhaustively exploring all possible configurations of M can be computationally expensive. To address this, we adopt the previous M_h configuration, where the edge weight of a parent node is set to be κ times larger than that of its child node. Under this M_h configuration, optimizing M reduces to finding the optimal scaling parameter κ.

To assess the influence of the hyperparameter on model performance, we run multiple trials by tuning the scaling parameter κ. For each trial, a complete four-fold cross-validation was performed. As shown in Figure 7, the model performance varies with increasing values of κ on the brain HSI dataset. We observed that model performance improved over the baseline by increasing κ. For , the model achieved its peak performance at κ = 50 with best result 0.66 mean F1 score. Beyond which both the average performance and its consistency across folds decreases.

Figure 7

6 Conclusion and discussion

We propose two semantically driven loss functions applicable for both sparse and dense supervised segmentation tasks, relying on a tree-structured label space defined by domain experts. Both the Wasserstein distance based segmentation loss and the tree-weighted semantic segmentation loss leverage prior knowledge of inter-class relationships. The former captures these relationships through a distance matrix in label space, while the latter extends the standard CE loss to incorporate weighted probabilities aggregated at each node in the tree. Additionally, we integrate these loss functions into a sparse positive-only learning framework for segmentation, which enables pixel-level background segmentation through an OOD detection approach.

Our experiments were designed to evaluate the proposed losses under strong established segmentation backbones rather than to benchmark alternative architectures. In particular, the WBP experiments use nnU-Net as a widely adopted reference framework for 3D medical image segmentation (Isensee et al., 2021, 2024), while the HSI experiments use the established U-Net-based sparse positive-only segmentation pipeline from prior work (Wang et al., 2026, 2025a).

Regarding the optimal weighting of hierarchical levels, our experiments on four distance matrices reveal that top-level weights exert the greatest influence on performance when the evaluation is conducted at the corresponding level. Furthermore, we found that the hierarchical weighting scheme further enhances performance, improving over the evaluated baselines for both top-level and leaf node labels. Moreover, error analysis and qualitative evaluations demonstrate that these approaches offer improved tissue differentiation compared with standard baselines.

References

• 1

  BahlA.HorganC. C.JanatkaM.MacCormacO. J.NoonanP.XieY.et al. (2023). Synthetic white balancing for intra-operative hyperspectral imaging. J. Med. Imaging10:046001. doi: 10.1117/1.JMI.10.4.046001

  • Pubmed Abstract
  • CrossRef
  • Google Scholar
  • View reference in article

• 2

  BertinettoL.MuellerR.TertikasK.SamangooeiS.LordN. A. (2020). “Making better mistakes: leveraging class hierarchies with deep networks,” in 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (Seattle, WA: IEEE), 12503–12512. doi: 10.1109/CVPR42600.2020.01252

  • CrossRef
  • Google Scholar
  • View reference in article

• 3

  BocchettaM.ToddE. G.BouziguesA.CashD. M.NicholasJ. M.ConveryR. S.et al. (2023). Structural MRI predicts clinical progression in presymptomatic genetic frontotemporal dementia: findings from the GENetic Frontotemporal dementia Initiative cohort. Brain Commun. 5:fcad061. doi: 10.1093/braincomms/fcad061

  • Pubmed Abstract
  • CrossRef
  • Google Scholar
  • View reference in article

• 4

  BuddC.Garcia-Peraza HerreraL. C.HuberM.OurselinS.VercauterenT. (2023). Rapid and robust endoscopic content area estimation: a lean GPU-based pipeline and curated benchmark dataset. Comput. Methods Biomech. Biomed. Eng.: Imaging Visualiz. 11, 1215–1224. doi: 10.1080/21681163.2022.2156393

  • Pubmed Abstract
  • CrossRef
  • Google Scholar
  • View reference in article

• 5

  BulusuS.KailkhuraB.LiB.VarshneyP. K.SongD. (2020). Anomalous example detection in deep learning: a survey. IEEE Access8, 132330–132347. doi: 10.1109/ACCESS.2020.3010274

  • CrossRef
  • Google Scholar
  • View reference in article

• 6

  CaiH.QiL.YuQ.ShiY.GaoY. (2023). “3D medical image segmentation with sparse annotation via cross-teaching between 3D and 2D networks,” in Medical Image Computing and Computer Assisted Intervention-MICCAI 2023: 26th International Conference (Berlin; Heidelberg: Springer Nature Switzerland), 614–624. doi: 10.1007/978-3-031-43898-1_59

  • CrossRef
  • Google Scholar
  • View reference in article

• 7

  CardosoM. J.ModatM.WolzR.MelbourneA.CashD.RueckertD.et al. (2015). Geodesic information flows: spatially-variant graphs and their application to segmentation and fusion. IEEE Trans. Med. Imaging34, 1976–1988. doi: 10.1109/TMI.2015.2418298

  • Pubmed Abstract
  • CrossRef
  • Google Scholar
  • View reference in article

• 8

  CarstensM.RinnerF. M.BodenstedtS.JenkeA. C.WeitzJ.DistlerM.et al. (2023). The dresden surgical anatomy dataset for abdominal organ segmentation in surgical data science. Sci. Data10:3. doi: 10.1038/s41597-022-01719-2

  • Pubmed Abstract
  • CrossRef
  • Google Scholar
  • View reference in article

• 9

  ÇiçekÖ.AbdulkadirA.LienkampS. S.BroxT.RonnebergerO. (2016). 3D U-“Net: learning dense volumetric segmentation from sparse annotation,” in Medical Image Computing and Computer-Assisted Intervention-MICCAI 2016 (Cham: Springer International Publishing), 424–432. doi: 10.1007/978-3-319-46723-8_49

  • CrossRef
  • Google Scholar
  • View reference in article

• 10

  DengC.ZhengC.ChenT.WangC.ZhangJ.LiY.et al. (2024). “HATs: hierarchical adaptive taxonomy segmentation for panoramic pathology image analysis,” in Medical Image Computing and Computer Assisted Intervention-MICCAI 2024 (Cham: Springer Nature Switzerland), 155–166. doi: 10.1007/978-3-031-72083-3_15

  • Pubmed Abstract
  • CrossRef
  • Google Scholar
  • View reference in article

• 11

  DengJ.BergA. C.LiK.Fei-FeiL. (2010). “What does classifying more than 10,000 image categories tell us?,” in Computer Vision-ECCV 2010 (Springer Berlin Heidelberg), 71–84. doi: 10.1007/978-3-642-15555-0_6

  • CrossRef
  • Google Scholar
  • View reference in article

• 12

  DengJ.DongW.SocherR.LiL.-J.LiK.Fei-FeiL. (2009). “ImageNet: a large-scale hierarchical image database,” in IEEE Conference on Computer Vision and Pattern Recognition (CVPR), (Miami, FL: IEEE) 248–255. doi: 10.1109/CVPR.2009.5206848

  • CrossRef
  • Google Scholar
  • View reference in article

• 13

  DorentR.JoutardS.ShapeyJ.KujawaA.ModatM.OurselinS.et al. (2021). “Inter extreme points geodesics for end-to-end weakly supervised image segmentation,” in Medical Image Computing and Computer Assisted Intervention-MICCAI 2021 (Cham: Springer International Publishing), 615–624. doi: 10.1007/978-3-030-87196-3_57

  • CrossRef
  • Google Scholar
  • View reference in article

• 14

  EickhoffS. B.YeoB. T. T.GenonS. (2018). Imaging-based parcellations of the human brain. Nat. Rev. Neurosci. 19, 672–686. doi: 10.1038/s41583-018-0071-7

  • Pubmed Abstract
  • CrossRef
  • Google Scholar
  • View reference in article

• 15

  EnQ.GuoY. (2024). “Annotation by clicks: a point-supervised contrastive variance method for medical semantic segmentation,” in 35th British Machine Vision Conference 2024, BMVC 2024 (Glasgow: BMVA), 1–13.

  • Google Scholar
  • View reference in article

• 16

  FabeloH.OrtegaS.RaviD.KiranB. R.SosaC.BultersD.et al. (2018). Spatio-spectral classification of hyperspectral images for brain cancer detection during surgical operations. PLoS ONE13:e0193721. doi: 10.1371/journal.pone.0193721

  • Pubmed Abstract
  • CrossRef
  • Google Scholar
  • View reference in article

• 17

  FidonL.LiW.Garcia-Peraza-HerreraL. C.EkanayakeJ.KitchenN.OurselinS.et al. (2018). “Generalised wasserstein dice score for imbalanced Multi-Class segmentation using holistic convolutional networks,” in 9th International MICCAI Brainlesion Workshop (Cham: Springer Nature Switzerland), 64–76. doi: 10.1007/978-3-319-75238-9_6

  • CrossRef
  • Google Scholar
  • View reference in article

• 18

  FidonL.ShitS.EzhovI.PaetzoldJ. C.OurselinS.VercauterenT. (2022). “Generalized wasserstein dice loss, test-time augmentation, and transformers for the BraTS 2021 challenge,” in Brainlesion: Glioma, Multiple Sclerosis, Stroke and Traumatic Brain Injuries (Cham: Springer Nature Switzerland), 187–196. doi: 10.1007/978-3-031-09002-8_17

  • CrossRef
  • Google Scholar
  • View reference in article

• 19

  FischlB.SalatD. H.BusaE.AlbertM.DieterichM.HaselgroveC.et al. (2002). Whole brain segmentation: automated labeling of neuroanatomical structures in the human brain. Neuron33, 341–355. doi: 10.1016/S0896-6273(02)00569-X

  • Pubmed Abstract
  • CrossRef
  • Google Scholar
  • View reference in article

• 20

  FrognerC.ZhangC.MobahiH.ArayaM.PoggioT. A. (2015). “Learning with a wasserstein loss,” in Advances in Neural Information Processing Systems, vol. 28 (Montreal, QC: Curran Associates, Inc.), 2053–2061.

  • Google Scholar
  • View reference in article

• 21

  Garcia Peraza HerreraL. C.HorganC.OurselinS.EbnerM.VercauterenT. (2023). Hyperspectral image segmentation: a preliminary study on the oral and dental spectral image database (ODSI-DB). Comput. Methods Biomech. Biomed. Eng.: Imag. Visualiz. 11, 1290–1298. doi: 10.1080/21681163.2022.2160377

  • CrossRef
  • Google Scholar
  • View reference in article

• 22

  GlockerB.ZikicD.KonukogluE.HaynorD. R.CriminisiA. (2013). “Vertebrae localization in pathological spine CT via dense classification from sparse annotations,” in Medical Image Computing and Computer-Assisted Intervention-MICCAI 2013 (Springer Berlin Heidelberg), 262–270. doi: 10.1007/978-3-642-40763-5_33

  • Pubmed Abstract
  • CrossRef
  • Google Scholar
  • View reference in article

• 23

  GonzálezC.GotkowskiK.FuchsM.BucherA.DadrasA.FischbachR.et al. (2022). Distance-based detection of out-of-distribution silent failures for COVID-19 lung lesion segmentation. Med. Image Anal. 82:102596. doi: 10.1016/j.media.2022.102596

  • Pubmed Abstract
  • CrossRef
  • Google Scholar
  • View reference in article

• 24

  GrahamM. S.SudreC. H.VarsavskyT.TudosiuP.-D.NachevP.OurselinS.et al. (2020). “Hierarchical brain parcellation with uncertainty,” in Uncertainty for Safe Utilization of Machine Learning in Medical Imaging, and Graphs in Biomedical Image Analysis (Cham: Springer International Publishing), 23–31. doi: 10.1007/978-3-030-60365-6_3

  • CrossRef
  • Google Scholar
  • View reference in article

• 25

  Guha RoyA.ConjetiS.NavabN.WachingerC. (2019). QuickNAT: a fully convolutional network for quick and accurate segmentation of neuroanatomy. Neuroimage186, 713–727. doi: 10.1016/j.neuroimage.2018.11.042

  • Pubmed Abstract
  • CrossRef
  • Google Scholar
  • View reference in article

• 26

  GuoC.PleissG.SunY.WeinbergerK. Q. (2017). “On calibration of modern neural networks,” in Proceedings of the 34th International Conference on Machine Learning (Sydney, NSW: PMLR), 1321–1330.

  • Google Scholar
  • View reference in article

• 27

  HendrycksD.GimpelK. (2017). “A baseline for detecting misclassified and out-of-distribution examples in neural networks,” in International Conference on Learning Representations (Sydney, NSW: PMLR), 1–12.

  • Google Scholar
  • View reference in article

• 28

  HenschelL.ConjetiS.EstradaS.DiersK.FischlB.ReuterM. (2020). Fastsurfer - a fast and accurate deep learning based neuroimaging pipeline. Neuroimage219:117012. doi: 10.1016/j.neuroimage.2020.117012

  • Pubmed Abstract
  • CrossRef
  • Google Scholar
  • View reference in article

• 29

  HsuY.-C.ShenY.JinH.KiraZ. (2020). “Generalized ODIN: detecting out-of-distribution image without learning from out-of-distribution data,” in 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (Seattle, WA: IEEE), 10948–10957. doi: 10.1109/CVPR42600.2020.01096

  • CrossRef
  • Google Scholar
  • View reference in article

• 30

  HyttinenJ.FältP.JäsbergH.KullaaA.Hauta-KasariM. (2020). Oral and dental spectral image database–ODSI-DB. Appl. Sci. 10:7246. doi: 10.3390/app10207246

  • CrossRef
  • Google Scholar
  • View reference in article

• 31

  IsenseeF.JaegerP. F.KohlS. A. A.PetersenJ.Maier-HeinK. H. (2021). nnU-Net: A self-configuring method for deep learning-based biomedical image segmentation. Nat. Methods18, 203–211. doi: 10.1038/s41592-020-01008-z

  • Pubmed Abstract
  • CrossRef
  • Google Scholar
  • View reference in article

• 32

  IsenseeF.WaldT.UlrichC.BaumgartnerM.RoyS.Maier-HeinK.et al. (2024). “nnU-Net revisited: a call for rigorous validation in 3d medical image segmentation,” in Medical Image Computing and Computer Assisted Intervention-MICCAI 2024 (Cham: Springer Nature Switzerland), 488–498. doi: 10.1007/978-3-031-72114-4_47

  • CrossRef
  • Google Scholar
  • View reference in article

• 33

  JegouS.DrozdzalM.VazquezD.RomeroA.BengioY. (2017). “The one hundred layers tiramisu: fully convolutional densenets for semantic segmentation,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops (Honolulu, HI: IEEE), 11–19. doi: 10.1109/CVPRW.2017.156

  • CrossRef
  • Google Scholar
  • View reference in article

• 34

  KarimiD.GholipourA. (2023). Improving calibration and out-of-distribution detection in deep models for medical image segmentation. IEEE Trans. Artif. Intell. 4, 383–397. doi: 10.1109/TAI.2022.3159510

  • Pubmed Abstract
  • CrossRef
  • Google Scholar
  • View reference in article

• 35

  KhanU.PahedingS.ElkinC. P.DevabhaktuniV. K. (2021). Trends in deep learning for medical hyperspectral image analysis. IEEE Access9, 79534–79548. doi: 10.1109/ACCESS.2021.3068392

  • CrossRef
  • Google Scholar
  • View reference in article

• 36

  KingmaD. P.BaJ. (2017). “Adam: a method for stochastic optimization,” in 3rd International Conference on Learning Representations (ICLR) (San Diego, CA: ICLR), 13.

  • Google Scholar
  • View reference in article

• 37

  KleinA.TourvilleJ. (2012). 101 labeled brain images and a consistent human cortical labeling protocol. Front. Neurosci. 6:171. doi: 10.3389/fnins.2012.00171

  • Pubmed Abstract
  • CrossRef
  • Google Scholar
  • View reference in article

• 38

  KuangZ.YanZ.YuL. (2024). Weakly supervised learning for Multi-Class medical image segmentation via feature decomposition. Comput. Biol. Med. 171:108228. doi: 10.1016/j.compbiomed.2024.108228

  • Pubmed Abstract
  • CrossRef
  • Google Scholar
  • View reference in article

• 39

  KujawaA.DorentR.OurselinS.VercauterenT. (2024). “Label merge-and-split: a graph-colouring approach for memory-efficient brain parcellation,” in Medical Image Computing and Computer Assisted Intervention-MICCAI 2024 (Cham: Springer Nature Switzerland), 350–360. doi: 10.1007/978-3-031-72114-4_34

  • CrossRef
  • Google Scholar
  • View reference in article

• 40

  LambertB.ForbesF.DoyleS.DehaeneH.DojatM. (2024). Trustworthy clinical AI solutions: a unified review of uncertainty quantification in deep learning models for medical image analysis. Artif. Intell. Med. 150:102830. doi: 10.1016/j.artmed.2024.102830

  • Pubmed Abstract
  • CrossRef
  • Google Scholar
  • View reference in article

• 41

  LeT.YamadaM.FukumizuK.CuturiM. (2019). “Tree-sliced variants of wasserstein distances,” in Advances in Neural Information Processing Systems (Vancouver, BC: Curran Associates, Inc.), vol. 32, 1–12.

  • Google Scholar
  • View reference in article

• 42

  LeeK.LeeK.LeeH.ShinJ. (2018). A simple unified framework for detecting out-of-distribution samples and adversarial attacks. In Advances in Neural Information Processing Systems, volume 31, pages 1–12.

  • Google Scholar
  • View reference in article

• 43

  LiL.WangW.ZhouT.QuanR.YangY. (2024a). Semantic hierarchy-aware segmentation. IEEE Trans. Pattern Anal. Mach. Intell. 46, 2123–2138. doi: 10.1109/TPAMI.2023.3332435

  • CrossRef
  • Google Scholar
  • View reference in article

• 44

  LiP.EbnerM.NoonanP.HorganC.BahlA.OurselinS.et al. (2022). Deep learning approach for hyperspectral image demosaicking, spectral correction and high-resolution RGB reconstruction. Comput. Methods Biomech. Biomed. Eng.: Imag. Visualiz. 10, 409–417. doi: 10.1080/21681163.2021.1997646

  • Pubmed Abstract
  • CrossRef
  • Google Scholar
  • View reference in article

• 45

  LiP.MacCormacO.ShapeyJ.VercauterenT. (2024b). “A self-supervised and adversarial approach to hyperspectral demosaicking and RGB reconstruction in surgical imaging,” in 35th British Machine Vision Conference 2024, BMVC 2024 (Glasgow: BMVA Press), 1–15.

  • Google Scholar
  • View reference in article

• 46

  LiangS.LiY.SrikantR. (2018). “Enhancing the reliability of out-of-distribution image detection in neural networks,” in International Conference on Learning Representations (Vancouver, BC: OpenReview.net), 1–12.

  • Google Scholar
  • View reference in article

• 47

  LuH.YanK.ZhangF.WangP.TianJ. (2024). “H2ASeg: hierarchical adaptive interaction and weighting network for tumor segmentation in PET/CT images,” in Medical Image Computing and Computer Assisted Intervention-MICCAI 2024 (Cham: Springer Nature Switzerland), 89–99. doi: 10.1007/978-3-031-72111-3_30

  • CrossRef
  • Google Scholar
  • View reference in article

• 48

  MaJ.ChenJ.NgM.HuangR.LiY.LiC.et al. (2021). Loss odyssey in medical image segmentation. Med. Image Anal. 71:102035. doi: 10.1016/j.media.2021.102035

  • Pubmed Abstract
  • CrossRef
  • Google Scholar
  • View reference in article

• 49

  Martén-PérezA.Martinez-VegaB.VillaM.LeonR.Martinez de TerneroA.FabeloH.et al. (2024). Machine learning performance trends: a comparative study of independent hyperspectral human brain cancer databases. SSRN Electron. J. doi: 10.2139/ssrn.4898113

  • CrossRef
  • Google Scholar
  • View reference in article

• 50

  MenzeB. H.JakabA.BauerS.Kalpathy-CramerJ.FarahaniK.KirbyJ.et al. (2015). The multimodal brain tumor image segmentation benchmark (BRATS). IEEE Trans. Med. Imaging34, 1993–2024. doi: 10.1109/TMI.2014.2377694

  • Pubmed Abstract
  • CrossRef
  • Google Scholar
  • View reference in article

• 51

  MocciaS.WirkertS. J.KenngottH.VemuriA. S.ApitzM.MayerB.et al. (2018). Uncertainty-aware organ classification for surgical data science applications in laparoscopy. IEEE Trans. Biomed. Eng. 65, 2649–2659. doi: 10.1109/TBME.2018.2813015

  • Pubmed Abstract
  • CrossRef
  • Google Scholar
  • View reference in article

• 52

  RavìD.FabeloH.CallicG. M.YangG.-Z. (2017). Manifold embedding and semantic segmentation for intraoperative guidance with hyperspectral brain imaging. IEEE Trans. Med. Imaging36, 1845–1857. doi: 10.1109/TMI.2017.2695523

  • Pubmed Abstract
  • CrossRef
  • Google Scholar
  • View reference in article

• 53

  RonnebergerO.FischerP.BroxT. (2015). “U-Net: convolutional networks for biomedical image segmentation,” in Medical Image Computing and Computer-Assisted Intervention-MICCAI 2015 (Cham: Springer International Publishing), 234–241. doi: 10.1007/978-3-319-24574-4_28

  • CrossRef
  • Google Scholar
  • View reference in article

• 54

  RoyA. G.ConjetiS.SheetD.KatouzianA.NavabN.WachingerC. (2017). “Error corrective boosting for learning fully convolutional networks with limited data,” in Medical Image Computing and Computer Assisted Intervention - MICCAI 2017 (Cham: Springer International Publishing), 231–239. doi: 10.1007/978-3-319-66179-7_27

  • CrossRef
  • Google Scholar
  • View reference in article

• 55

  SeidlitzS.SellnerJ.OdenthalJ.özdemirB.Studier-FischerA.KnödlerS.et al. (2022). Robust deep learning-based semantic organ segmentation in hyperspectral images. Med. Image Anal. 80:102488. doi: 10.1016/j.media.2022.102488

  • Pubmed Abstract
  • CrossRef
  • Google Scholar
  • View reference in article

• 56

  ShapeyJ.XieY.NabaviE.BradfordR.SaeedS. R.OurselinS.et al. (2019). Intraoperative multispectral and hyperspectral label-free imaging: a systematic review of in vivo clinical studies. J. Biophotonics12:e201800455. doi: 10.1002/jbio.201800455

  • Pubmed Abstract
  • CrossRef
  • Google Scholar
  • View reference in article

• 57

  SnoekL.van der MiesenM. M.BeemsterboerT.van der LeijA.EigenhuisA.Steven ScholteH. (2021). The amsterdam open MRI collection, a set of multimodal MRI datasets for individual difference analyses. Sci. Data8:85. doi: 10.1038/s41597-021-00870-6

  • Pubmed Abstract
  • CrossRef
  • Google Scholar
  • View reference in article

• 58

  Studier-FischerA.SeidlitzS.SellnerJ.BressanM.ÒzdemirB.AyalaL.et al. (2023). HeiPorSPECTRAL - the heidelberg porcine HyperSPECTRAL imaging dataset of 20 physiological organs. Sci. Data10:414. doi: 10.1038/s41597-023-02315-8

  • Pubmed Abstract
  • CrossRef
  • Google Scholar
  • View reference in article

• 59

  TakezawaY.SatoR.YamadaM. (2021). “Supervised tree-wasserstein distance,” in Proceedings of the 38th International Conference on Machine Learning (Virtual: PMLR), 10086–10095.

  • Google Scholar
  • View reference in article

• 60

  TanM.LeQ. (2019). “EfficientNet: rethinking model scaling for convolutional neural networks,” in Proceedings of the 36th International Conference on Machine Learning (Long Beach, CA: PMLR), 6105–6114.

  • Google Scholar
  • View reference in article

• 61

  TrajanovskiS.ShanC.WeijtmansP. J. C.de KoningS. G. B.RuersT. J. M. (2021). Tongue tumor detection in hyperspectral images using deep learning semantic segmentation. IEEE Trans. Biomed. Eng. 68, 1330–1340. doi: 10.1109/TBME.2020.3026683

  • Pubmed Abstract
  • CrossRef
  • Google Scholar
  • View reference in article

• 62

  VermaN.MahajanD.SellamanickamS.NairV. (2012). “Learning hierarchical similarity metrics,” in 2012 IEEE Conference on Computer Vision and Pattern Recognition (Providence, RI: IEEE), 2280–2287. doi: 10.1109/CVPR.2012.6247938

  • CrossRef
  • Google Scholar
  • View reference in article

• 63

  WangG.LiW.ZuluagaM. A.PrattR.PatelP. A.AertsenM.et al. (2018). Interactive medical image segmentation using deep learning with image-specific fine tuning. IEEE Trans. Med. Imaging37, 1562–1573. doi: 10.1109/TMI.2018.2791721

  • Pubmed Abstract
  • CrossRef
  • Google Scholar
  • View reference in article

• 64

  WangG.ZuluagaM. A.LiW.PrattR.PatelP. A.AertsenM.et al. (2019). DeepIGeoS: a deep interactive geodesic framework for medical image segmentation. IEEE Trans. Pattern Anal. Mach. Intell. 41, 1559–1572. doi: 10.1109/TPAMI.2018.2840695

  • Pubmed Abstract
  • CrossRef
  • Google Scholar
  • View reference in article

• 65

  WangJ.MaccormacO.RochfordW.KujawaA.ShapeyJ.VercauterenT. (2025a). “Tree-based semantic losses: application to sparsely-supervised large Multi-Class hyperspectral segmentation,” in Medical Image Computing and Computer Assisted Intervention-MICCAI 2025, volume 15967 of Lecture Notes in Computer Science (Cham: Springer Nature Switzerland), 584–594. doi: 10.1007/978-3-032-04984-1_56

  • CrossRef
  • Google Scholar
  • View reference in article

• 66

  WangJ.WangZ.MacCormacO.ShapeyJ.VercauterenT. (2026). OOD-SEG: exploiting out-of-distribution detection techniques for learning image segmentation from sparse multi-class positive-only annotations. Med. Image Anal. 111:104046. doi: 10.1016/j.media.2026.104046

  • Pubmed Abstract
  • CrossRef
  • Google Scholar
  • View reference in article

• 67

  WangJ.XiaB. (2021). “Bounding box tightness prior for weakly supervised image segmentation,” in Medical Image Computing and Computer Assisted Intervention-MICCAI 2021 (Cham: Springer International Publishing), 526–536. doi: 10.1007/978-3-030-87196-3_49

  • CrossRef
  • Google Scholar
  • View reference in article

• 68

  WangZ.YeY.ChenZ.ShuM.ZhangY.XiaY. (2025b). From few to more: scribble-based medical image segmentation via masked context modeling and continuous pseudo labels. IEEE J. Biomed. Health Inform. 30, 2419–2431. doi: 10.1109/JBHI.2025.3599066

  • Pubmed Abstract
  • CrossRef
  • Google Scholar
  • View reference in article

• 69

  WeiJ.HuY.CuiS.ZhouS. K.LiZ. (2023). “WeakPolyp: you only look bounding box for polyp segmentation,” in Medical Image Computing and Computer Assisted Intervention-MICCAI 2023, volume 14222 of Lecture Notes in Computer Science (Cham: Springer Nature Switzerland), 757–766. doi: 10.1007/978-3-031-43898-1_72

  • CrossRef
  • Google Scholar
  • View reference in article

• 70

  XuY.ZhuJ.-Y.ChangE. I.-C.LaiM.TuZ. (2014). Weakly supervised histopathology cancer image segmentation and classification. Med. Image Anal. 18, 591–604. doi: 10.1016/j.media.2014.01.010

  • Pubmed Abstract
  • CrossRef
  • Google Scholar
  • View reference in article

• 71

  YoungA. L.BocchettaM.RussellL. L.ConveryR. S.PeakmanG.ToddE.et al. (2021). Characterizing the clinical features and atrophy patterns of MAPT-related frontotemporal dementia with disease progression modeling. Neurology97, e941–e952. doi: 10.1212/WNL.0000000000012410

  • Pubmed Abstract
  • CrossRef
  • Google Scholar
  • View reference in article

• 72

  ZhaoB.LiF.XingE. (2011). “Large-scale category structure aware image categorization,” in Advances in Neural Information Processing Systems (Granada: Curran Associates, Inc.), vol. 24, 1251–1259.

  • Google Scholar
  • View reference in article