Abstract
Accurate image segmentation plays a crucial role in computer vision and medical image analysis. In this study, we developed a novel uncertainty guided deep learning strategy (UGLS) to enhance the performance of an existing neural network (i.e., U-Net) in segmenting multiple objects of interest from images with varying modalities. In the developed UGLS, a boundary uncertainty map was introduced for each object based on its coarse segmentation (obtained by the U-Net) and then combined with input images for the fine segmentation of the objects. We validated the developed method by segmenting optic cup (OC) regions from color fundus images and left and right lung regions from Xray images. Experiments on public fundus and Xray image datasets showed that the developed method achieved a average Dice Score (DS) of 0.8791 and a sensitivity (SEN) of 0.8858 for the OC segmentation, and 0.9605, 0.9607, 0.9621, and 0.9668 for the left and right lung segmentation, respectively. Our method significantly improved the segmentation performance of the U-Net, making it comparable or superior to five sophisticated networks (i.e., AU-Net, BiO-Net, AS-Net, Swin-Unet, and TransUNet).
1 Introduction
Image segmentation is an important research direction of computer vision and medical image analysis, and widely used as a preprocessing step for various object detection and disease diagnosis (; ). It can divide an image into several disjoint regions by performing a pixel-level classification and largely simplify the assessment of morphological and positional characteristics of object regions (; ). To accurately segment images, a number of image segmentation algorithms have been developed for many different applications, such as threshold based methods (; ), active contour based methods (), and random field based methods (; ). Among these methods, deep learning based methods (; Wang Y. et al., 2022) have gained considerable popularity in the past decade because they can obtain remarkable segmentation performances comparable to manual annotations. Moreover, they are able to automatically extract and flexibly integrate different types of feature information by learning the intrinsic laws and representation levels of images to be segmented.
Despite promising performances, deep learning based methods are often faced with two key challenges in image segmentation (; ), one is how to obtain rich local information, the other is how to robustly extract high-level semantics. Given the large number of parameters in deep learning networks, the spatial resolution of images generally decreases with the increase of network depth in order to speed up the learning of feature information. This resolution decrease can bring about the loss of local information, but the increase of network depth is beneficial to the acquisition of global semantic and context information. To mitigate these two challenges, different deep learning networks (; ) have been constantly emerging to accurately segment images with varying modalities. Alom et al. (Alom et al., 2019) put forward the RU-Net and R2U-Net, respectively by adding different cyclic convolutional blocks to the U-Net for feature detection and accumulation. Seo et al. () proposed a mU-Net model by introducing learnable deconvolution network structures into the U-Net to improve its learning ability at different resolutions and image segmentation performance. Huang et al. () proposed a U-Net 3+ model that combines high-level semantics with low-level semantics using full-scale jump concatenation to overcome the drawbacks of the U-Net and U-Net++ (). Cao et al. () and Chen et al. () proposed different transformer based networks (i.e., Swin-Unet and TransUNet), respectively for accurate image segmentation. These network models demonstrated reasonable segmentation accuracy as compared to the U-Net, but their network structures were often more complex. This may not be conducive to network construction and training as well as image segmentation.
To avoid the design of complex network structures, we develop an uncertainty guided deep learning strategy (UGLS) in this study based on a existing network (i.e., U-Net) for accurate image segmentation. We first train the U-Net to obtain a coarse segmentation result and then use morphological operations and Gaussian filters to identify a potential boundary region for each target object based on the obtained result. The boundary region has a unique intensity distribution to indicate the probability of each pixel belonging to object boundaries and is termed as the boundary uncertainty map (BUM) of the objects. With boundary uncertainty maps and original input images, we retrain the U-Net for the fine segmentation of target objects and can obtain a better performance, as compared to its coarse segmentation performance.
2 Methods
2.1 Scheme overview
Figure 1 shows the entire workflow of the developed deep learning strategy (UGLS) based on a available network (i.e., U-Net) for image segmentation purposes. The UGLS consists of three key steps, namely, the coarse segmentation of target objects, generation of boundary uncertainty maps for each object, and object fine segmentation. The coarse segmentation is used to detect potential object regions and exclude irrelevant background far away from the detected regions. With the coarse segmentation, we can identify the regions where object boundaries are likely to appear and then generate boundary uncertainty maps for these objects, which can largely enhance the information about object boundaries and facilitate the boundary detection. We integrate these uncertainty maps and original input images and feed them into the given network for a more fine segmentation. After performing these three steps, the network can obtain a significantly improved segmentation performance.
FIGURE 1
2.2 Object coarse segmentation
We first trained the U-Net based on the given images and their manual annotations leveraging a plain network training scheme to obtain a relatively coarse segmentation result for desirable objects. This train procedure can be given by:where and indicate the input image and its corresponding prediction map, respectively, denotes the U-Net with the network parameter . The prediction map was relatively coarse as compared with manual annotations of objects because the U-Net has a simple network structure and thereby limited potential to handle images with varying qualities.
2.3 Boundary uncertainty map
The obtained coarse segmentation results were often different from manual annotations of objects in certain image regions, especially object boundary regions, but they can provide some important position information for desirable objects. To effectively use the position information, we processed the coarse segmentation results leveraging morphological dilation and erosion operations (), leading to two different object regions. Based on the two object regions, we can identify a potential boundary region (PBR) and a background excluded image (BEI) for each target object, which were separately given bywhere and are the morphological dilation and erosion operations, respectively, is a circular structuring element with a radius of . The PBR is a binary image and marks the region where object boundaries are most likely to appear, while the BEI merely retains the original image information located in the PBR and can reduce the impact of redundant background in image segmentation, as shown in Figure 2. To take fully advantage of edge position information in coarse segmentation results, we smoothed the PBR using a Gaussian filter with a rectangle window of and a standard deviation of to generate a boundary uncertainty map. The pixels in the uncertainty map took larger values when they were close to the center of the PBR and reduced ones when far away from this center. Moreover, A larger value generally means a higher probability that a pixel in the uncertainty map belongs to object boundaries. The unique intensity distribution made the boundary uncertainty map able to provide more relevant position information about object boundaries, as compared to the PBR.
FIGURE 2
2.4 Object fine segmentation
After obtaining the boundary uncertainty map and background excluded image, we concatenated these two types of images and fed them into the segmentation network. Since the concatenated images were different from the original images and contained very little background information, the segmentation network can easily detect object boundaries and thereby extract the whole object regions accurately using a simple experiment configuration. Specifically, we implemented the fine segmentation of desirable objects using the same configuration as their coarse segmentation (e.g., the cost function, optimizer and batch size).
2.5 Experiment datasets
To validate the developed learning strategy, we performed a series of segmentation experiments on two public dataset, as shown in Figure 3. The first dataset was from the Retinal Fundus Glaucoma Challenge (REFUGE) () and contained 1,200 retinal fundus images acquired by two different cameras, together with manual annotations for the optic disc (OD) and cup (OC) regions. These images and their annotations were evenly split into three subsets for training (n = 400), validation (n = 400) and testing (n = 400) purposes, respectively, in the REFUGE challenge, which were also used in this study for segmentation purposes. We normalized these images to reduce the influence of light exposure and cameras and then extracted local disc patches using the dimensions that approximated three times the radius of the OD regions (). The extracted patches were then resized to 256 × 256 pixels and fed into the U-Net for network training.
FIGURE 3
The second dataset was from a tuberculosis screening program in Montgomery County (TSMC) () and contained 138 chest Xray images acquired using a Eureka stationary Xray machine. Among these Xray images, 80 were normal and 58 were abnormal with manifestations of tuberculosis. All images were de-identified and had a dimension of either 4,020 × 4,892 or 4,892 × 4,020 pixels. The left and right lungs depicted on these Xray images were manually annotated by a radiologist. We also split these Xray images equally into three disjoint subsets for network training (n = 46), validation (n = 46) and testing (n = 46), and resized them to the same dimension of 256 × 256 pixels.
2.6 Performance evaluation
We assessed the performance of the UGLS based on the U-Net (short for the developed method, https://github.com/wmuLei/ODsegmentation) on a 64-bit Windows 10 PC with 2.20 GHz 2.19 GHz Intel(R) Xeon(R) Gold 5120 CPU, 64 GB RAM and NVIDIA GeForce GTX 2080Ti by segmenting 1) the OC region from color fundus images and 2) the left and right lungs from the Xray images, where the was set to 25 and 35, respectively for these two datasets. We used the Dice Score (DS) () as the cost function to assess the similarity between the segmentation results and their corresponding manual annotations for each object:where denotes the DS for object , and is the total number of objects of interest. and are the output probabilities of a specific input image obtained by the U-Net and manual annotation, respectively for pixel and object , denotes the entire image domain. We used the RMSprop optimizer to maximize the cost function and set its initial learning rate to 0.001, along with a batch size of eight and an epoch number of 100. To reduce the network training time, we halted the entire training procedure when the performance of the U-Net did not increase for 20 consecutive epochs. In addition, we randomly augmented input images during network training using some transformations, such as horizontal/vertical flip, scaling from 0.9 to 1.1, translation by −10 to 10 percent per axis, rotation from −180 to 180 in degree, and shearing from −5 to 5 in degree. After training, we binarized the prediction map of the U-Net using a given threshold of 0.5 to obtain desirable output results.
With these output results, we evaluated our developed method using the DS, Matthew’s correlation coefficient (MCC) (), sensitivity (SEN) (), and Hausdorff distance (HSD, in pixel).where , , and denote the true positive, false positive, true negative and false negative, respectively. is the directed HSD from point set to . The larger the DS, MCC and SEN are and the smaller the HSD is, the better the segmentation performance of the network is. To show the advantage of the UGLS, we compared the developed method with the Attention U-Net (AU-Net) (), BiO-Net (), asymmetric U-Net (AS-Net) (), Swin-Unet (in tiny scale version), and TransUNet. Among these networks, U-Net and its variants (i.e., AU-Net, BiO-Net, AS-Net) shared the similar network architecture (e.g., the number of convolution filters increased from 32 to 1,024) and were trained from scratch based on a given dimension of 256 × 256 pixels and a learning rate of 0.001, while Swin-Unet and TransUNet were trained from initial ImageNet weights based on a dimension of 224 × 224 pixels and a learning rate of 0.01. All these networks were trained six times (by randomly arranging three different subsets for network training, validation and testing, respectively) using the same configurations (except for image dimension and learning rate) for each dataset. The paired t-test was used to evaluate the differences among the involved networks on the DS metric. A p-value less than 0.05 was considered statistically significant ().
3 Results
3.1 Object coarse segmentation
Tables 1 and 2 summarized six coarse segmentation results of the U-Net with the developed UGLS strategy in extracting the OC from retinal fundus images and the left and right lungs from Xray images, respectively. As demonstrated by the results, the U-Net achieved a relatively low performance in segmenting the OC depicted on fundus images (due to the high similarity between the OD and OC regions), with a average DS, MCC, SEN and HSD of 0.8642, 0.8585, 0.8674 and 2.6420, respectively. In contrast, it obtained a better accuracy for the left and right lungs (with the average DS of 0.9408 and 0.9477, respectively) and can compete with their manual annotations.
TABLE 1
| Object | Result | DS | MCC | SEN | HSD |
|---|---|---|---|---|---|
| Mean ± SD | Mean ± SD | Mean ± SD | Mean ± SD | ||
| OC | Seg1 | 0.8687 ± 0.0636 | 0.8629 ± 0.0627 | 0.8654 ± 0.1076 | 2.6266 ± 0.6213 |
| Seg2 | 0.8439 ± 0.0767 | 0.8380 ± 0.0749 | 0.8592 ± 0.1363 | 2.7302 ± 0.6845 | |
| Seg3 | 0.8784 ± 0.0579 | 0.8720 ± 0.0563 | 0.9344 ± 0.0856 | 2.5711 ± 0.5217 | |
| Seg4 | 0.8646 ± 0.0769 | 0.8586 ± 0.0757 | 0.8508 ± 0.1239 | 2.5952 ± 0.6782 | |
| Seg5 | 0.8399 ± 0.0886 | 0.8358 ± 0.0847 | 0.7980 ± 0.1413 | 2.7719 ± 0.7082 | |
| Seg6 | 0.8898 ± 0.0595 | 0.8839 ± 0.0590 | 0.8969 ± 0.0861 | 2.4492 ± 0.5737 | |
| Overall | 0.8642 ± 0.0736 | 0.8585 ± 0.0717 | 0.8674 ± 0.1230 | 2.6240 ± 0.6435 |
Results of our proposed method for the coarse segmentation of the OC regions based on six experiments (i.e., Seg1-6) in terms of the mean and standard deviation (SD) of DS, MCC, SEN and HSD (in pixel).
TABLE 2
| Object | Result | DS | MCC | SEN | HSD |
|---|---|---|---|---|---|
| Mean ± SD | Mean ± SD | Mean ± SD | Mean ± SD | ||
| LL | Seg1 | 0.9763 ± 0.0197 | 0.9733 ± 0.0216 | 0.9672 ± 0.0369 | 3.9530 ± 1.5336 |
| Seg2 | 0.9551 ± 0.0401 | 0.9512 ± 0.0399 | 0.9258 ± 0.0675 | 4.3733 ± 1.0238 | |
| Seg3 | 0.9019 ± 0.1429 | 0.8938 ± 0.1520 | 0.9258 ± 0.0899 | 5.3338 ± 2.3926 | |
| Seg4 | 0.9546 ± 0.0924 | 0.9521 ± 0.0906 | 0.9569 ± 0.0168 | 4.2231 ± 2.0033 | |
| Seg5 | 0.9540 ± 0.0870 | 0.9518 ± 0.0798 | 0.9838 ± 0.0135 | 4.1766 ± 1.8032 | |
| Seg6 | 0.9028 ± 0.0846 | 0.8977 ± 0.0789 | 0.9899 ± 0.0197 | 5.9666 ± 1.5560 | |
| Overall | 0.9408 ± 0.0917 | 0.9367 ± 0.0925 | 0.9582 ± 0.0558 | 4.6711 ± 1.9139 | |
| Object | Result | DS | MCC | SEN | HSD |
| Mean ± SD | Mean ± SD | Mean ± SD | Mean ± SD | ||
| RL | Seg1 | 0.9656 ± 0.0418 | 0.9616 ± 0.0433 | 0.9826 ± 0.0500 | 4.6004 ± 1.6841 |
| Seg2 | 0.9519 ± 0.0464 | 0.9471 ± 0.0464 | 0.9242 ± 0.0787 | 5.0881 ± 1.1806 | |
| Seg3 | 0.8962 ± 0.1268 | 0.8807 ± 0.1519 | 0.8991 ± 0.0567 | 6.6095 ± 2.9507 | |
| Seg4 | 0.9552 ± 0.0893 | 0.9512 ± 0.0920 | 0.9537 ± 0.0181 | 5.0575 ± 2.2573 | |
| Seg5 | 0.9644 ± 0.0899 | 0.9613 ± 0.0896 | 0.9784 ± 0.0407 | 4.6336 ± 1.5349 | |
| Seg6 | 0.9531 ± 0.0707 | 0.9496 ± 0.0687 | 0.9655 ± 0.0543 | 4.7648 ± 1.7171 | |
| Overall | 0.9477 ± 0.0860 | 0.9419 ± 0.0940 | 0.9506 ± 0.0608 | 5.1257 ± 2.0893 |
The performance of the developed method for segmenting the left and right lungs (LL and RL) from Xray images.
3.2 Object fine segmentation
Tables 3 and 4 demonstrated the fine segmentation results of the U-Net with the developed UGLS strategy for three different objects depicted on fundus and Xray images, respectively. The U-Net achieved the average DS and SEN of 0.8791 and 0.8858 for the OC region, and 0.9605, 0.9607, 0.9621, and 0.9668 for the left and righ lungs, respectively. As compared with its coarse segmentation results, the U-Net obtained a significantly better overall performance for six different experiments on two types of images with varying modalities (p < 0.01). Specifically, the U-Net had better performances for five fine segmentation experiments for the OC, as compared to its coarse results, as shown in Table 3. Similarly, its performances were also increased in large increments for each experiment in the fine segmentation of the left and right lungs.
TABLE 3
| Object | Result | DS | MCC | SEN | HSD |
|---|---|---|---|---|---|
| Mean ± SD | Mean ± SD | Mean ± SD | Mean ± SD | ||
| OC | Seg1 | 0.8696 ± 0.0647 | 0.8638 ± 0.0633 | 0.8804 ± 0.1036 | 2.5887 ± 0.6049 |
| Seg2 | 0.8442 ± 0.0755 | 0.8384 ± 0.0735 | 0.8764 ± 0.1306 | 2.7324 ± 0.6512 | |
| Seg3 | 0.8691 ± 0.0661 | 0.8623 ± 0.0661 | 0.8835 ± 0.1147 | 2.5945 ± 0.6272 | |
| Seg4 | 0.8982 ± 0.0547 | 0.8938 ± 0.0521 | 0.9141 ± 0.1027 | 2.2824 ± 0.4505 | |
| Seg5 | 0.9033 ± 0.0527 | 0.8993 ± 0.0509 | 0.8573 ± 0.0922 | 2.2555 ± 0.5148 | |
| Seg6 | 0.8904 ± 0.0612 | 0.8843 ± 0.0608 | 0.9033 ± 0.0821 | 2.4461 ± 0.5930 | |
| Overall | 0.8791 ± 0.0662 | 0.8737 ± 0.0651 | 0.8858 ± 0.1071 | 2.4833 ± 0.6031 |
Fine segmentation results of the developed method for the OC regions in terms of the DS, MCC, SEN and HSD (in pixel) metrics.
TABLE 4
| Object | Result | DS | MCC | SEN | HSD |
|---|---|---|---|---|---|
| Mean ± SD | Mean ± SD | Mean ± SD | Mean ± SD | ||
| LL | Seg1 | 0.9764 ± 0.0206 | 0.9735 ± 0.0224 | 0.9709 ± 0.0349 | 3.9756 ± 1.4742 |
| Seg2 | 0.9636 ± 0.0405 | 0.9601 ± 0.0415 | 0.9552 ± 0.0612 | 4.0821 ± 0.9145 | |
| Seg3 | 0.9291 ± 0.1325 | 0.9229 ± 0.1403 | 0.9148 ± 0.1511 | 4.6107 ± 1.8065 | |
| Seg4 | 0.9619 ± 0.0940 | 0.9585 ± 0.1008 | 0.9675 ± 0.1007 | 3.6484 ± 1.5086 | |
| Seg5 | 0.9680 ± 0.0412 | 0.9650 ± 0.0416 | 0.9794 ± 0.0182 | 3.9206 ± 1.2240 | |
| Seg6 | 0.9639 ± 0.0565 | 0.9615 ± 0.0557 | 0.9762 ± 0.0341 | 4.1280 ± 1.7205 | |
| Overall | 0.9605 ± 0.0760 | 0.9569 ± 0.0800 | 0.9607 ± 0.0840 | 4.0609 ± 1.5006 | |
| Object | Result | DS | MCC | SEN | HSD |
| Mean ± SD | Mean ± SD | Mean ± SD | Mean ± SD | ||
| RL | Seg1 | 0.9661 ± 0.0625 | 0.9630 ± 0.0616 | 0.9749 ± 0.0512 | 4.4255 ± 1.6023 |
| Seg2 | 0.9657 ± 0.0356 | 0.9618 ± 0.0365 | 0.9541 ± 0.0636 | 4.7426 ± 1.1453 | |
| Seg3 | 0.9541 ± 0.0576 | 0.9480 ± 0.0636 | 0.9473 ± 0.0696 | 5.1637 ± 1.5636 | |
| Seg4 | 0.9664 ± 0.0736 | 0.9625 ± 0.0809 | 0.9738 ± 0.0387 | 4.4790 ± 1.5192 | |
| Seg5 | 0.9685 ± 0.0632 | 0.9653 ± 0.0654 | 0.9773 ± 0.0407 | 4.4856 ± 1.4704 | |
| Seg6 | 0.9517 ± 0.0885 | 0.9491 ± 0.0850 | 0.9732 ± 0.0493 | 4.7916 ± 1.8466 | |
| Overall | 0.9621 ± 0.0658 | 0.9583 ± 0.0677 | 0.9668 ± 0.0546 | 4.6813 ± 1.5598 |
Fine segmentation results of the developed method for segmenting the left and right lungs (LL and RL) from the Xray images in terms of the DS, MCC, SEN and HSD (in pixel) metrics.
3.3 Performance comparison
Table 5 summarized the segmentation results of the involved networks (i.e., the U-Net, AU-Net, BiO-Net, AS-Net, Swin-Unet, and TransUNet) in extracting three different objects from fund and Xray images, respectively. As demonstrated by these results, the developed UGLS strategy can significantly improve the performance of the U-Net (p < 0.01) by merely leveraging the its coarse segmentation results in a reasonable way, instead of changing its network structure. Specifically, the average DS of the U-Net increased from 0.8792 to 0.8945 for three different object regions depicted on fundus and Xray images after using our developed deep learning strategy. This strategy made our developed method superior or comparable to the AU-Net (0.8803, p < 0.001), BiO-Net (0.8843, p < 0.005), AS-Net (0.8859, p < 0.005), Swin-Unet (0.8811, p < 0.001), and TransUNet (0.8900, p < 0.05) with all the p-values less than 0.05 for the two segmentation tasks. Figures 4 and 5 showed the performance differences among the involved networks on several fundus and Xray images.
TABLE 5
| Object | Method | DS | MCC | SEN | HSD |
|---|---|---|---|---|---|
| Mean ± SD | Mean ± SD | Mean ± SD | Mean ± SD | ||
| OC | U-Net | 0.8642 ± 0.0736 | 0.8585 ± 0.0717 | 0.8674 ± 0.1230 | 2.6240 ± 0.6435 |
| AU-Net | 0.8619 ± 0.0780 | 0.8569 ± 0.0746 | 0.8667 ± 0.1293 | 2.5896 ± 0.6180 | |
| BiO-Net | 0.8663 ± 0.0721 | 0.8605 ± 0.0704 | 0.8801 ± 0.1181 | 2.5960 ± 0.6374 | |
| AS-Net | 0.8676 ± 0.0725 | 0.8620 ± 0.0701 | 0.8757 ± 0.1186 | 2.6119 ± 0.6281 | |
| Swin-Unet | 0.8647 ± 0.0706 | 0.8582 ± 0.0689 | 0.8799 ± 0.1145 | 2.7134 ± 0.5767 | |
| TransUNet | 0.8737 ± 0.0643 | 0.8679 ± 0.0620 | 0.8894 ± 0.1071 | 2.4974 ± 0.5416 | |
| Proposed | 0.8791 ± 0.0662 | 0.8737 ± 0.0651 | 0.8858 ± 0.1071 | 2.4833 ± 0.6031 | |
| Object | Method | DS | MCC | SEN | HSD |
| Mean ± SD | Mean ± SD | Mean ± SD | Mean ± SD | ||
| LL | U-Net | 0.9408 ± 0.0917 | 0.9367 ± 0.0925 | 0.9582 ± 0.0558 | 4.6711 ± 1.9139 |
| AU-Net | 0.9607 ± 0.0719 | 0.9572 ± 0.0758 | 0.9649 ± 0.0502 | 4.1158 ± 1.8849 | |
| BiO-Net | 0.9614 ± 0.0681 | 0.9582 ± 0.0673 | 0.9637 ± 0.0543 | 4.1554 ± 1.6769 | |
| AS-Net | 0.9649 ± 0.0493 | 0.9609 ± 0.0506 | 0.9623 ± 0.0633 | 4.7912 ± 1.6847 | |
| Swin-Unet | 0.9502 ± 0.0316 | 0.9446 ± 0.0317 | 0.9530 ± 0.0479 | 5.0613 ± 0.7958 | |
| TransUNet | 0.9617 ± 0.0357 | 0.9574 ± 0.0377 | 0.9604 ± 0.0476 | 4.1025 ± 1.0577 | |
| Proposed | 0.9605 ± 0.0760 | 0.9569 ± 0.0800 | 0.9607 ± 0.0840 | 4.0609 ± 1.5006 | |
| Object | Method | DS | MCC | SEN | HSD |
| Mean ± SD | Mean ± SD | Mean ± SD | Mean ± SD | ||
| RL | U-Net | 0.9477 ± 0.0860 | 0.9419 ± 0.0940 | 0.9506 ± 0.0608 | 5.1257 ± 2.0893 |
| AU-Net | 0.9595 ± 0.0736 | 0.9545 ± 0.0804 | 0.9674 ± 0.0444 | 4.7496 ± 1.8938 | |
| BiO-Net | 0.9637 ± 0.0651 | 0.9597 ± 0.0661 | 0.9693 ± 0.0483 | 4.7352 ± 1.6805 | |
| AS-Net | 0.9663 ± 0.0516 | 0.9628 ± 0.0552 | 0.9636 ± 0.0537 | 4.1806 ± 1.6053 | |
| Swin-Unet | 0.9549 ± 0.0280 | 0.9488 ± 0.0281 | 0.9562 ± 0.0457 | 5.6140 ± 0.8730 | |
| TransUNet | 0.9602 ± 0.0407 | 0.9552 ± 0.0404 | 0.9606 ± 0.0421 | 4.8674 ± 1.1996 | |
| Proposed | 0.9621 ± 0.0658 | 0.9583 ± 0.0677 | 0.9668 ± 0.0546 | 4.6813 ± 1.5598 |
Performance differences among the involved networks in segmenting the OC, left and right lungs depicted on fundus and Xray images, respectively.
FIGURE 4
FIGURE 5
3.4 Effect of the BUM
Table 6 showed the results of the developed method in extracting the left and right lungs from Xray images using boundary uncertainty maps in three different ways. As demonstrated by the results, our developed method obtained the lowest segmentation performance, with the average DS of 0.9437 when merely trained on boundary uncertainty maps, but it had increased performance when combining the uncertainty maps with the original images or their background excluded version for network training (with the average DS of 0.9611 and 0.9613). Moreover, the background excluded images can better improve the performance of our developed method since they reduced the impact of irrelevant background information away from desirable objects.
TABLE 6
| Object | Method | DS | MCC | SEN | HSD |
|---|---|---|---|---|---|
| Mean ± SD | Mean ± SD | Mean ± SD | Mean ± SD | ||
| LL | BUM | 0.9347 ± 0.1227 | 0.9297 ± 0.1245 | 0.9356 ± 0.1284 | 4.3859 ± 1.5356 |
| ORI + BUM | 0.9587 ± 0.0920 | 0.9553 ± 0.0956 | 0.9603 ± 0.0936 | 4.0907 ± 1.6055 | |
| BEI + BUM | 0.9605 ± 0.0760 | 0.9569 ± 0.0800 | 0.9607 ± 0.0840 | 4.0609 ± 1.5006 | |
| Object | Method | DS | MCC | SEN | HSD |
| Mean ± SD | Mean ± SD | Mean ± SD | Mean ± SD | ||
| RL | BUM | 0.9527 ± 0.0743 | 0.9476 ± 0.0771 | 0.9517 ± 0.0819 | 5.0905 ± 1.5155 |
| ORI + BUM | 0.9634 ± 0.0683 | 0.9597 ± 0.0708 | 0.9651 ± 0.0636 | 4.6374 ± 1.5137 | |
| BEI + BUM | 0.9621 ± 0.0658 | 0.9583 ± 0.0677 | 0.9668 ± 0.0546 | 4.6813 ± 1.5598 |
The results of the developed method trained on the boundary uncertainty map (BUM) or its combination with the original image (ORI) or its background excluded version (BEI) for the left and right lung segmentation.
3.5 Effect of parameter
Table 7 summarized the impact of the parameter on the performance of the developed method in segmenting three different objects from fundus and Xray images. The developed method achieved the best overall performance when this parameter was set to 25 in the OC segmentation and 35 in the left and right lung segmentation, respectively, for the morphological operations and Gaussian filter. These two parameter values ensured a good balance between object information and irrelevant background for our developed method, making it able to accurately detect object boundaries. Table 8 showed the performance of the developed method when using different values for the parameters in the morphological operations and Gaussian filter. From the table, our developed method obtained a superior overall performance when the morphological operations and Gaussian filter shared the same value for each image dataset, which can effectively highlight the center regions of boundary uncertainty maps, as shown in Figure 6.
TABLE 7
| Object | DS | MCC | SEN | HSD | |
|---|---|---|---|---|---|
| Mean ± SD | Mean ± SD | Mean ± SD | Mean ± SD | ||
| OC | 15 | 0.8678 ± 0.0696 | 0.8626 ± 0.0680 | 0.8622 ± 0.1220 | 2.5426 ± 0.6246 |
| 25 | 0.8791 ± 0.0662 | 0.8737 ± 0.0651 | 0.8858 ± 0.1071 | 2.4833 ± 0.6031 | |
| 35 | 0.8735 ± 0.0655 | 0.8679 ± 0.0639 | 0.8848 ± 0.1112 | 2.5257 ± 0.5995 | |
| Object | DS | MCC | SEN | HSD | |
| Mean ± SD | Mean ± SD | Mean ± SD | Mean ± SD | ||
| LL | 15 | 0.9512 ± 0.1117 | 0.9474 ± 0.1173 | 0.9519 ± 0.1114 | 4.1688 ± 1.6956 |
| 25 | 0.9543 ± 0.0968 | 0.9512 ± 0.0964 | 0.9566 ± 0.0987 | 4.1234 ± 1.6199 | |
| 35 | 0.9605 ± 0.0760 | 0.9569 ± 0.0800 | 0.9607 ± 0.0840 | 4.0609 ± 1.5006 | |
| 45 | 0.9566 ± 0.0996 | 0.9537 ± 0.0993 | 0.9582 ± 0.1017 | 4.1346 ± 1.5093 | |
| 55 | 0.9554 ± 0.1014 | 0.9523 ± 0.1034 | 0.9584 ± 0.1023 | 4.0513 ± 1.5453 | |
| Object | DS | MCC | SEN | HSD | |
| Mean ± SD | Mean ± SD | Mean ± SD | Mean ± SD | ||
| RL | 15 | 0.9614 ± 0.0649 | 0.9578 ± 0.0643 | 0.9627 ± 0.0616 | 4.7130 ± 1.5723 |
| 25 | 0.9617 ± 0.0713 | 0.9578 ± 0.0725 | 0.9646 ± 0.0529 | 4.6753 ± 1.6561 | |
| 35 | 0.9621 ± 0.0658 | 0.9583 ± 0.0677 | 0.9668 ± 0.0546 | 4.6813 ± 1.5598 | |
| 45 | 0.9625 ± 0.0683 | 0.9589 ± 0.0692 | 0.9662 ± 0.0558 | 4.6934 ± 1.5860 | |
| 55 | 0.9636 ± 0.0646 | 0.9602 ± 0.0641 | 0.9662 ± 0.0559 | 4.6227 ± 1.5216 |
The results of the developed method on fundus and Xray images by setting different values for parameters .
TABLE 8
| Object | (, ) | DS | MCC | SEN | HSD |
|---|---|---|---|---|---|
| Mean ± SD | Mean ± SD | Mean ± SD | Mean ± SD | ||
| OC | (25, 15) | 0.8669 ± 0.0651 | 0.8611 ± 0.0639 | 0.8748 ± 0.1064 | 2.5936 ± 0.6169 |
| (25, 25) | 0.8696 ± 0.0647 | 0.8638 ± 0.0633 | 0.8804 ± 0.1036 | 2.5887 ± 0.6049 | |
| (25, 35) | 0.8675 ± 0.0655 | 0.8619 ± 0.0642 | 0.8708 ± 0.1083 | 2.5815 ± 0.6131 | |
| Object | (, ) | DS | MCC | SEN | HSD |
| Mean ± SD | Mean ± SD | Mean ± SD | Mean ± SD | ||
| LL | (35, 15) | 0.9695 ± 0.0333 | 0.9663 ± 0.0354 | 0.9679 ± 0.0459 | 4.2534 ± 1.7802 |
| (35, 25) | 0.9733 ± 0.0259 | 0.9703 ± 0.0275 | 0.9702 ± 0.0378 | 4.1451 ± 1.7235 | |
| (35, 35) | 0.9764 ± 0.0206 | 0.9735 ± 0.0224 | 0.9709 ± 0.0349 | 3.9756 ± 1.4742 | |
| (35, 45) | 0.9750 ± 0.0262 | 0.9722 ± 0.0277 | 0.9752 ± 0.0357 | 3.9758 ± 1.5872 | |
| Object | (, ) | DS | MCC | SEN | HSD |
| Mean ± SD | Mean ± SD | Mean ± SD | Mean ± SD | ||
| RL | (35, 15) | 0.9681 ± 0.0459 | 0.9646 ± 0.0465 | 0.9754 ± 0.0508 | 4.4583 ± 1.5158 |
| (35, 25) | 0.9698 ± 0.0440 | 0.9666 ± 0.0446 | 0.9752 ± 0.0533 | 4.3794 ± 1.5588 | |
| (35, 35) | 0.9661 ± 0.0625 | 0.9630 ± 0.0616 | 0.9749 ± 0.0512 | 4.4255 ± 1.6023 | |
| (35, 45) | 0.9639 ± 0.0627 | 0.9610 ± 0.0608 | 0.9755 ± 0.0527 | 4.5328 ± 1.5872 |
The results of the developed method for the first experiment on fundus and Xray images using different values for parameter in morphological operations and Gaussian filter (short for and , respectively).
FIGURE 6
4 Discussion
In this paper, we developed a novel network training strategy (termed UGLS) for accurate image segmentation and assessed its effectiveness based on an existing network (i.e., the U-Net) by extracting three different objects depicted (i.e., the OC, left and right lungs) on fundus and Xray images. In the developed method, the U-Net was first trained using the traditional training strategy on the original images and their manual annotations for the coarse-grained segmentation of desirable objects. The segmentation results were then proposed to locate a potential boundary region for each object, which was combined with the original images for the fine segmentation of the objects. We validated the developed method on two public datasets (i.e., REFUGE and TSMC) and compared it with five available networks (i.e., the AU-Net, BiO-Net, AS-Net, Swin-Unet and TransUNet) under the similar experiment configurations. Extensive experiments showed that the developed method can largely improve the segmentation performance of the U-Net and was comparable or superior to the AU-Net, BiO-Net, AS-Net, Swin-Unet and TransUNet, all of which had much more complex network structures than the U-Net.
The developed method achieved promising overall performance in segmenting multiple different objects, as compared to three existing networks. This may be attributed to the following reasons: First, the coarse segmentation of the objects was able to detect various types of image features and provide some important location information for each object and its boundaries. Second, the introduction of boundary uncertainty maps made the potential boundary region have a unique intensity distribution. This distribution largely facilitated the detection of object boundaries and enhanced the sensitivity and accuracy of the U-Net in segmenting objects of interest. Third, the use of background excluded images can not only ensure a reasonable balance between object information and its surrounding background, but also ensure that the U-Net performs the learning of various features in the specified region, thereby leading to a increased segmentation performance and a reduced influence of undesirable background. Due to these reasons, the developed method can significantly improve the segmentation performance of a relatively simple network (i.e., the U-Net) and make it comparable or superior to several existing sophisticated networks.
We further assessed the influence of boundary uncertainty maps and the parameter on the performance of the developed method. Segmentation results in Tables 6–8 showed that (Eq. 1) the developed method achieved better segmentation performance when trained on the combination of boundary uncertainty maps and the background excluded images, as compared to the counterparts trained merely on boundary uncertainty maps or the original images. This may be due to the fact that there are no enough texture information relative to targe objects and their boundaries in boundary uncertainty maps, but too much background information in the original images, both of which can reduce the learning potential of the U-Net and deteriorate its segmentation performance. 2) The developed method obtained relatively high segmentation accuracy when the parameter was assigned to 25 for the OC segmentation and 35 for the left and right lung segmentation. This parameter controlled the amount of information about desirable objects and their surrounding background in the boundary uncertainty maps. A proper value for the parameter can ensure a good balance between the two types of image information and significantly improve the fine segmentation performance of our developed method. If the parameter value was set too small or large, our developed method would have a final result that was very close to its coarse segmentation results or contained lots of undesirable background. 3) The parameter was used simultaneously in morphological operations and Gaussian filter since it can ensure that pixels in the center region of boundary uncertainty map have more high contrast or intensity, as compared to the counterparts in other regions. 4) Boundary uncertainty maps can be generated using different strategies, but their corresponding segmentation performance was very similar (i.e., 0.8791 vs. 0.8721 for the OC segmentation), based on our previous study ().
5 Conclusion
We developed a uncertainty guided deep learning strategy (UGLS) to improve the performance of existing segmentation neural networks and validated it based on the classical U-Net by segmenting the OC from color fundus images and the left and right lungs from Xray images. The novelty of our developed method lies in the introduction of boundary uncertainty maps and their integration with the input images for accurate image segmentation. Extensive experiments on public fundus and Xray image datasets demonstrated that the developed method had the potential to effectively extract the OC from fundus images and the left and right lungs from Xray images, largely improved the performance of the U-Net, and can compete with several sophisticated networks (i.e., the AU-Net, BiO-Net, AS-Net, Swin-Net, and TransUNet).
Statements
Data availability statement
Publicly available datasets were analyzed in this study. This data can be found here: Retinal Fundus Glaucoma Challenge (REFUGE), a tuberculosis screening program in Montgomery County.
Author contributions
XY: Methodology, Software, Writing–original draft. YZ: Resources, Validation, Writing–original draft. CM: Formal Analysis, Investigation, Visualization, Writing–review and editing. GJ: Methodology, Software, Visualization, Writing–original draft. BT: Investigation, Validation, Visualization, Writing–original draft. LW: Methodology, Supervision, Writing–review and editing.
Funding
The author(s) declare that financial support was received for the research, authorship, and/or publication of this article. This work is supported in part by Natural Science Foundation of Zhejiang Province (Grant No. LGF22H090022 and LTGY23H090014), Health Commission of Zhejiang Province (Grant No. 2022KY1206), and Wenzhou Municipal Science and Technology Bureau (Grant No. ZY2022024).
Conflict of interest
The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.
Publisher’s note
All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.
References
1
AlomM. Z.YakopcicC.HasanM.TahaT. M.AsariV. K. (2019). Recurrent residual U-Net for medical image segmentation. J. Med. Imaging (Bellingham)6, 014006. 10.1117/1.Jmi.6.1.014006
2
CaoH.WangY.ChenJ.JiangD.ZhangX.TianQ.et al (2022). Swin-unet: unet-like pure transformer for medical image segmentation. Eur. Conf. Comput. Vis. (ECCV).
3
ChenJ.LuY.YuQ.LuoX.AdeliE.WangY.et al “TransUNet: transformers make strong encoders for medical image segmentation,” in 2021 IEEE/CVF Conference on Computer Vision and Pattern, Nashville, TN, USA, June 2021.
4
FangX.ShenY.ZhengB.ZhuS.WuM. “Optic disc segmentation based on phase-fusion PSPNet,” in Proceedings of the 2nd International Symposium on Artificial Intelligence for Medicine Sciences, Beijing, China, October 2021 (Association for Computing Machinery).
5
GawlikowskiJ.TassiC.AliM.LeeJ.HumtM.FengJ.et al (2023). A survey of uncertainty in deep neural networks. Artif. Intell. Rev.56, 1513–1589. 10.1007/s10462-023-10562-9
6
HanD. J. J. O. I.Graphics (2006). A survey on image segmentation using active contour and level set method. Neurocomputing452. 10.1016/j.neucom.2020.07.141
7
HossainN. I.RezaS. “Blood vessel detection from fundus image using Markov random field based image segmentation,” in 2017 4th International Conference on Advances in Electrical Engineering ICAEE, Dhaka, Bangladesh, 28-30 September 2017, 123–127.
8
HuangH.LinL.TongR.HuH.ZhangQ.IwamotoY.et al “UNet 3+: a full-scale connected UNet for medical image segmentation,” in ICASSP 2020 - 2020 IEEE International Conference on Acoustics, Speech and Signal Processing ICASSP, Barcelona, Spain, 4-8 May 2020, 1055–1059.
9
JaegerS.CandemirS.AntaniS.WángY. X.LuP. X.ThomaG. (2014). Two public chest X-ray datasets for computer-aided screening of pulmonary diseases. Quant. Imaging Med. Surg.4, 475–477. 10.3978/j.issn.2223-4292.2014.11.20
10
JunT. J.KweonJ.KimY. H.KimD. (2020). T-Net: nested encoder-decoder architecture for the main vessel segmentation in coronary angiography. Neural Netw.128, 216–233. 10.1016/j.neunet.2020.05.002
11
KhenedM.VargheseA.KrishnamurthiG. J. M. I. A. (2018). Fully convolutional multi‐scale residual DenseNets for cardiac segmentation and automated cardiac diagnosis using ensemble of classifiers. Med. Image Anal.51, 21–45. 10.1016/j.media.2018.10.004
12
LiW.LiJ.PolsonJ.WangZ.SpeierW.ArnoldC. (2022). High resolution histopathology image generation and segmentation through adversarial training. Med. Image Anal.75, 102251. 10.1016/j.media.2021.102251
13
OktayO.SchlemperJ.FolgocL. L.LeeM. J.HeinrichM. P.MisawaK.et al (2018). Attention U-net: learning where to look for the pancreas. https://arxiv.org/abs/1804.03999.
14
OrlandoJ. I.FuH.Barbosa BredaJ.Van KeerK.BathulaD. R.Diaz-PintoA.et al (2020). REFUGE Challenge: a unified framework for evaluating automated methods for glaucoma assessment from fundus photographs. Med. Image Anal.59, 101570. 10.1016/j.media.2019.101570
15
PareS.KumarA.SinghG. K.BajajV. J. I. J. O. S.TechnologyT. O. E. E. (2019). Image segmentation using multilevel thresholding: a research review. Iran. J. Sci. Technol. Trans. Electr. Eng.44, 1–29. 10.1007/s40998-019-00251-1
16
PoggiG.RagoziniR. P. (1999). Image segmentation by tree-structured Markov random fields. IEEE Signal Process. Lett.6, 155–157. 10.1109/97.769356
17
RonnebergerO.FischerP.BroxT. (2015). “U-net: convolutional networks for biomedical image segmentation,” in Medical image computing and computer-assisted intervention – miccai 2015. Editors NavabN.HorneggerJ.WellsW. M.FrangiA. F. (Cham: Springer International Publishing), 234–241.
18
SeoH.HuangC.BassenneM.XiaoR.XingL. (2020). Modified U-net (mU-Net) with incorporation of object-dependent high level features for improved liver and liver-tumor segmentation in CT images. IEEE Trans. Med. Imaging39, 1316–1325. 10.1109/TMI.2019.2948320
19
SeoniS.JahmunahV.SalviM.BaruaP.MolinariF.AcharyaU. (2023). Application of uncertainty quantification to artificial intelligence in healthcare: a review of last decade (2013-2023). Comput. Biol. Med.165, 107441. 10.1016/j.compbiomed.2023.107441
20
ShahamatH.Saniee AbadehM. (2020). Brain MRI analysis using a deep learning based evolutionary approach. Neural Netw.126, 218–234. 10.1016/j.neunet.2020.03.017
21
ShiC.ZhangJ.ZhangX.ShenM.ChenH.WangL. (2022). A recurrent skip deep learning network for accurate image segmentation. Biomed. Signal Process. Control74, 103533. 10.1016/j.bspc.2022.103533
22
WangL.ChenK.WenH.ZhengQ.ChenY.PuJ.et al (2021a). Feasibility assessment of infectious keratitis depicted on slit-lamp and smartphone photographs using deep learning. Int. J. Med. Inf.155, 104583. 10.1016/j.ijmedinf.2021.104583
23
WangL.GuJ.ChenY.LiangY.ZhangW.PuJ.et al (2021b). Automated segmentation of the optic disc from fundus images using an asymmetric deep learning network. Pattern Recognit.112, 107810. 10.1016/j.patcog.2020.107810
24
WangL.LiuH.LuY.ChenH.ZhangJ.PuJ. (2019). A coarse-to-fine deep learning framework for optic disc segmentation in fundus images. Biomed. Signal Process. Control51, 82–89. 10.1016/j.bspc.2019.01.022
25
WangL.ShenM.ChangQ.ShiC.ChenY.ZhouY.et al (2021c). Automated delineation of corneal layers on OCT images using a boundary-guided CNN. Pattern Recognit.120, 108158. 10.1016/j.patcog.2021.108158
26
WangL.ShenM.ShiC.ZhouY.ChenY.PuJ.et al (2022). EE-Net: an edge-enhanced deep learning network for jointly identifying corneal micro-layers from optical coherence tomography. Biomed. Signal Process. Control71, 103213. 10.1016/j.bspc.2021.103213
27
WangY.YuX.WuC. (2022). An efficient hierarchical optic disc and cup segmentation network combined with multi-task learning and adversarial learning. J. Digit. Imaging35, 638–653. 10.1007/s10278-021-00579-3
28
XiangT.ZhangC.LiuD.SongY.HuangH.CaiW. (2020). “BiO-net: learning recurrent Bi-directional connections for encoder-decoder architecture,” in Medical image computing and computer assisted intervention – miccai 2020. Editors MartelA. L.AbolmaesumiP.StoyanovD.MateusD.ZuluagaM. A.ZhouS. K.et al (Cham: Springer International Publishing), 74–84.
29
ZhangJ.MeiC.LiZ.YingJ.ZhengQ.YiQ.et al (2023). Automated segmentation of optic disc and cup depicted on color fundus images using a distance-guided deep learning strategy. Biomed. Signal Process. Control86, 105163. 10.1016/j.bspc.2023.105163
30
ZhengQ.ZhangX.ZhangJ.BaiF.HuangS.PuJ.et al (2022). A texture-aware U-Net for identifying incomplete blinking from eye videography. Biomed. Signal Process. Control75, 103630. 10.1016/j.bspc.2022.103630
31
ZhouZ.Rahman SiddiqueeM. M.TajbakhshN.LiangJ. (2018). “UNet++: a nested U-net architecture for medical image segmentation,” in Deep learning in medical image analysis and multimodal learning for clinical decision support. Editors StoyanovD.TaylorZ.CarneiroG.Syeda-MahmoodT.MartelA.Maier-HeinL.et al (Cham: Springer International Publishing), 3–11.
32
ZhuQ.LuP. (2020). Stem cell transplantation for amyotrophic lateral sclerosis. Pattern Recognit. Lett.136, 71–97. 10.1007/978-981-15-4370-8_6
Summary
Keywords
deep learning, training strategy, image segmentation, fundus image, optic cup deep learning, optic cup
Citation
Yang X, Zheng Y, Mei C, Jiang G, Tian B and Wang L (2024) UGLS: an uncertainty guided deep learning strategy for accurate image segmentation. Front. Physiol. 15:1362386. doi: 10.3389/fphys.2024.1362386
Received
05 January 2024
Accepted
26 March 2024
Published
08 April 2024
Volume
15 - 2024
Edited by
Cristiana Corsi, University of Bologna, Italy
Reviewed by
Massimo Salvi, Polytechnic University of Turin, Italy
Jolita Bernatavičienė, Vilnius University, Lithuania
Updates
Copyright
© 2024 Yang, Zheng, Mei, Jiang, Tian and Wang.
This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) and the copyright owner(s) are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.
*Correspondence: Lei Wang, lei.wang@eye.ac.cn
† These authors have contributed equally to this work and share first authorship
Disclaimer
All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article or claim that may be made by its manufacturer is not guaranteed or endorsed by the publisher.