ORIGINAL RESEARCH article

Front. Physiol., 08 April 2024

Sec. Computational Physiology and Medicine

Volume 15 - 2024 | https://doi.org/10.3389/fphys.2024.1362386

UGLS: an uncertainty guided deep learning strategy for accurate image segmentation

  • 1. Wenzhou People’s Hospital, The Third Affiliated Hospital of Shanghai University, Wenzhou, China

  • 2. School of Ophthalmology and Optometry, Eye Hospital, Wenzhou Medical University, Wenzhou, China

Abstract

Accurate image segmentation plays a crucial role in computer vision and medical image analysis. In this study, we developed a novel uncertainty guided deep learning strategy (UGLS) to enhance the performance of an existing neural network (i.e., U-Net) in segmenting multiple objects of interest from images with varying modalities. In the developed UGLS, a boundary uncertainty map was introduced for each object based on its coarse segmentation (obtained by the U-Net) and then combined with input images for the fine segmentation of the objects. We validated the developed method by segmenting optic cup (OC) regions from color fundus images and left and right lung regions from Xray images. Experiments on public fundus and Xray image datasets showed that the developed method achieved a average Dice Score (DS) of 0.8791 and a sensitivity (SEN) of 0.8858 for the OC segmentation, and 0.9605, 0.9607, 0.9621, and 0.9668 for the left and right lung segmentation, respectively. Our method significantly improved the segmentation performance of the U-Net, making it comparable or superior to five sophisticated networks (i.e., AU-Net, BiO-Net, AS-Net, Swin-Unet, and TransUNet).

1 Introduction

Image segmentation is an important research direction of computer vision and medical image analysis, and widely used as a preprocessing step for various object detection and disease diagnosis (; ). It can divide an image into several disjoint regions by performing a pixel-level classification and largely simplify the assessment of morphological and positional characteristics of object regions (; ). To accurately segment images, a number of image segmentation algorithms have been developed for many different applications, such as threshold based methods (; ), active contour based methods (), and random field based methods (; ). Among these methods, deep learning based methods (; Wang Y. et al., 2022) have gained considerable popularity in the past decade because they can obtain remarkable segmentation performances comparable to manual annotations. Moreover, they are able to automatically extract and flexibly integrate different types of feature information by learning the intrinsic laws and representation levels of images to be segmented.

Despite promising performances, deep learning based methods are often faced with two key challenges in image segmentation (; ), one is how to obtain rich local information, the other is how to robustly extract high-level semantics. Given the large number of parameters in deep learning networks, the spatial resolution of images generally decreases with the increase of network depth in order to speed up the learning of feature information. This resolution decrease can bring about the loss of local information, but the increase of network depth is beneficial to the acquisition of global semantic and context information. To mitigate these two challenges, different deep learning networks (; ) have been constantly emerging to accurately segment images with varying modalities. Alom et al. (Alom et al., 2019) put forward the RU-Net and R2U-Net, respectively by adding different cyclic convolutional blocks to the U-Net for feature detection and accumulation. Seo et al. () proposed a mU-Net model by introducing learnable deconvolution network structures into the U-Net to improve its learning ability at different resolutions and image segmentation performance. Huang et al. () proposed a U-Net 3+ model that combines high-level semantics with low-level semantics using full-scale jump concatenation to overcome the drawbacks of the U-Net and U-Net++ (). Cao et al. () and Chen et al. () proposed different transformer based networks (i.e., Swin-Unet and TransUNet), respectively for accurate image segmentation. These network models demonstrated reasonable segmentation accuracy as compared to the U-Net, but their network structures were often more complex. This may not be conducive to network construction and training as well as image segmentation.

To avoid the design of complex network structures, we develop an uncertainty guided deep learning strategy (UGLS) in this study based on a existing network (i.e., U-Net) for accurate image segmentation. We first train the U-Net to obtain a coarse segmentation result and then use morphological operations and Gaussian filters to identify a potential boundary region for each target object based on the obtained result. The boundary region has a unique intensity distribution to indicate the probability of each pixel belonging to object boundaries and is termed as the boundary uncertainty map (BUM) of the objects. With boundary uncertainty maps and original input images, we retrain the U-Net for the fine segmentation of target objects and can obtain a better performance, as compared to its coarse segmentation performance.

2 Methods

2.1 Scheme overview

Figure 1 shows the entire workflow of the developed deep learning strategy (UGLS) based on a available network (i.e., U-Net) for image segmentation purposes. The UGLS consists of three key steps, namely, the coarse segmentation of target objects, generation of boundary uncertainty maps for each object, and object fine segmentation. The coarse segmentation is used to detect potential object regions and exclude irrelevant background far away from the detected regions. With the coarse segmentation, we can identify the regions where object boundaries are likely to appear and then generate boundary uncertainty maps for these objects, which can largely enhance the information about object boundaries and facilitate the boundary detection. We integrate these uncertainty maps and original input images and feed them into the given network for a more fine segmentation. After performing these three steps, the network can obtain a significantly improved segmentation performance.

FIGURE 1

2.2 Object coarse segmentation

We first trained the U-Net based on the given images and their manual annotations leveraging a plain network training scheme to obtain a relatively coarse segmentation result for desirable objects. This train procedure can be given by:where and indicate the input image and its corresponding prediction map, respectively, denotes the U-Net with the network parameter . The prediction map was relatively coarse as compared with manual annotations of objects because the U-Net has a simple network structure and thereby limited potential to handle images with varying qualities.

2.3 Boundary uncertainty map

The obtained coarse segmentation results were often different from manual annotations of objects in certain image regions, especially object boundary regions, but they can provide some important position information for desirable objects. To effectively use the position information, we processed the coarse segmentation results leveraging morphological dilation and erosion operations (), leading to two different object regions. Based on the two object regions, we can identify a potential boundary region (PBR) and a background excluded image (BEI) for each target object, which were separately given bywhere and are the morphological dilation and erosion operations, respectively, is a circular structuring element with a radius of . The PBR is a binary image and marks the region where object boundaries are most likely to appear, while the BEI merely retains the original image information located in the PBR and can reduce the impact of redundant background in image segmentation, as shown in Figure 2. To take fully advantage of edge position information in coarse segmentation results, we smoothed the PBR using a Gaussian filter with a rectangle window of and a standard deviation of to generate a boundary uncertainty map. The pixels in the uncertainty map took larger values when they were close to the center of the PBR and reduced ones when far away from this center. Moreover, A larger value generally means a higher probability that a pixel in the uncertainty map belongs to object boundaries. The unique intensity distribution made the boundary uncertainty map able to provide more relevant position information about object boundaries, as compared to the PBR.

FIGURE 2

2.4 Object fine segmentation

After obtaining the boundary uncertainty map and background excluded image, we concatenated these two types of images and fed them into the segmentation network. Since the concatenated images were different from the original images and contained very little background information, the segmentation network can easily detect object boundaries and thereby extract the whole object regions accurately using a simple experiment configuration. Specifically, we implemented the fine segmentation of desirable objects using the same configuration as their coarse segmentation (e.g., the cost function, optimizer and batch size).

2.5 Experiment datasets

To validate the developed learning strategy, we performed a series of segmentation experiments on two public dataset, as shown in Figure 3. The first dataset was from the Retinal Fundus Glaucoma Challenge (REFUGE) () and contained 1,200 retinal fundus images acquired by two different cameras, together with manual annotations for the optic disc (OD) and cup (OC) regions. These images and their annotations were evenly split into three subsets for training (n = 400), validation (n = 400) and testing (n = 400) purposes, respectively, in the REFUGE challenge, which were also used in this study for segmentation purposes. We normalized these images to reduce the influence of light exposure and cameras and then extracted local disc patches using the dimensions that approximated three times the radius of the OD regions (). The extracted patches were then resized to 256 × 256 pixels and fed into the U-Net for network training.

FIGURE 3

The second dataset was from a tuberculosis screening program in Montgomery County (TSMC) () and contained 138 chest Xray images acquired using a Eureka stationary Xray machine. Among these Xray images, 80 were normal and 58 were abnormal with manifestations of tuberculosis. All images were de-identified and had a dimension of either 4,020 × 4,892 or 4,892 × 4,020 pixels. The left and right lungs depicted on these Xray images were manually annotated by a radiologist. We also split these Xray images equally into three disjoint subsets for network training (n = 46), validation (n = 46) and testing (n = 46), and resized them to the same dimension of 256 × 256 pixels.

2.6 Performance evaluation

We assessed the performance of the UGLS based on the U-Net (short for the developed method, https://github.com/wmuLei/ODsegmentation) on a 64-bit Windows 10 PC with 2.20 GHz 2.19 GHz Intel(R) Xeon(R) Gold 5120 CPU, 64 GB RAM and NVIDIA GeForce GTX 2080Ti by segmenting 1) the OC region from color fundus images and 2) the left and right lungs from the Xray images, where the was set to 25 and 35, respectively for these two datasets. We used the Dice Score (DS) () as the cost function to assess the similarity between the segmentation results and their corresponding manual annotations for each object:where denotes the DS for object , and is the total number of objects of interest. and are the output probabilities of a specific input image obtained by the U-Net and manual annotation, respectively for pixel and object , denotes the entire image domain. We used the RMSprop optimizer to maximize the cost function and set its initial learning rate to 0.001, along with a batch size of eight and an epoch number of 100. To reduce the network training time, we halted the entire training procedure when the performance of the U-Net did not increase for 20 consecutive epochs. In addition, we randomly augmented input images during network training using some transformations, such as horizontal/vertical flip, scaling from 0.9 to 1.1, translation by −10 to 10 percent per axis, rotation from −180 to 180 in degree, and shearing from −5 to 5 in degree. After training, we binarized the prediction map of the U-Net using a given threshold of 0.5 to obtain desirable output results.

With these output results, we evaluated our developed method using the DS, Matthew’s correlation coefficient (MCC) (), sensitivity (SEN) (), and Hausdorff distance (HSD, in pixel).where , , and denote the true positive, false positive, true negative and false negative, respectively. is the directed HSD from point set to . The larger the DS, MCC and SEN are and the smaller the HSD is, the better the segmentation performance of the network is. To show the advantage of the UGLS, we compared the developed method with the Attention U-Net (AU-Net) (), BiO-Net (), asymmetric U-Net (AS-Net) (), Swin-Unet (in tiny scale version), and TransUNet. Among these networks, U-Net and its variants (i.e., AU-Net, BiO-Net, AS-Net) shared the similar network architecture (e.g., the number of convolution filters increased from 32 to 1,024) and were trained from scratch based on a given dimension of 256 × 256 pixels and a learning rate of 0.001, while Swin-Unet and TransUNet were trained from initial ImageNet weights based on a dimension of 224 × 224 pixels and a learning rate of 0.01. All these networks were trained six times (by randomly arranging three different subsets for network training, validation and testing, respectively) using the same configurations (except for image dimension and learning rate) for each dataset. The paired t-test was used to evaluate the differences among the involved networks on the DS metric. A p-value less than 0.05 was considered statistically significant ().

3 Results

3.1 Object coarse segmentation

Tables 1 and 2 summarized six coarse segmentation results of the U-Net with the developed UGLS strategy in extracting the OC from retinal fundus images and the left and right lungs from Xray images, respectively. As demonstrated by the results, the U-Net achieved a relatively low performance in segmenting the OC depicted on fundus images (due to the high similarity between the OD and OC regions), with a average DS, MCC, SEN and HSD of 0.8642, 0.8585, 0.8674 and 2.6420, respectively. In contrast, it obtained a better accuracy for the left and right lungs (with the average DS of 0.9408 and 0.9477, respectively) and can compete with their manual annotations.

TABLE 1

ObjectResultDSMCCSENHSD
Mean ± SDMean ± SDMean ± SDMean ± SD
OCSeg10.8687 ± 0.06360.8629 ± 0.06270.8654 ± 0.10762.6266 ± 0.6213
Seg20.8439 ± 0.07670.8380 ± 0.07490.8592 ± 0.13632.7302 ± 0.6845
Seg30.8784 ± 0.05790.8720 ± 0.05630.9344 ± 0.08562.5711 ± 0.5217
Seg40.8646 ± 0.07690.8586 ± 0.07570.8508 ± 0.12392.5952 ± 0.6782
Seg50.8399 ± 0.08860.8358 ± 0.08470.7980 ± 0.14132.7719 ± 0.7082
Seg60.8898 ± 0.05950.8839 ± 0.05900.8969 ± 0.08612.4492 ± 0.5737
Overall0.8642 ± 0.07360.8585 ± 0.07170.8674 ± 0.12302.6240 ± 0.6435

Results of our proposed method for the coarse segmentation of the OC regions based on six experiments (i.e., Seg1-6) in terms of the mean and standard deviation (SD) of DS, MCC, SEN and HSD (in pixel).

TABLE 2

ObjectResultDSMCCSENHSD
Mean ± SDMean ± SDMean ± SDMean ± SD
LLSeg10.9763 ± 0.01970.9733 ± 0.02160.9672 ± 0.03693.9530 ± 1.5336
Seg20.9551 ± 0.04010.9512 ± 0.03990.9258 ± 0.06754.3733 ± 1.0238
Seg30.9019 ± 0.14290.8938 ± 0.15200.9258 ± 0.08995.3338 ± 2.3926
Seg40.9546 ± 0.09240.9521 ± 0.09060.9569 ± 0.01684.2231 ± 2.0033
Seg50.9540 ± 0.08700.9518 ± 0.07980.9838 ± 0.01354.1766 ± 1.8032
Seg60.9028 ± 0.08460.8977 ± 0.07890.9899 ± 0.01975.9666 ± 1.5560
Overall0.9408 ± 0.09170.9367 ± 0.09250.9582 ± 0.05584.6711 ± 1.9139
ObjectResultDSMCCSENHSD
Mean ± SDMean ± SDMean ± SDMean ± SD
RLSeg10.9656 ± 0.04180.9616 ± 0.04330.9826 ± 0.05004.6004 ± 1.6841
Seg20.9519 ± 0.04640.9471 ± 0.04640.9242 ± 0.07875.0881 ± 1.1806
Seg30.8962 ± 0.12680.8807 ± 0.15190.8991 ± 0.05676.6095 ± 2.9507
Seg40.9552 ± 0.08930.9512 ± 0.09200.9537 ± 0.01815.0575 ± 2.2573
Seg50.9644 ± 0.08990.9613 ± 0.08960.9784 ± 0.04074.6336 ± 1.5349
Seg60.9531 ± 0.07070.9496 ± 0.06870.9655 ± 0.05434.7648 ± 1.7171
Overall0.9477 ± 0.08600.9419 ± 0.09400.9506 ± 0.06085.1257 ± 2.0893

The performance of the developed method for segmenting the left and right lungs (LL and RL) from Xray images.

3.2 Object fine segmentation

Tables 3 and 4 demonstrated the fine segmentation results of the U-Net with the developed UGLS strategy for three different objects depicted on fundus and Xray images, respectively. The U-Net achieved the average DS and SEN of 0.8791 and 0.8858 for the OC region, and 0.9605, 0.9607, 0.9621, and 0.9668 for the left and righ lungs, respectively. As compared with its coarse segmentation results, the U-Net obtained a significantly better overall performance for six different experiments on two types of images with varying modalities (p < 0.01). Specifically, the U-Net had better performances for five fine segmentation experiments for the OC, as compared to its coarse results, as shown in Table 3. Similarly, its performances were also increased in large increments for each experiment in the fine segmentation of the left and right lungs.

TABLE 3

ObjectResultDSMCCSENHSD
Mean ± SDMean ± SDMean ± SDMean ± SD
OCSeg10.8696 ± 0.06470.8638 ± 0.06330.8804 ± 0.10362.5887 ± 0.6049
Seg20.8442 ± 0.07550.8384 ± 0.07350.8764 ± 0.13062.7324 ± 0.6512
Seg30.8691 ± 0.06610.8623 ± 0.06610.8835 ± 0.11472.5945 ± 0.6272
Seg40.8982 ± 0.05470.8938 ± 0.05210.9141 ± 0.10272.2824 ± 0.4505
Seg50.9033 ± 0.05270.8993 ± 0.05090.8573 ± 0.09222.2555 ± 0.5148
Seg60.8904 ± 0.06120.8843 ± 0.06080.9033 ± 0.08212.4461 ± 0.5930
Overall0.8791 ± 0.06620.8737 ± 0.06510.8858 ± 0.10712.4833 ± 0.6031

Fine segmentation results of the developed method for the OC regions in terms of the DS, MCC, SEN and HSD (in pixel) metrics.

TABLE 4

ObjectResultDSMCCSENHSD
Mean ± SDMean ± SDMean ± SDMean ± SD
LLSeg10.9764 ± 0.02060.9735 ± 0.02240.9709 ± 0.03493.9756 ± 1.4742
Seg20.9636 ± 0.04050.9601 ± 0.04150.9552 ± 0.06124.0821 ± 0.9145
Seg30.9291 ± 0.13250.9229 ± 0.14030.9148 ± 0.15114.6107 ± 1.8065
Seg40.9619 ± 0.09400.9585 ± 0.10080.9675 ± 0.10073.6484 ± 1.5086
Seg50.9680 ± 0.04120.9650 ± 0.04160.9794 ± 0.01823.9206 ± 1.2240
Seg60.9639 ± 0.05650.9615 ± 0.05570.9762 ± 0.03414.1280 ± 1.7205
Overall0.9605 ± 0.07600.9569 ± 0.08000.9607 ± 0.08404.0609 ± 1.5006
ObjectResultDSMCCSENHSD
Mean ± SDMean ± SDMean ± SDMean ± SD
RLSeg10.9661 ± 0.06250.9630 ± 0.06160.9749 ± 0.05124.4255 ± 1.6023
Seg20.9657 ± 0.03560.9618 ± 0.03650.9541 ± 0.06364.7426 ± 1.1453
Seg30.9541 ± 0.05760.9480 ± 0.06360.9473 ± 0.06965.1637 ± 1.5636
Seg40.9664 ± 0.07360.9625 ± 0.08090.9738 ± 0.03874.4790 ± 1.5192
Seg50.9685 ± 0.06320.9653 ± 0.06540.9773 ± 0.04074.4856 ± 1.4704
Seg60.9517 ± 0.08850.9491 ± 0.08500.9732 ± 0.04934.7916 ± 1.8466
Overall0.9621 ± 0.06580.9583 ± 0.06770.9668 ± 0.05464.6813 ± 1.5598

Fine segmentation results of the developed method for segmenting the left and right lungs (LL and RL) from the Xray images in terms of the DS, MCC, SEN and HSD (in pixel) metrics.

3.3 Performance comparison

Table 5 summarized the segmentation results of the involved networks (i.e., the U-Net, AU-Net, BiO-Net, AS-Net, Swin-Unet, and TransUNet) in extracting three different objects from fund and Xray images, respectively. As demonstrated by these results, the developed UGLS strategy can significantly improve the performance of the U-Net (p < 0.01) by merely leveraging the its coarse segmentation results in a reasonable way, instead of changing its network structure. Specifically, the average DS of the U-Net increased from 0.8792 to 0.8945 for three different object regions depicted on fundus and Xray images after using our developed deep learning strategy. This strategy made our developed method superior or comparable to the AU-Net (0.8803, p < 0.001), BiO-Net (0.8843, p < 0.005), AS-Net (0.8859, p < 0.005), Swin-Unet (0.8811, p < 0.001), and TransUNet (0.8900, p < 0.05) with all the p-values less than 0.05 for the two segmentation tasks. Figures 4 and 5 showed the performance differences among the involved networks on several fundus and Xray images.

TABLE 5

ObjectMethodDSMCCSENHSD
Mean ± SDMean ± SDMean ± SDMean ± SD
OCU-Net0.8642 ± 0.07360.8585 ± 0.07170.8674 ± 0.12302.6240 ± 0.6435
AU-Net0.8619 ± 0.07800.8569 ± 0.07460.8667 ± 0.12932.5896 ± 0.6180
BiO-Net0.8663 ± 0.07210.8605 ± 0.07040.8801 ± 0.11812.5960 ± 0.6374
AS-Net0.8676 ± 0.07250.8620 ± 0.07010.8757 ± 0.11862.6119 ± 0.6281
Swin-Unet0.8647 ± 0.07060.8582 ± 0.06890.8799 ± 0.11452.7134 ± 0.5767
TransUNet0.8737 ± 0.06430.8679 ± 0.06200.8894 ± 0.10712.4974 ± 0.5416
Proposed0.8791 ± 0.06620.8737 ± 0.06510.8858 ± 0.10712.4833 ± 0.6031
ObjectMethodDSMCCSENHSD
Mean ± SDMean ± SDMean ± SDMean ± SD
LLU-Net0.9408 ± 0.09170.9367 ± 0.09250.9582 ± 0.05584.6711 ± 1.9139
AU-Net0.9607 ± 0.07190.9572 ± 0.07580.9649 ± 0.05024.1158 ± 1.8849
BiO-Net0.9614 ± 0.06810.9582 ± 0.06730.9637 ± 0.05434.1554 ± 1.6769
AS-Net0.9649 ± 0.04930.9609 ± 0.05060.9623 ± 0.06334.7912 ± 1.6847
Swin-Unet0.9502 ± 0.03160.9446 ± 0.03170.9530 ± 0.04795.0613 ± 0.7958
TransUNet0.9617 ± 0.03570.9574 ± 0.03770.9604 ± 0.04764.1025 ± 1.0577
Proposed0.9605 ± 0.07600.9569 ± 0.08000.9607 ± 0.08404.0609 ± 1.5006
ObjectMethodDSMCCSENHSD
Mean ± SDMean ± SDMean ± SDMean ± SD
RLU-Net0.9477 ± 0.08600.9419 ± 0.09400.9506 ± 0.06085.1257 ± 2.0893
AU-Net0.9595 ± 0.07360.9545 ± 0.08040.9674 ± 0.04444.7496 ± 1.8938
BiO-Net0.9637 ± 0.06510.9597 ± 0.06610.9693 ± 0.04834.7352 ± 1.6805
AS-Net0.9663 ± 0.05160.9628 ± 0.05520.9636 ± 0.05374.1806 ± 1.6053
Swin-Unet0.9549 ± 0.02800.9488 ± 0.02810.9562 ± 0.04575.6140 ± 0.8730
TransUNet0.9602 ± 0.04070.9552 ± 0.04040.9606 ± 0.04214.8674 ± 1.1996
Proposed0.9621 ± 0.06580.9583 ± 0.06770.9668 ± 0.05464.6813 ± 1.5598

Performance differences among the involved networks in segmenting the OC, left and right lungs depicted on fundus and Xray images, respectively.

FIGURE 4

FIGURE 5

3.4 Effect of the BUM

Table 6 showed the results of the developed method in extracting the left and right lungs from Xray images using boundary uncertainty maps in three different ways. As demonstrated by the results, our developed method obtained the lowest segmentation performance, with the average DS of 0.9437 when merely trained on boundary uncertainty maps, but it had increased performance when combining the uncertainty maps with the original images or their background excluded version for network training (with the average DS of 0.9611 and 0.9613). Moreover, the background excluded images can better improve the performance of our developed method since they reduced the impact of irrelevant background information away from desirable objects.

TABLE 6

ObjectMethodDSMCCSENHSD
Mean ± SDMean ± SDMean ± SDMean ± SD
LLBUM0.9347 ± 0.12270.9297 ± 0.12450.9356 ± 0.12844.3859 ± 1.5356
ORI + BUM0.9587 ± 0.09200.9553 ± 0.09560.9603 ± 0.09364.0907 ± 1.6055
BEI + BUM0.9605 ± 0.07600.9569 ± 0.08000.9607 ± 0.08404.0609 ± 1.5006
ObjectMethodDSMCCSENHSD
Mean ± SDMean ± SDMean ± SDMean ± SD
RLBUM0.9527 ± 0.07430.9476 ± 0.07710.9517 ± 0.08195.0905 ± 1.5155
ORI + BUM0.9634 ± 0.06830.9597 ± 0.07080.9651 ± 0.06364.6374 ± 1.5137
BEI + BUM0.9621 ± 0.06580.9583 ± 0.06770.9668 ± 0.05464.6813 ± 1.5598

The results of the developed method trained on the boundary uncertainty map (BUM) or its combination with the original image (ORI) or its background excluded version (BEI) for the left and right lung segmentation.

3.5 Effect of parameter

Table 7 summarized the impact of the parameter on the performance of the developed method in segmenting three different objects from fundus and Xray images. The developed method achieved the best overall performance when this parameter was set to 25 in the OC segmentation and 35 in the left and right lung segmentation, respectively, for the morphological operations and Gaussian filter. These two parameter values ensured a good balance between object information and irrelevant background for our developed method, making it able to accurately detect object boundaries. Table 8 showed the performance of the developed method when using different values for the parameters in the morphological operations and Gaussian filter. From the table, our developed method obtained a superior overall performance when the morphological operations and Gaussian filter shared the same value for each image dataset, which can effectively highlight the center regions of boundary uncertainty maps, as shown in Figure 6.

TABLE 7

ObjectDSMCCSENHSD
Mean ± SDMean ± SDMean ± SDMean ± SD
OC150.8678 ± 0.06960.8626 ± 0.06800.8622 ± 0.12202.5426 ± 0.6246
250.8791 ± 0.06620.8737 ± 0.06510.8858 ± 0.10712.4833 ± 0.6031
350.8735 ± 0.06550.8679 ± 0.06390.8848 ± 0.11122.5257 ± 0.5995
ObjectDSMCCSENHSD
Mean ± SDMean ± SDMean ± SDMean ± SD
LL150.9512 ± 0.11170.9474 ± 0.11730.9519 ± 0.11144.1688 ± 1.6956
250.9543 ± 0.09680.9512 ± 0.09640.9566 ± 0.09874.1234 ± 1.6199
350.9605 ± 0.07600.9569 ± 0.08000.9607 ± 0.08404.0609 ± 1.5006
450.9566 ± 0.09960.9537 ± 0.09930.9582 ± 0.10174.1346 ± 1.5093
550.9554 ± 0.10140.9523 ± 0.10340.9584 ± 0.10234.0513 ± 1.5453
ObjectDSMCCSENHSD
Mean ± SDMean ± SDMean ± SDMean ± SD
RL150.9614 ± 0.06490.9578 ± 0.06430.9627 ± 0.06164.7130 ± 1.5723
250.9617 ± 0.07130.9578 ± 0.07250.9646 ± 0.05294.6753 ± 1.6561
350.9621 ± 0.06580.9583 ± 0.06770.9668 ± 0.05464.6813 ± 1.5598
450.9625 ± 0.06830.9589 ± 0.06920.9662 ± 0.05584.6934 ± 1.5860
550.9636 ± 0.06460.9602 ± 0.06410.9662 ± 0.05594.6227 ± 1.5216

The results of the developed method on fundus and Xray images by setting different values for parameters .

TABLE 8

Object(, )DSMCCSENHSD
Mean ± SDMean ± SDMean ± SDMean ± SD
OC(25, 15)0.8669 ± 0.06510.8611 ± 0.06390.8748 ± 0.10642.5936 ± 0.6169
(25, 25)0.8696 ± 0.06470.8638 ± 0.06330.8804 ± 0.10362.5887 ± 0.6049
(25, 35)0.8675 ± 0.06550.8619 ± 0.06420.8708 ± 0.10832.5815 ± 0.6131
Object(, )DSMCCSENHSD
Mean ± SDMean ± SDMean ± SDMean ± SD
LL(35, 15)0.9695 ± 0.03330.9663 ± 0.03540.9679 ± 0.04594.2534 ± 1.7802
(35, 25)0.9733 ± 0.02590.9703 ± 0.02750.9702 ± 0.03784.1451 ± 1.7235
(35, 35)0.9764 ± 0.02060.9735 ± 0.02240.9709 ± 0.03493.9756 ± 1.4742
(35, 45)0.9750 ± 0.02620.9722 ± 0.02770.9752 ± 0.03573.9758 ± 1.5872
Object(, )DSMCCSENHSD
Mean ± SDMean ± SDMean ± SDMean ± SD
RL(35, 15)0.9681 ± 0.04590.9646 ± 0.04650.9754 ± 0.05084.4583 ± 1.5158
(35, 25)0.9698 ± 0.04400.9666 ± 0.04460.9752 ± 0.05334.3794 ± 1.5588
(35, 35)0.9661 ± 0.06250.9630 ± 0.06160.9749 ± 0.05124.4255 ± 1.6023
(35, 45)0.9639 ± 0.06270.9610 ± 0.06080.9755 ± 0.05274.5328 ± 1.5872

The results of the developed method for the first experiment on fundus and Xray images using different values for parameter in morphological operations and Gaussian filter (short for and , respectively).

FIGURE 6

4 Discussion

In this paper, we developed a novel network training strategy (termed UGLS) for accurate image segmentation and assessed its effectiveness based on an existing network (i.e., the U-Net) by extracting three different objects depicted (i.e., the OC, left and right lungs) on fundus and Xray images. In the developed method, the U-Net was first trained using the traditional training strategy on the original images and their manual annotations for the coarse-grained segmentation of desirable objects. The segmentation results were then proposed to locate a potential boundary region for each object, which was combined with the original images for the fine segmentation of the objects. We validated the developed method on two public datasets (i.e., REFUGE and TSMC) and compared it with five available networks (i.e., the AU-Net, BiO-Net, AS-Net, Swin-Unet and TransUNet) under the similar experiment configurations. Extensive experiments showed that the developed method can largely improve the segmentation performance of the U-Net and was comparable or superior to the AU-Net, BiO-Net, AS-Net, Swin-Unet and TransUNet, all of which had much more complex network structures than the U-Net.

The developed method achieved promising overall performance in segmenting multiple different objects, as compared to three existing networks. This may be attributed to the following reasons: First, the coarse segmentation of the objects was able to detect various types of image features and provide some important location information for each object and its boundaries. Second, the introduction of boundary uncertainty maps made the potential boundary region have a unique intensity distribution. This distribution largely facilitated the detection of object boundaries and enhanced the sensitivity and accuracy of the U-Net in segmenting objects of interest. Third, the use of background excluded images can not only ensure a reasonable balance between object information and its surrounding background, but also ensure that the U-Net performs the learning of various features in the specified region, thereby leading to a increased segmentation performance and a reduced influence of undesirable background. Due to these reasons, the developed method can significantly improve the segmentation performance of a relatively simple network (i.e., the U-Net) and make it comparable or superior to several existing sophisticated networks.

We further assessed the influence of boundary uncertainty maps and the parameter on the performance of the developed method. Segmentation results in Tables 68 showed that (Eq. 1) the developed method achieved better segmentation performance when trained on the combination of boundary uncertainty maps and the background excluded images, as compared to the counterparts trained merely on boundary uncertainty maps or the original images. This may be due to the fact that there are no enough texture information relative to targe objects and their boundaries in boundary uncertainty maps, but too much background information in the original images, both of which can reduce the learning potential of the U-Net and deteriorate its segmentation performance. 2) The developed method obtained relatively high segmentation accuracy when the parameter was assigned to 25 for the OC segmentation and 35 for the left and right lung segmentation. This parameter controlled the amount of information about desirable objects and their surrounding background in the boundary uncertainty maps. A proper value for the parameter can ensure a good balance between the two types of image information and significantly improve the fine segmentation performance of our developed method. If the parameter value was set too small or large, our developed method would have a final result that was very close to its coarse segmentation results or contained lots of undesirable background. 3) The parameter was used simultaneously in morphological operations and Gaussian filter since it can ensure that pixels in the center region of boundary uncertainty map have more high contrast or intensity, as compared to the counterparts in other regions. 4) Boundary uncertainty maps can be generated using different strategies, but their corresponding segmentation performance was very similar (i.e., 0.8791 vs. 0.8721 for the OC segmentation), based on our previous study ().

5 Conclusion

We developed a uncertainty guided deep learning strategy (UGLS) to improve the performance of existing segmentation neural networks and validated it based on the classical U-Net by segmenting the OC from color fundus images and the left and right lungs from Xray images. The novelty of our developed method lies in the introduction of boundary uncertainty maps and their integration with the input images for accurate image segmentation. Extensive experiments on public fundus and Xray image datasets demonstrated that the developed method had the potential to effectively extract the OC from fundus images and the left and right lungs from Xray images, largely improved the performance of the U-Net, and can compete with several sophisticated networks (i.e., the AU-Net, BiO-Net, AS-Net, Swin-Net, and TransUNet).

Statements

Data availability statement

Publicly available datasets were analyzed in this study. This data can be found here: Retinal Fundus Glaucoma Challenge (REFUGE), a tuberculosis screening program in Montgomery County.

Author contributions

XY: Methodology, Software, Writing–original draft. YZ: Resources, Validation, Writing–original draft. CM: Formal Analysis, Investigation, Visualization, Writing–review and editing. GJ: Methodology, Software, Visualization, Writing–original draft. BT: Investigation, Validation, Visualization, Writing–original draft. LW: Methodology, Supervision, Writing–review and editing.

Funding

The author(s) declare that financial support was received for the research, authorship, and/or publication of this article. This work is supported in part by Natural Science Foundation of Zhejiang Province (Grant No. LGF22H090022 and LTGY23H090014), Health Commission of Zhejiang Province (Grant No. 2022KY1206), and Wenzhou Municipal Science and Technology Bureau (Grant No. ZY2022024).

Conflict of interest

The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Publisher’s note

All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.

References

  • 1

    AlomM. Z.YakopcicC.HasanM.TahaT. M.AsariV. K. (2019). Recurrent residual U-Net for medical image segmentation. J. Med. Imaging (Bellingham)6, 014006. 10.1117/1.Jmi.6.1.014006

  • 2

    CaoH.WangY.ChenJ.JiangD.ZhangX.TianQ.et al (2022). Swin-unet: unet-like pure transformer for medical image segmentation. Eur. Conf. Comput. Vis. (ECCV).

  • 3

    ChenJ.LuY.YuQ.LuoX.AdeliE.WangY.et alTransUNet: transformers make strong encoders for medical image segmentation,” in 2021 IEEE/CVF Conference on Computer Vision and Pattern, Nashville, TN, USA, June 2021.

  • 4

    FangX.ShenY.ZhengB.ZhuS.WuM.Optic disc segmentation based on phase-fusion PSPNet,” in Proceedings of the 2nd International Symposium on Artificial Intelligence for Medicine Sciences, Beijing, China, October 2021 (Association for Computing Machinery).

  • 5

    GawlikowskiJ.TassiC.AliM.LeeJ.HumtM.FengJ.et al (2023). A survey of uncertainty in deep neural networks. Artif. Intell. Rev.56, 15131589. 10.1007/s10462-023-10562-9

  • 6

    HanD. J. J. O. I.Graphics (2006). A survey on image segmentation using active contour and level set method. Neurocomputing452. 10.1016/j.neucom.2020.07.141

  • 7

    HossainN. I.RezaS.Blood vessel detection from fundus image using Markov random field based image segmentation,” in 2017 4th International Conference on Advances in Electrical Engineering ICAEE, Dhaka, Bangladesh, 28-30 September 2017, 123127.

  • 8

    HuangH.LinL.TongR.HuH.ZhangQ.IwamotoY.et alUNet 3+: a full-scale connected UNet for medical image segmentation,” in ICASSP 2020 - 2020 IEEE International Conference on Acoustics, Speech and Signal Processing ICASSP, Barcelona, Spain, 4-8 May 2020, 10551059.

  • 9

    JaegerS.CandemirS.AntaniS.WángY. X.LuP. X.ThomaG. (2014). Two public chest X-ray datasets for computer-aided screening of pulmonary diseases. Quant. Imaging Med. Surg.4, 475477. 10.3978/j.issn.2223-4292.2014.11.20

  • 10

    JunT. J.KweonJ.KimY. H.KimD. (2020). T-Net: nested encoder-decoder architecture for the main vessel segmentation in coronary angiography. Neural Netw.128, 216233. 10.1016/j.neunet.2020.05.002

  • 11

    KhenedM.VargheseA.KrishnamurthiG. J. M. I. A. (2018). Fully convolutional multi‐scale residual DenseNets for cardiac segmentation and automated cardiac diagnosis using ensemble of classifiers. Med. Image Anal.51, 2145. 10.1016/j.media.2018.10.004

  • 12

    LiW.LiJ.PolsonJ.WangZ.SpeierW.ArnoldC. (2022). High resolution histopathology image generation and segmentation through adversarial training. Med. Image Anal.75, 102251. 10.1016/j.media.2021.102251

  • 13

    OktayO.SchlemperJ.FolgocL. L.LeeM. J.HeinrichM. P.MisawaK.et al (2018). Attention U-net: learning where to look for the pancreas. https://arxiv.org/abs/1804.03999.

  • 14

    OrlandoJ. I.FuH.Barbosa BredaJ.Van KeerK.BathulaD. R.Diaz-PintoA.et al (2020). REFUGE Challenge: a unified framework for evaluating automated methods for glaucoma assessment from fundus photographs. Med. Image Anal.59, 101570. 10.1016/j.media.2019.101570

  • 15

    PareS.KumarA.SinghG. K.BajajV. J. I. J. O. S.TechnologyT. O. E. E. (2019). Image segmentation using multilevel thresholding: a research review. Iran. J. Sci. Technol. Trans. Electr. Eng.44, 129. 10.1007/s40998-019-00251-1

  • 16

    PoggiG.RagoziniR. P. (1999). Image segmentation by tree-structured Markov random fields. IEEE Signal Process. Lett.6, 155157. 10.1109/97.769356

  • 17

    RonnebergerO.FischerP.BroxT. (2015). “U-net: convolutional networks for biomedical image segmentation,” in Medical image computing and computer-assisted intervention – miccai 2015. Editors NavabN.HorneggerJ.WellsW. M.FrangiA. F. (Cham: Springer International Publishing), 234241.

  • 18

    SeoH.HuangC.BassenneM.XiaoR.XingL. (2020). Modified U-net (mU-Net) with incorporation of object-dependent high level features for improved liver and liver-tumor segmentation in CT images. IEEE Trans. Med. Imaging39, 13161325. 10.1109/TMI.2019.2948320

  • 19

    SeoniS.JahmunahV.SalviM.BaruaP.MolinariF.AcharyaU. (2023). Application of uncertainty quantification to artificial intelligence in healthcare: a review of last decade (2013-2023). Comput. Biol. Med.165, 107441. 10.1016/j.compbiomed.2023.107441

  • 20

    ShahamatH.Saniee AbadehM. (2020). Brain MRI analysis using a deep learning based evolutionary approach. Neural Netw.126, 218234. 10.1016/j.neunet.2020.03.017

  • 21

    ShiC.ZhangJ.ZhangX.ShenM.ChenH.WangL. (2022). A recurrent skip deep learning network for accurate image segmentation. Biomed. Signal Process. Control74, 103533. 10.1016/j.bspc.2022.103533

  • 22

    WangL.ChenK.WenH.ZhengQ.ChenY.PuJ.et al (2021a). Feasibility assessment of infectious keratitis depicted on slit-lamp and smartphone photographs using deep learning. Int. J. Med. Inf.155, 104583. 10.1016/j.ijmedinf.2021.104583

  • 23

    WangL.GuJ.ChenY.LiangY.ZhangW.PuJ.et al (2021b). Automated segmentation of the optic disc from fundus images using an asymmetric deep learning network. Pattern Recognit.112, 107810. 10.1016/j.patcog.2020.107810

  • 24

    WangL.LiuH.LuY.ChenH.ZhangJ.PuJ. (2019). A coarse-to-fine deep learning framework for optic disc segmentation in fundus images. Biomed. Signal Process. Control51, 8289. 10.1016/j.bspc.2019.01.022

  • 25

    WangL.ShenM.ChangQ.ShiC.ChenY.ZhouY.et al (2021c). Automated delineation of corneal layers on OCT images using a boundary-guided CNN. Pattern Recognit.120, 108158. 10.1016/j.patcog.2021.108158

  • 26

    WangL.ShenM.ShiC.ZhouY.ChenY.PuJ.et al (2022). EE-Net: an edge-enhanced deep learning network for jointly identifying corneal micro-layers from optical coherence tomography. Biomed. Signal Process. Control71, 103213. 10.1016/j.bspc.2021.103213

  • 27

    WangY.YuX.WuC. (2022). An efficient hierarchical optic disc and cup segmentation network combined with multi-task learning and adversarial learning. J. Digit. Imaging35, 638653. 10.1007/s10278-021-00579-3

  • 28

    XiangT.ZhangC.LiuD.SongY.HuangH.CaiW. (2020). “BiO-net: learning recurrent Bi-directional connections for encoder-decoder architecture,” in Medical image computing and computer assisted intervention – miccai 2020. Editors MartelA. L.AbolmaesumiP.StoyanovD.MateusD.ZuluagaM. A.ZhouS. K.et al (Cham: Springer International Publishing), 7484.

  • 29

    ZhangJ.MeiC.LiZ.YingJ.ZhengQ.YiQ.et al (2023). Automated segmentation of optic disc and cup depicted on color fundus images using a distance-guided deep learning strategy. Biomed. Signal Process. Control86, 105163. 10.1016/j.bspc.2023.105163

  • 30

    ZhengQ.ZhangX.ZhangJ.BaiF.HuangS.PuJ.et al (2022). A texture-aware U-Net for identifying incomplete blinking from eye videography. Biomed. Signal Process. Control75, 103630. 10.1016/j.bspc.2022.103630

  • 31

    ZhouZ.Rahman SiddiqueeM. M.TajbakhshN.LiangJ. (2018). “UNet++: a nested U-net architecture for medical image segmentation,” in Deep learning in medical image analysis and multimodal learning for clinical decision support. Editors StoyanovD.TaylorZ.CarneiroG.Syeda-MahmoodT.MartelA.Maier-HeinL.et al (Cham: Springer International Publishing), 311.

  • 32

    ZhuQ.LuP. (2020). Stem cell transplantation for amyotrophic lateral sclerosis. Pattern Recognit. Lett.136, 7197. 10.1007/978-981-15-4370-8_6

Summary

Keywords

deep learning, training strategy, image segmentation, fundus image, optic cup deep learning, optic cup

Citation

Yang X, Zheng Y, Mei C, Jiang G, Tian B and Wang L (2024) UGLS: an uncertainty guided deep learning strategy for accurate image segmentation. Front. Physiol. 15:1362386. doi: 10.3389/fphys.2024.1362386

Received

05 January 2024

Accepted

26 March 2024

Published

08 April 2024

Volume

15 - 2024

Edited by

Cristiana Corsi, University of Bologna, Italy

Reviewed by

Massimo Salvi, Polytechnic University of Turin, Italy

Jolita Bernatavičienė, Vilnius University, Lithuania

Updates

Copyright

*Correspondence: Lei Wang,

† These authors have contributed equally to this work and share first authorship

Disclaimer

All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article or claim that may be made by its manufacturer is not guaranteed or endorsed by the publisher.

Outline

Figures

Cite article

Copy to clipboard


Export citation file


Share article

Article metrics