Nuclear Segmentation in Histopathological Images Using Two-Stage Stacked U-Nets With Attention Mechanism

Abstract

Nuclei segmentation is a fundamental but challenging task in histopathological image analysis. One of the main problems is the existence of overlapping regions which increases the difficulty of independent nuclei separation. In this study, to solve the segmentation of nuclei and overlapping regions, we introduce a nuclei segmentation method based on two-stage learning framework consisting of two connected Stacked U-Nets (SUNets). The proposed SUNets consists of four parallel backbone nets, which are merged by the attention generation model. In the first stage, a Stacked U-Net is utilized to predict pixel-wise segmentation of nuclei. The output binary map together with RGB values of the original images are concatenated as the input of the second stage of SUNets. Due to the sizable imbalance of overlapping and background regions, the first network is trained with cross-entropy loss, while the second network is trained with focal loss. We applied the method on two publicly available datasets and achieved state-of-the-art performance for nuclei segmentation–mean Aggregated Jaccard Index (AJI) results were 0.5965 and 0.6210, and F1 scores were 0.8247 and 0.8060, respectively; our method also segmented the overlapping regions between nuclei, with average AJI = 0.3254. The proposed two-stage learning framework outperforms many current segmentation methods, and the consistent good segmentation performance on images from different organs indicates the generalized adaptability of our approach.

Introduction

Morphological changes in the cell nucleus are considered an important signal in many diseases (Gurcan et al., 2009) and can provide clinically meaningful information during diagnosis, especially for cancers (Chow et al., 2015). The conventional method involves manual inspection and analyses performed by pathologists to make diagnostic assessments based on certain morphology features of the nucleus. However, this manual assessment is a tedious and time-consuming task that can be beset by shortcomings such as poor sensitivity, specificity, and low reproducibility. This fact underscores the urgent need to develop and refine rapid and automated histology image analysis methods; nuclear segmentation is often the most important and fundamental one (Fuchs and Buhmann, 2011).

Deep neural networks, especially deep convolutional neural networks (CNNs), have been the dominant techniques for visual analysis and have recently achieved great success for biological object detection and segmentation in medical images (Aoki et al., 2020; Wen et al., 2020; Xiang et al., 2020; Yao et al., 2020; Zhang et al., 2020). U-Net (Ronneberger et al., 2015) is a classical architecture based on fully convolutional network (FCN) (Long et al., 2015), which has been widely used and has obtained promising performance when applied to the task of image segmentation (Litjens et al., 2017; Kong et al., 2020). Furthermore, many studies have improved the original U-Net, such as Res-Unet (Xiao et al., 2018) or dense-Unet (Zhu et al., 2019). Among many improved networks, multi-scale and stacked networks have attracted intensive studies. For example, Wu et al. (2018) utilized the multi-scale network followed networks (MS-NFN) model to segment blood vessels in retinal images, while Sevastopolsky et al. (2018) proposed a special cascade network which stacked two kinds of blocks, U-Net or Res-UNet, for optical disc and cup segmentation. On the other hand, Stacked U-Nets (SUNets) (Shah et al., 2018) can be considered as further improvement as they iteratively combine features from different image scales while maintaining resolution. Leveraging the feature computation power of U-Nets in a deeper network architecture, SUNets are capable of handling images with increased complexity.

Due to the complexity of nuclei shape, imperfect slide preparation or staining, overlapping nuclei, and scanning artifacts, automatic nuclei instance segmentation is still a computationally challenging task. Compared to manual nuclei segmentation, however, automated segmentation methods based on cutting-edge deep learning technology have the potential to foster improvement.

Inspired by the attention mechanism idea (Vaswani et al., 2017) and the aforementioned segmentation approaches, we developed a two-stage learning framework based on two SUNets to solve the challenges in nuclei segmentation in histopathological images. We converted the nuclei segmentation task into a two-stage task; both stages were composed of a SUNets with the same architecture. The outputs of our SUNets were then post-processed through a watershed algorithm (Roerdink and Meijster, 2000) to achieve the instance-level segmentation. We also compared our method with current existing popular algorithms. When applied to a publicly available multi-organ dataset, our method achieved improved segmentation accuracy results and solved the segmentation challenge of overlapped nuclei regions with high fidelity. In addition, we applied our method on another publicly available dataset and obtained reliable segmentation results as well. Details of our method are described in section “Methodology”, comparisons on two independent image sets are elaborated in section “Results and Discussions”, and finally, conclusion is presented in section “Conclusion.”

Methodology

In this work, a two-stage framework is proposed to automate segmentation of nuclei regions and regions of overlapping nuclei. The flow of the two-stage framework is shown in Figure 1A. Two SUNets (Sevastopolsky et al., 2018; Figure 1B) with the same architecture are utilized in both stages. The first stage aims to segment nuclei regions, and the second stage is designed to segment regions of overlapping nuclei. Nucleus instance segmentation results from the first stage are updated by adding overlapped regions derived from the second stage. In this section, we present our two-stage method in detail.

FIGURE 1

Segmentation of Nucleic Regions

Stage 1 in our nuclei segmentation framework predicts the region of each nucleus using CNNs, specifically SUNets. The SUNets consist of four parallel backbone nets. Four images derived from the original histopathology image by scaling at 1.25×, 1.0×, 0.75×, and 0.50× serve as inputs to Stage 1. Four semantic feature maps of differently sized backbone nets are extracted and fed into an attention generation module, which contains eight CNN layers belonging to the first two blocks of ResNet34 (Gulshan et al., 2016) together with one ReLU (rectified linear unit) layer. Finally, a weight matrix for each scaled backbone net is returned and used to weight and sum the predictions of the four scaled backbone nets and generate the final result. The architecture of the backbone net of our SUNets is illustrated in Figure 1C. The backbone nets which extract semantic information from the input image is based on a modified deep network–VGG16 (Wang et al., 2015). The down-sampling part of the network (the first 21 layers from the left side in Figure 1C) contains a series of convolutional layers with ReLU activation function (Nair and Hinton, 2010). The last down-sampling layer represents the semantic features of the input image. At the end of Stage 1, the mask segmentation of nuclei regions is generated. Image features are transformed into same-sized mask segmentation result with pixel gray value of 0 (background region) and 1 (nuclei region). Then the watershed algorithm is utilized to get the first round of nucleus instance segmentation result.

Segmentation of Overlapped Nuclei Regions

Stage 2 of our framework is segmenting the overlapped nuclei region. To achieve this, we utilize the same SUNets as the ones used in the first stage. The difference is that we construct new generalized images as input images. These input images contain not only the original RGB images but also the binary masks predicted by Stage 1 as the fourth set of values. Due to the imbalanced distribution of overlapping nuclei regions and background regions, in the training step we employ focal loss (Lin et al., 2017), which is a method that was first proposed to address the sizable imbalance between the positive and negative distributions. Compared with the traditional cross entropy loss, focal loss introduces a function to measure the contribution of the hard and easily classified sample to the total loss. At the end of Stage 2, the segmentation result of overlapped regions is generated with pixel gray value of 0 (not overlapped regions) and 1 (overlapped region). Finally, nucleus instance segmentation results from the first stage are updated by adding the corresponding overlapped objects which have at least 10 overlapped pixels with the segmented objects derived from the second stage.

Evaluation Metric

We evaluate our method by using two types of metrics: object-level and pixel-level metrics.

The commonly used Aggregated Jaccard Index (AJI) (Kumar et al., 2017) is utilized as object-level evaluation metric. The AJI is an extension of the Jaccard Index, and is defined as

where GT_i (i = 1, 2,…, K) is a pixel belonging to ground truth (GT) nuclei object, PRD_j (j = 1, 2,…, L) is a pixel belonging to the predicted nuclei objects. PRD_j^∗(i) is the connected component object from the predicted objects that have the maximum Jaccard Index with the GT nuclei, and U is the union of predicted objects that does not have intersections with any GT objects.

We also employ precision, recall rate, and F1-score (Sasaki, 2007) as the pixel-level evaluation metrics, which are defined as follows:

where FP, TP, and FN denote false positive, true positive, and false negative, respectively.

Datasets

We evaluated our method by utilizing two publicly available datasets sourced from the Cancer Genome Atlas (TCGA)¹ (Kumar et al., 2017) and the Triple-Negative Breast Cancer (TNBC) (Naylor et al., 2017). The two image datasets used herein are subsets of the TCGA and TNBC; the GT nuclear segmentation for these sets is manually performed by experts, and these datasets are widely used as the gold standard for nuclei segmentation studies.

The first dataset (selected from the TCGA database) (Tomczak et al., 2015) consisted of 30 hematoxylin and eosin (H&E) stained images. In this dataset, images were collected from seven different organs [bladder, breast, colon, kidney, liver, prostate, and stomach; we manually extracted 1,000 × 1,000-pixel size small patches from whole slide images (WSIs)] and used as the training (n = 16) and testing (n = 14) image sets.

The second dataset (selected from the TNBC database) contained 50 H&E stained images with 512 × 512 resolution. All images in the second dataset were extracted from 11 TNBC patients with multiple cell types including endothelial cells, inflammatory cells, and myoepithelial breast cells. We used this dataset to compare the performance of our nucleus instance segmentation with other popular methods.

Network Implementation Details

All the experiments were conducted using TensorFlow (Abadi et al., 2016). Both SUNets used in the two-stage framework were trained using the same strategy–the learning rate used for training was 0.0001, stochastic gradient descent was used as an optimizer to minimize the loss function with momentum 0.9, and the batch size was set to 4. For the consideration of GPU memory limitation and performance, the input image size of all networks was set to 384 × 384 pixels. Additionally, we employed various augmentation techniques during the training process such as image rotation, vertical flipping, and horizontal flipping. Due to the sizable imbalance of overlapping and background regions, we trained the second Stacked U-Net with focal loss, whereas the first one was trained with cross-entropy loss. The networks were trained for 8 h on two NVIDIA Tesla P100 GPU cells for 20 epochs with CUDA 9.0 (NVIDIA, United States) library.

Results and Discussion

Our method outperformed present state-of-the-art methods on the two datasets (described in section “Datasets”) in the integrity of the segmentation of a single nucleus and the segmentation accuracy, and especially in the segmentation of overlapped nuclei regions. We compared our method against several deep learning based methods listed in Table 1, such as FCN-8 (Long et al., 2015), Mask R-CNN (He et al., 2015), U-Net (Ronneberger et al., 2015), CNN3 (Kumar et al., 2017), DIST (Naylor et al., 2019), SUNets, U-Net (DLA), a two-stage U-net (Mahbod et al., 2019), and two-stage learning U-Net (DLA) (Kang et al., 2019). In order to make the comparison objectively, we followed the same training and testing set split criteria suggested by Kumar et al. (2017). The results of the comparison confirmed the superiority of our method which achieved an average AJI of 59.65% and F1-score of 82.47% (Tables 1, 2 and Supplementary File S1). The performance results of the algorithms utilized in this comparison are sourced from the respective publication (Tables 1–3). Regarding the comparison of segmentation results (i.e., the extracted part of the whole image) of our method and the corresponding GT segmentation on different tissue types in the test image (displayed in Figure 2), our method was able to segment the majority of overlapped nuclei regions (the average AJI value of segmented overlapping regions was 0.3254) (Figure 3). The segmentation results of our algorithm are further illustrated in Figure 3, where we selected two examples of segmentation at random and compared them with the GT. We also applied our method on the TNBC dataset and compared the experimental results with other methods (Table 3)–DeconvNet (Noh et al., 2015), FCN-8 (Long et al., 2015), U-Net (Ronneberger et al., 2015), Ensemble method (Naylor et al., 2017), DIST (Naylor et al., 2019), and two-stage learning U-Net (DLA) method (Kang et al., 2019). Our method has the top AJI value (AJI = 0.621) but the second highest F1 score (F1 score = 0.806). This result demonstrates that our method has a high generalization ability since these images vary from tissue types to cell types. The values of AJI and F1 score of other methods in Tables 1–3 are taken from published works (Kang et al., 2019; Naylor et al., 2019).

FIGURE 2

FIGURE 3

Conclusion

Nuclei segmentation has a wide utility in multiple biologically related tasks such as the quantitative analyses of the cellular constitution of tissues. Nuclei segmentation, however, is a notoriously challenging problem due to shape variation, imperfect slide digitalization, and the existence of overlapped or contact regions. In this study, we present a Stacked U-Net-based two-stage learning framework for nuclei segmentation of histopathological images. We divide the process of nuclei segmentation into the following steps: in the first step we segment the nuclei regions, and in the second step we divide the overlapping regions. Finally, nuclei instance segmentation results are updated by merging the two segmentation results. The results on two diverse public datasets show that our method outperforms most of the current standard segmentation methods and achieves state-of-the-art segmentation of not only the nuclei instances but also the overlapped regions.

References

• 1

  AbadiM.BarhamP.ChenJ.ChenZ.DevinM.DavisA.et al (2016). “TensorFlow: a system for large-Scale machine learning,” in Proceedings of the Operating Systems Design and Implementation, Amsterdam, 265–283.

  • Google Scholar
  • View reference in article

• 2

  AokiT.YamadaA.KatoY.SaitoH.TsuboiA.NakadaA.et al (2020). Automatic detection of various abnormalities in capsule endoscopy videos by a deep learning-based system: a multicenter study.Gastrointest. Endosc.10.1016/j.gie.2020.04.080[Epub ahead of print].

  • Pubmed Abstract
  • CrossRef
  • Google Scholar
  • View reference in article

• 3

  ChowK.-H.FactorR. E.UllmanK. S. (2015). The nuclear envelope environment and its cancer connections.Nat. Rev. Cancer12196–209. 10.1038/jid.2014.371

  • Pubmed Abstract
  • CrossRef
  • Google Scholar
  • View reference in article

• 4

  FuchsT. J.BuhmannJ. M. (2011). Computational pathology: challenges and promises for tissue analysis.Comput. Med. Imaging Graph.35515–530. 10.1016/j.compmedimag.2011.02.006

  • Pubmed Abstract
  • CrossRef
  • Google Scholar
  • View reference in article

• 5

  GulshanV.PengL.CoramM.StumpeM. C.WuD.NarayanaswamyA.et al (2016). Development and validation of a deep learning algorithm for detection of diabetic retinopathy in retinal fundus photographs.J. Am. Med. Assoc.3162402–2410. 10.1001/jama.2016.17216

  • CrossRef
  • Google Scholar
  • View reference in article

• 6

  GurcanM. N.BoucheronL. E.CanA.MadabhushiA.RajpootN. M.YenerB. (2009). Histopathological image analysis: a review.IEEE Rev. Biomed. Eng.2147–171. 10.1109/RBME.2009.2034865

  • Pubmed Abstract
  • CrossRef
  • Google Scholar
  • View reference in article

• 7

  HeK.GkioxariG.DollarP.GirshickR. (2015). “Mask R-CNN,” in Proceedings of the IEEE International Conference on Computer Vision, (Venice: IEEE), 1440–1448.

  • Google Scholar
  • View reference in article

• 8

  KangQ.LaoQ.FevensT. (2019). “Nuclei segmentation in histopathological images using two-stage learning,” in Medical Image Computing and Computer Assisted Intervention, ed.ShenD. (Cham: Springer International Publishing), 703–711. 10.1007/978-3-030-32239-7_78

  • CrossRef
  • Google Scholar
  • View reference in article

• 9

  KongY.LiH.RenY.GenchevG.WangX.ZhaoH.et al (2020). Automated yeast cells counting using a parallel U-Net based two-stage framework.OSA Contin.3982–993. 10.1364/osac.388082

  • CrossRef
  • Google Scholar
  • View reference in article

• 10

  KumarN.VermaR.SharmaS.BhargavaS.VahadaneA.SethiA. (2017). A dataset and a technique for generalized nuclear segmentation for computational pathology.IEEE Trans. Med. Imaging361550–1560. 10.1109/tmi.2017.2677499

  • Pubmed Abstract
  • CrossRef
  • Google Scholar
  • View reference in article

• 11

  LinT. Y.GoyalP.GirshickR.HeK.DollarP. (2017). “Focal loss for dense object detection,” in Proceedings of the IEEE International Conference on Computer Vision, (Cambridge, MA: IEEE), 2999–3007. 10.1109/ICCV.2017.324

  • CrossRef
  • Google Scholar
  • View reference in article

• 12

  LitjensG.KooiT.BejnordiB. E.SetioA. A. A.CiompiF.GhafoorianM.et al (2017). A survey on deep learning in medical image analysis.Med. Image Anal.4260–88. 10.1016/j.media.2017.07.005

  • Pubmed Abstract
  • CrossRef
  • Google Scholar
  • View reference in article

• 13

  LongJ.ShelhamerE.DarrellT. (2015). “Fully convolutional networks for semantic segmentation,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, (Boston, MA: IEEE), 3431–3440. 10.1109/ICCVW.2019.00113

  • CrossRef
  • Google Scholar
  • View reference in article

• 14

  MahbodA.SchaeferG.EllingerI.EckerR.SmedbyÖ.WangC. (2019). “A two-stage U-net algorithm for segmentation of nuclei in H&E-stained tissues,” in Digital Pathology, edsReyes-AldasoroC. C.JanowczykA.VetaM.BankheadP.SirinukunwattanaK. (Cham: Springer International Publishing), 75–82. 10.1007/978-3-030-23937-4_9

  • CrossRef
  • Google Scholar
  • View reference in article

• 15

  NairV.HintonG. E. (2010). “Rectified linear units improve restricted boltzmann machines vinod,” in Proceedings of the 27th International Conference on Machine Learning ICML, Haifa.

  • Google Scholar
  • View reference in article

• 16

  NaylorP.LaeM.ReyalF.WalterT. (2017). “Nuclei segmentation in histopathology images using deep neural networks,” in Proceedings of the 14th International Symposium on Biomedical Imaging, Melbourne, 933–936. 10.1109/ISBI.2017.7950669

  • CrossRef
  • Google Scholar
  • View reference in article

• 17

  NaylorP.LaéM.ReyalF.WalterT. (2019). Segmentation of nuclei in histopathology images by deep regression of the distance map.IEEE Trans. Med. Imaging38448–459. 10.1109/TMI.2018.2865709

  • CrossRef
  • Google Scholar
  • View reference in article

• 18

  NohH.HongS.HanB. (2015). “Learning deconvolution network for semantic segmentation,” in Proceedings of the International Conference on Computer Vision, (Santiago: IEEE), 1520–1528. 10.1109/ICCV.2015.178

  • CrossRef
  • Google Scholar
  • View reference in article

• 19

  RoerdinkJ. B. T. M.MeijsterA. (2000). The Watershed transform: definitions, algorithms and parallelization strategies.Fundam. Inform.41187–228. 10.3233/fi-2000-411207

  • CrossRef
  • Google Scholar
  • View reference in article

• 20

  RonnebergerO.FischerP.BroxT. (2015). U-Net: convolutional networks for biomedical image segmentation.Med. Image Comput. Comput. Assist. Interv.9351234–241. 10.1007/978-3-319-24574-4_28

  • CrossRef
  • Google Scholar
  • View reference in article

• 21

  SasakiY. (2007). The truth of the F-measure.Teach Tutor Mater11–5. 10.1109/tmag.2013.2278570

  • CrossRef
  • Google Scholar
  • View reference in article

• 22

  SevastopolskyA.DrapakS.KiselevK.SnyderB. M.KeenanJ. D.GeorgievskayaA. (2018). Stack-U-Net: refinement network for improved optic disc and cup image segmentation.arXiv [Preprint]10.1117/12.2511572

  • CrossRef
  • Google Scholar
  • View reference in article

• 23

  ShahS.GhoshP.DavisL. S.GoldsteinT. (2018). Stacked U-Nets: A No-Frills Approach to Natural Image Segmentation. Available online at: http://arxiv.org/abs/1804.10343(accessed April 27, 2018).

  • Google Scholar
  • View reference in article

• 24

  TomczakK.CzerwiñskaP.WiznerowiczM. (2015). Review the cancer genome atlas (TCGA): an immeasurable source of knowledge.Wspolczesna Onkol. Oncol.201568–77. 10.5114/wo.2014.47136

  • Pubmed Abstract
  • CrossRef
  • Google Scholar
  • View reference in article

• 25

  VaswaniA.ShazeerN.ParmarN.UszkoreitJ.JonesL.GomezA. N.et al (2017). Attention is all you need.arXiv [Preprint]10.1109/2943.974352

  • CrossRef
  • Google Scholar
  • View reference in article

• 26

  WangL.ShengG.HuangW.QiaoY. (2015). Places205-VGGNet Models for Scene Recognition.Available online at: arXiv.org > cs > arXiv:1508.01667 (accessed August 7, 2015).

  • Google Scholar
  • View reference in article

• 27

  WenJ.Thibeau-SutreE.Diaz-MeloM.Samper-GonzalezJ.RoutierA.BottaniS.et al (2020). Convolutional neural networks for classification of Alzheimer’s disease: overview and reproducible evaluation.Med. Image Anal.63:101694. 10.1016/j.media.2020.101694

  • CrossRef
  • Google Scholar
  • View reference in article

• 28

  WuY.XiaY.SongY.ZhangY.CaiW. (2018). “Multiscale network followed network model for retinal vessel segmentation,” in Medical Image Computing and Computer Assisted Intervention, edsFrangiA. F.SchnabelJ. A.DavatzikosC.Alberola-LópezC.FichtingerG. (Cham: Springer International Publishing), 119–126. 10.1007/978-3-030-00934-2_14

  • CrossRef
  • Google Scholar
  • View reference in article

• 29

  XiangH.LimH.FesslerJ. A.DewarajaY. K. (2020). A deep neural network for fast and accurate scatter estimation in quantitative SPECT/CT under challenging scatter conditions.Eur. J. Nucl. Med. Mol. Imaging10.1007/s00259-020-04840-9[Epub ahead of print].

  • Pubmed Abstract
  • CrossRef
  • Google Scholar
  • View reference in article

• 30

  XiaoX.LianS.LuoZ.LiS. (2018). “Weighted res-UNet for high-quality retina vessel segmentation,” in Proceedings of the International Conference on Information Technology in Medicine and Education, (Hangzhou: IEEE), 327–331.

  • Google Scholar
  • View reference in article

• 31

  YaoS.YanJ.WuM.YangX.ZhangW.LuH.et al (2020). Texture synthesis based thyroid nodule detection from medical ultrasound images: interpreting and suppressing the adversarial effect of in-place manual annotation.Front. Bioeng. Biotechnol.8:599. 10.3389/fbioe.2020.00599

  • Pubmed Abstract
  • CrossRef
  • Google Scholar
  • View reference in article

• 32

  ZhangY.de HaanK.RivensonY.LiJ.DelisA.OzcanA. (2020). Digital synthesis of histological stains using micro-structured and multiplexed virtual staining of label-free tissue.Light Sci. Appl.9:78. 10.1038/s41377-020-0315-y

  • Pubmed Abstract
  • CrossRef
  • Google Scholar
  • View reference in article

• 33

  ZhuH.ShiF.WangL.HungS.ChenM. H.WangS.et al (2019). Dilated dense U-net for infant hippocampus subfield segmentation.Front. Neuroinform.13:30. 10.3389/fninf.2019.00030

  • Pubmed Abstract
  • CrossRef
  • Google Scholar
  • View reference in article