Specular Highlight Detection Based on Color Distribution for Endoscopic Images

1. Introduction

Endoscopic imaging systems have been widely used in disease diagnosis and minimally invasive surgery, which, compared to traditional surgery, takes shorter recovery time. Since the angles of lighting source and camera are almost identical, specular reflection always exists in endoscopic images and may conceal valuable information such as vessels and lesions, which is essential for surgeons’ observation and judgment. Therefore, specular detection and removal are significant for endoscopic techniques, where the former is the prerequisite of the latter.

For natural images, most of the pioneer works are based on various color spaces. Xia et al. [1] proposed a joint hue, saturation, value (HSV) and red, green, blue (RGB) space threshold detection set using gradient magnitude in dark channel in RGB space. In [2], based on the global brightness of the images, the threshold of HSV color space is automatically set to separate specular reflections. Besides, the dichromatic reflection model is widely used for specular reflection detection of natural images [3]. Specifically, this method uses intensity ratio to extract specular and diffuse components from images [4, 5]. Owing to the different characteristics of natural images, such as uneven color distribution and lighting, oversaturated specular highlight, the above methods designed for natural images are not applicable for endoscopic images ones.

For endoscopic images, the highlight detection methods can be mainly classified into methods based on different color spaces and the ones with classifier. Considering the real-time detection on highlight of endoscopic images, methods based on color spaces entail lower computations in comparison with those using machine learning techniques [6, 7] and thus have advantages in practice, even though the latter can achieve higher accuracy. At present, the most commonly used color spaces are gray-level [8, 9], HSV [10, 11], and RGB [12, 13]. Specifically, preset thresholds on different color spaces are employed to determine specular highlight. To reduce the impact of halo effect of highlight, Shen et al. [8] proposed a specular detection method by adopting a morphological dilation operation to enlarge the specular reflection regions which is obtained by preset threshold on gray-level image. To address the problem that some pixels in specular reflection areas have lower intensity than nonspecular reflection areas, Oh et al. [10] defined specular reflection area into absolute bright area and relative bright area, which is determined by outlier detection. However, the detected relative bright area might include not only specular highlight, but also white tissues. Zimmerman-Moreno et al. [11] used probabilistic modeling for precise extraction of highlight from segmentation on hue and saturation components for specular regions detection inside the coarse regions. In order to detect the less intense specular highlight, Arnold et al. [13] compared the original image and median filtered image, which is modified by filling each possible specular region with surrounding information. In addition to the application of traditional color spaces, some innovative approaches are proposed. Akbari et al. [14] applied nonlinear SVM classifier trained by statistical features including the mean and standard deviation extracted from each channel of both RGB and HSV color spaces to evaluate the specular detection methods with adaptive threshold. Meslouhi et al. [15] applied the luminance and normalized chrominance of the CIE-XYZ color space to identify specular regions by thresholding.

However, specular reflection detection based on color space mainly suffers from two challenges. On one hand, empirical thresholds are set in advance and, thus, are unable to be adjusted adaptively in different scenes. On the other hand, large specular reflection regions with high intensity cannot be well detected due to the limited difference between highlights and the neighboring areas [16].

To address the above issues, we, in this study, propose an adaptive detection method based on the color distribution characteristics of endoscopic images. The main contributions are summarized in the following two points.

We take advantage of color distribution and propose a criterion characterized by the ratio of the red channel to both the green and blue ones, which provides a more distinguishable feature of specular reflection of endoscopic images.

We propose an adaptive threshold for specular reflection detection. By taking advantage of the difference between the red channel and the other ones, and integrating with overlapped windowing, the proposed adaptive threshold is applicable for large highlight regions with high intensity.

2. Proposed Method

Without loss of generality, given that the intensity of highlight pixel is higher in the surrounding area, this pixel in an image is defined as

$I\left(x\right)>\alpha \ast I_{mean},(1)$

where $I\left(x\right)$ is the intensity of pixel x, $I_{mean}$ is the mean intensity of small region to which pixel x belongs, and α represents a constant parameter. The above criterion is suitable for small specular region with high intensity which is more sensitive to surrounding information. However, pixels within large specular reflection regions are hard to be recognized due to higher $I_{mean}$.

To address the above issue, we propose a criterion based on the distribution of RGB color space. Basically, in nonhighlight cases, most of endoscopic images are reddish due to the presence of hemoglobin; the value of red channel could be higher than those of green and blue ones. In highlight cases, however, the values of all three channels are nearly identical and saturated, especially for large highlight regions. Therefore, we introduce the criterion for saturated specular pixels by using the ratio between red channel and the other ones as follows:

$R=\frac{I_R\left(x\right)}{\frac{1}{2}\left(I_G\left(x\right)+I_B\left(x\right)\right)},(2)$

where $I_R\left(x\right)$, $I_G\left(x\right)$, and $I_B\left(x\right)$ are the intensity of RGB channels of pixel x, respectively. From (2), R of nonhighlight pixels is higher than that of highlight ones. Therefore, a threshold t is introduced to discriminate highlight and nonhighlight pixels, which is defined as

$t={\mu }_1\ast d^2+{\mu }_2\ast d+{\mu }_3,(3)$

where d is the difference between the red channel and green and blue ones of each small region,

$d=mean\left(I_R\left(x\right)-\frac{1}{2}\left(I_G\left(x\right)+I_B\left(x\right)\right)\right)x\in \mathbf{A},(4)$

where A represents the set of all pixels in each patch. With the defined threshold, pixels with R lower than t are marked as specular highlight pixels,

$R<t,(5)$

However, for darker pixels, the values of all three channels are nearly identical, like highlight pixels. To avoid the confusion between the dark and the highlight cases, we apply an empirical value of threshold in grayscale of 200 as [8], i.e.,

$I\left(x\right)>200.(6)$

Using the above designed criterion, the pixels of an image satisfying (5) and (6) are detected as highlight. In summary, the overall flowchart of the proposed criteria with adaptive threshold for highlight detection is shown in Figure 1.

FIGURE 1

FIGURE 1. Overall flowchart of the proposed highlight detection method.

To further improve the adaptation of the criteria of threshold for highlight detection, we apply the preprocess of segmentation by splitting an image into overlapped equal-sized windows. Therefore, the problem of uneven illumination caused by single light source can be solved through setting adaptive threshold for each patch. Specifically, in each patch, the $I_{mean}$ in (1) is defined as

$I_{mean}=mean\left(I\left(x\right)\right)x\in \mathbf{A}.(7)$

Additionally, as shown in (3) and (4), d is calculated from the information of each patch, which means t in (5) computed by d is adaptively set for each patch.

3. Results and Discussion

To evaluate the proposed method, we conduct experiments with clinical data provided by Guangzhou Improve Medical Instruments Co., Ltd., and open dataset, namely, CVC-ClinicSpec, which annotated specular highlight ground truth of endoscopic images [6]. The size of images from clinical data and CVC-ClinicSpec is 768 × 576 and 384 × 288, respectively. According to extensive experiments on clinical data using quadratic polynomial fitting, the parameter α in (1) is 2.4 and the parameters ${\mu }_1$, ${\mu }_2$, and ${\mu }_3$ in (3) are set to $-2.151\text{*}{10}^{-5}$, $2.031\text{*}{10}^{-3}$, and 1.221, respectively.

For measurement, we assess the performance of the proposed method with visual inspection and quantitative evaluation. Specifically, true positive pixels (TPPs) and false positive pixels (FPPs) are the pixels detected as highlights within and not in the ground truth, respectively. Similarly, true negative pixels (TNPs) and false negative pixels (FNPs) are the pixels marked as nonhighlights not in and within the ground truth, respectively. In this study, we employ Precision, Recall, Accuracy, and F1-score to quantify the performance of the proposed highlight detection method, which can be written as

$\begin{array}{c}Precision=\frac{TPPs}{TPPs+FPPs}\\ Recall=\frac{TPPs}{TPPs+FNPs}\\ Accuracy=\frac{TPPs+TNPs}{TPPs+TNPs+FPPs+FNPs}\\ F1-score=2\cdot \frac{Precision\cdot Recall}{Precision+Recall}.\end{array}(8)$

In the results of the experiment shown below, for the detection results generated from clinical data, the white denotes the detected highlight. As for the results from CVC-ClinicSpec, the white, black, red, and blue denote TPPs, TNPs, FPPs, and FNPs, respectively.

Firstly, we evaluate the detected results of highlight on clinical data and CVC-ClinicSpec database, respectively. To be specific, we conduct experiments on Figures 2A,B,C using different criteria to evaluate the performance of each procedure of the proposed detection method. Figures 2A,B are the original images from clinical data, and Figure 2C is from CVC-ClinicSpec. Figures 2D,E,F show the results using the detection procedure in (1). Clearly, only sporadic specular pixels can be detected due to the limitation of using only local information for comparison. By using the proposed criteria in (1) and (5), highlight can be well detected, especially for the case of large region, as shown in Figures 2G,H,I. Besides, from the comparison between Figure 2I and L, the effectiveness of distinguishing highlight from dark regions using (6) is verified. We can learn from Figures 2J–L that the proposed method can detect highlight precisely and achieve visually satisfying results.

FIGURE 2

FIGURE 2. Performance of the proposed highlight detection method. (A)–(B) Original images from clinical data. (C) Original image from CVC-ClinicSpec. (D)–(F) Applying (1). (G)–(I) Applying (1) and (5). (J)–(L) Proposed method.

Next, we examine the performance improvement by using the preprocess of image segmentation. As elaborated in Proposed Method, in order to improve the adaptation of the thresholds in (1) and (5) for highlight detection, we apply image segmentation to the images from CVC-ClinicSpec database (Figure 3A,B,C) and divide each image into nine equisized subgraphs with 50% overlapping with each other. In Figures 3G,H,I, it can be observed that there is misjudgment at the boundaries of each window due to the inability to effectively use the boundary information. As shown in Figure 3, the highlight detection with image segmentation provides a superior solution in comparison with that without image segmentation. The result is expected, since the proposed adaptive thresholds in (1) and (5) are sensitive to uneven lighting, which leads to errors for pixels with high intensity of highlight or dark regions, as shown in Figures 3J–L. Comparatively, the image segmentation with overlapping can effectively reduce the effect of uneven lighting and enables a superior detection performance to that with nonoverlapped windows and without segmentation.

FIGURE 3

FIGURE 3. Performance of the proposed highlight detection method with different traversal methods. (A)–(C) Original images from CVC-ClinicSpec. (D)–(F) 50% overlapped windows one-fourth the size of the original images. (G)–(I) Nonoverlapped windows one-ninth the size of the original images. (J)–(L) Using global information.

To further validate the superiority of the proposed method, we compare with the benchmark specular detection methods in [8, 13], as shown in Figure 4. Figures 4A and B‐C are the original images from clinical data and CVC-ClinicSpec, respectively. As it illustrated in Figures 4D,E,F, it is seen that method of [8] tends to detect a larger area than the actual highlight because of the dilation operation. Figure 4G shows the method in [13] is unable to detect large specular highlight region due to the application of local surrounding information. From Figures 4J–L, we can observe that the proposed method can obtain accurate highlight detection results and can take both large and small highlight regions into account. To further compare the performance of different methods, we perform quantitative evaluation on images of Figure 4. As shown in Table 1, although the proposed method performs inferior to that of [8, 13] in terms of Recall due to missing specular pixels, which are marked in blue in Figures 4K,L, our method obtains the best performance in terms of Precision, Accuracy, and F1-score.

FIGURE 4

FIGURE 4. Comparison between different methods. (A) Original image from clinical data. (B)–(C) Original images from CVC-ClinicSpec. (D)–(F) [8]. (G)–(I) [13]. (J)–(L) Proposed method.

TABLE 1

TABLE 1. Comparison between TPPs, FPPs, Precision, Recall, Accuracy, and F1-score.

Finally, to validate the robustness of the proposed method, we test 24 images from CVC-ClinicSpec and compare the proposed method with the benchmark methods from different perspective. Figure 5 illustrates the statistical comparison on Precision, Recall, Accuracy, and F1-score using boxplots. It can be observed that the proposed method outperforms the benchmarks in terms of Precision, Accuracy, and F1-score. Theoretically, Precision and Recall are negatively correlated. Although the proposed method yields a lower averaged Recall value, the efficiency of the method can be verified by the highest F1-score, which is the harmonic mean of Precision and Recall.

FIGURE 5

FIGURE 5. Comparison of Precision, Recall, Accuracy, and F1-score, where the crosses mark averages. (A) Comparison of Precision. (B) Comparison of Recall. (C) Comparison of Accuracy. (D) Comparison of F1-score.

4. Conclusion

In this paper, an adaptive specular highlight detection method for endoscopic images is proposed. Taking the color distribution characteristics of endoscopic images into account, the criteria with adaptive thresholds for highlight detection are developed. The experimental results demonstrate that the proposed method outperforms conventional schemes based on color space and can achieve averaged Precision, Accuracy, and F1-score of 88.76%, 99.60% and 72.56%, respectively.

Data Availability Statement

The original contributions presented in the study are included in the article/Supplementary Material, further inquiries can be directed to the corresponding author.

Author Contributions

BY, WC, and HZ contributed the ideas. WC performed the experiments and drafted the paper. BY, WC, QZ, and HZ all did the revision.

Funding

This work was supported by the Natural Science Foundation of Guangdong Province, Grant No. 2019A1515011940; Science and Technology Program of Guangzhou, Grant Nos. 2019050001 and 202002030353; Science and Technology Planning Project of Guangdong Province, Grant Nos. 2017B030308009, 2017KZ010101, and 2018A030313990; Special Project for Youth Top-notch Scholars of Guangdong Province, Grant No. 2016TQ03X100; and also Scientific Research Cultivation Project for Young Scholars of South China Normal University, Grant No. 19KJ16.

Conflict of Interest

Authors BY and HZ were employed by the company SCNU Qingyuan Institute of Science and Technology Innovation Co., Ltd.

The remaining authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

References

1. Xia W, Chen ECS, Pautler SE, Peters TM. A global optimization method for specular highlight removal from a single image. IEEE Access (2019) 7:125976–90. doi:10.1109/ACCESS.2019.2939229

CrossRef Full Text | Google Scholar

2. Morgand A, Tamaazousti M. Generic and real-time detection of specular reflections in images. In: 2014 International conference on computer vision theory and applications (VISAPP); 2014 January 5–8; Lisbon, Portugal (2014). p. 274–82.

CrossRef Full Text | Google Scholar

3. Shafer SA. Using color to separate reflection components. Color Res Appl (1985) 10:210–8. doi:10.1002/col.5080100409

CrossRef Full Text | Google Scholar

4. Shen HL, Zheng ZH. Real-time highlight removal using intensity ratio. Appl Optic (2013) 52 4483–93. doi:10.1364/AO.52.004483

CrossRef Full Text | Google Scholar

5. Suo J, An D, Ji X, Wang H, Dai Q. Fast and high quality highlight removal from a single image. IEEE Trans Image Process (2016) 25:5441–54. doi:10.1109/TIP.2016.2605002

PubMed Abstract | CrossRef Full Text | Google Scholar

6. Sánchez F, Bernal J, Sánchez-Montes C, Rodríguez de Miguel C, Fernández-Esparrach G. Bright spot regions segmentation and classification for specular highlights detection in colonoscopy videos. Mach Vis Appl (2017) 28:917–36. doi:10.1007/s00138-017-0864-0

CrossRef Full Text | Google Scholar

7. Gao Y, Yang J, Ma S, Ai D, Lin T, Tang S, et al. Dynamic searching and classification for highlight removal on endoscopic image. Procedia Comp Sci (2017) 107:762–7. doi:10.1016/j.procs.2017.03.161

CrossRef Full Text | Google Scholar

8. Shen DF, Guo JJ, Lin GS, Lin JY. Content-aware specular reflection suppression based on adaptive image inpainting and neural network for endoscopic images. Comput Methods Progr Biomed (2020) 192 105414. doi:10.1016/j.cmpb.2020.105414

CrossRef Full Text | Google Scholar

9. Chu Y, Li H, Li X, Ding Y, Yang X, Ai D, et al. Endoscopic image feature matching via motion consensus and global bilateral regression. Comput Methods Progr Biomed (2020) 190 105370. doi:10.1016/j.cmpb.2020.105370

CrossRef Full Text | Google Scholar

10. Oh J, Hwang S, Lee J, Tavanapong W, Wong J, de Groen PC. Informative frame classification for endoscopy video. Med Image Anal (2007) 11 110–27. doi:10.1016/j.media.2006.10.003

PubMed Abstract | CrossRef Full Text | Google Scholar

11. Zimmerman-Moreno G, Greenspan H. Automatic detection of specular reflections in uterine cervix images. SPIE Med Imaging (2006) 6144:2037–45. doi:10.1117/12.653089

CrossRef Full Text | Google Scholar

12. Alsaleh SM, Aviles-Rivero AI, Hahn JK. Retouchimg: fusioning from-local-to-global context detection and graph data structures for fully-automatic specular reflection removal for endoscopic images. Comput Med Imag Graph (2019) 73 39–48. doi:10.1016/j.compmedimag.2019.02.002

CrossRef Full Text | Google Scholar

13. Arnold M, Ghosh A, Ameling S, Lacey G. Automatic segmentation and inpainting of specular highlights for endoscopic imaging. EURASIP Journal on Image and Video Processing (2010) 1–12. doi:10.1155/2010/814319

CrossRef Full Text | Google Scholar

14. Akbari M, Mohrekesh M, Najariani K, Karimi N, Samavi S, Soroushmehr SMR. Adaptive specular reflection detection and inpainting in colonoscopy video frames. In: 2018 25th IEEE International conference on image processing; 2018 October 7–10; Athens, Greece. (ICIP) (2018). p. 3134–8.

CrossRef Full Text | Google Scholar

15. Meslouhi O, Kardouchi M, Allali H, Gadi T, Benkaddour Y. Automatic detection and inpainting of specular reflections for colposcopic images. Open Computer Science (2011) 1:341–54. doi:10.2478/s13537-011-0020-2

CrossRef Full Text | Google Scholar

16. Voronin V, Semenishchev E, Zelensky A, Agaian S. Specular reflection detection algorithm for endoscopic images. Electron Imag (2019) 2019:222–1–222–6. doi:10.2352/ISSN.2470-1173.2019.11.IPAS-222

CrossRef Full Text | Google Scholar