Specular Highlight Detection Based on Color Distribution for Endoscopic Images

Endoscopic imaging systems have been widely used in disease diagnosis and minimally invasive surgery. Practically, specular reflection (a.k.a. highlight) always exists in endoscopic images and significantly affects surgeons’ observation and judgment. Motivated by the fact that the values of the red channel in nonhighlight area of endoscopic images are higher than those of the green and blue ones, this paper proposes an adaptive specular highlight detection method for endoscopic images. Specifically, for each pixel, we design a criterion for specular highlight detection based on the ratio of the red channel to both the green and blue channels. With the designed criteria, we take advantage of image segmentation and then develop an adaptive threshold with respect to the differences between the red channel and the other ones of neighboring pixels. To validate the proposed method, we conduct experiments on clinical data and CVC-ClinicSpec open database. The experimental results demonstrate that the proposed method yields an averaged Precision, Accuracy, and F1-score rate of 88.76%, 99.60% and 72.56%, respectively, and outperforms the state-of-the-art approaches based on color distribution reported for endoscopic highlight detection.


INTRODUCTION
Endoscopic imaging systems have been widely used in disease diagnosis and minimally invasive surgery, which, compared to traditional surgery, takes shorter recovery time. Since the angles of lighting source and camera are almost identical, specular reflection always exists in endoscopic images and may conceal valuable information such as vessels and lesions, which is essential for surgeons' observation and judgment. Therefore, specular detection and removal are significant for endoscopic techniques, where the former is the prerequisite of the latter.
For natural images, most of the pioneer works are based on various color spaces. Xia et al. [1] proposed a joint hue, saturation, value (HSV) and red, green, blue (RGB) space threshold detection set using gradient magnitude in dark channel in RGB space. In [2], based on the global brightness of the images, the threshold of HSV color space is automatically set to separate specular reflections. Besides, the dichromatic reflection model is widely used for specular reflection detection of natural images [3]. Specifically, this method uses intensity ratio to extract specular and diffuse components from images [4,5]. Owing to the different characteristics of natural images, such as uneven color distribution and lighting, oversaturated specular highlight, the above methods designed for natural images are not applicable for endoscopic images ones.
For endoscopic images, the highlight detection methods can be mainly classified into methods based on different color spaces and the ones with classifier. Considering the real-time detection on highlight of endoscopic images, methods based on color spaces entail lower computations in comparison with those using machine learning techniques [6,7] and thus have advantages in practice, even though the latter can achieve higher accuracy. At present, the most commonly used color spaces are gray-level [8,9], HSV [10,11], and RGB [12,13]. Specifically, preset thresholds on different color spaces are employed to determine specular highlight. To reduce the impact of halo effect of highlight, Shen et al. [8] proposed a specular detection method by adopting a morphological dilation operation to enlarge the specular reflection regions which is obtained by preset threshold on gray-level image. To address the problem that some pixels in specular reflection areas have lower intensity than nonspecular reflection areas, Oh et al. [10] defined specular reflection area into absolute bright area and relative bright area, which is determined by outlier detection. However, the detected relative bright area might include not only specular highlight, but also white tissues. Zimmerman-Moreno et al. [11] used probabilistic modeling for precise extraction of highlight from segmentation on hue and saturation components for specular regions detection inside the coarse regions. In order to detect the less intense specular highlight, Arnold et al. [13] compared the original image and median filtered image, which is modified by filling each possible specular region with surrounding information. In addition to the application of traditional color spaces, some innovative approaches are proposed. Akbari et al. [14] applied nonlinear SVM classifier trained by statistical features including the mean and standard deviation extracted from each channel of both RGB and HSV color spaces to evaluate the specular detection methods with adaptive threshold. Meslouhi et al. [15] applied the luminance and normalized chrominance of the CIE-XYZ color space to identify specular regions by thresholding.
However, specular reflection detection based on color space mainly suffers from two challenges. On one hand, empirical thresholds are set in advance and, thus, are unable to be adjusted adaptively in different scenes. On the other hand, large specular reflection regions with high intensity cannot be well detected due to the limited difference between highlights and the neighboring areas [16].
To address the above issues, we, in this study, propose an adaptive detection method based on the color distribution characteristics of endoscopic images. The main contributions are summarized in the following two points.
(1) We take advantage of color distribution and propose a criterion characterized by the ratio of the red channel to both the green and blue ones, which provides a more distinguishable feature of specular reflection of endoscopic images. (2) We propose an adaptive threshold for specular reflection detection. By taking advantage of the difference between the red channel and the other ones, and integrating with overlapped windowing, the proposed adaptive threshold is applicable for large highlight regions with high intensity.

PROPOSED METHOD
Without loss of generality, given that the intensity of highlight pixel is higher in the surrounding area, this pixel in an image is defined as where I(x) is the intensity of pixel x, I mean is the mean intensity of small region to which pixel x belongs, and α represents a constant parameter. The above criterion is suitable for small specular region with high intensity which is more sensitive to surrounding information. However, pixels within large specular reflection regions are hard to be recognized due to higher I mean .
To address the above issue, we propose a criterion based on the distribution of RGB color space. Basically, in nonhighlight cases, most of endoscopic images are reddish due to the presence of hemoglobin; the value of red channel could be higher than those of green and blue ones. In highlight cases, however, the values of all three channels are nearly identical and saturated, especially for large highlight regions. Therefore, we introduce the criterion for saturated specular pixels by using the ratio between red channel and the other ones as follows: where I R (x), I G (x), and I B (x) are the intensity of RGB channels of pixel x, respectively. From (2), R of nonhighlight pixels is higher than that of highlight ones. Therefore, a threshold t is introduced to discriminate highlight and nonhighlight pixels, which is defined as where d is the difference between the red channel and green and blue ones of each small region, where A represents the set of all pixels in each patch. With the defined threshold, pixels with R lower than t are marked as specular highlight pixels, However, for darker pixels, the values of all three channels are nearly identical, like highlight pixels. To avoid the confusion between the dark and the highlight cases, we apply an empirical value of threshold in grayscale of 200 as [8], i.e., Using the above designed criterion, the pixels of an image satisfying (5) and (6) are detected as highlight. In summary, the overall flowchart of the proposed criteria with adaptive threshold for highlight detection is shown in Figure 1.
To further improve the adaptation of the criteria of threshold for highlight detection, we apply the preprocess of segmentation by splitting an image into overlapped equal-sized windows. Therefore, the problem of uneven illumination caused by single light source can be solved through setting adaptive threshold for each patch. Specifically, in each patch, the I mean in (1) is defined as Additionally, as shown in (3) and (4), d is calculated from the information of each patch, which means t in (5) computed by d is adaptively set for each patch.

RESULTS AND DISCUSSION
To evaluate the proposed method, we conduct experiments with clinical data provided by Guangzhou Improve Medical Instruments Co., Ltd., and open dataset, namely, CVC-ClinicSpec, which annotated specular highlight ground truth of endoscopic images [6]. The size of images from clinical data and CVC-ClinicSpec is 768 × 576 and 384 × 288, respectively. According to extensive experiments on clinical data using quadratic polynomial fitting, the parameter α in (1) is 2.4 and the parameters μ 1 , μ 2 , and μ 3 in (3) are set to −2.151*10 − 5 , 2.031*10 − 3 , and 1.221, respectively.
For measurement, we assess the performance of the proposed method with visual inspection and quantitative evaluation. Specifically, true positive pixels (TPPs) and false positive pixels (FPPs) are the pixels detected as highlights within and not in the ground truth, respectively. Similarly, true negative pixels (TNPs) and false negative pixels (FNPs) are the pixels marked as nonhighlights not in and within the ground truth, respectively. In this study, we employ Precision, Recall, Accuracy, and F1-score to quantify the performance of the proposed highlight detection method, which can be written as

Accuracy
TPPs + TNPs TPPs + TNPs + FPPs + FNPs In the results of the experiment shown below, for the detection results generated from clinical data, the white denotes the detected highlight. As for the results from CVC-ClinicSpec, the white, black, red, and blue denote TPPs, TNPs, FPPs, and FNPs, respectively.
Firstly, we evaluate the detected results of highlight on clinical data and CVC-ClinicSpec database, respectively. To be specific, we conduct experiments on Figures 2A,B,C using different criteria to evaluate the performance of each procedure of the proposed detection method. Figures 2A,B are the original images from clinical data, and Figure 2C is from CVC-ClinicSpec. Figures 2D,E,F show the results using the detection procedure in (1). Clearly, only sporadic specular pixels can be detected due to the limitation of using only local information for comparison. By using the proposed criteria in (1) and (5), highlight can be well detected, especially for the case of large region, as shown in Figures 2G,H,I. Besides, from the comparison between Figure 2I and L, the effectiveness of distinguishing highlight from dark regions using (6) is verified. We can learn from Figures 2J-L that the proposed method can detect highlight precisely and achieve visually satisfying results.
Next, we examine the performance improvement by using the preprocess of image segmentation. As elaborated in Proposed Method, in order to improve the adaptation of the thresholds in (1) and (5) for highlight detection, we apply image segmentation to the images from CVC-ClinicSpec database ( Figure 3A,B,C) and divide each image into nine equisized subgraphs with 50% overlapping with each other. In Figures 3G,H,I, it can be observed that there is misjudgment at the boundaries of each window due to the inability to effectively use the boundary information. As shown in Figure 3, the highlight detection with image segmentation provides a superior solution in comparison with that without image segmentation. The result is expected, since the proposed adaptive thresholds in (1) and (5) are sensitive to uneven lighting, which leads to errors for pixels with high intensity of highlight or dark regions, as shown in Figures 3J-L. Comparatively, the image segmentation with overlapping can effectively reduce the effect of uneven lighting and enables a superior detection performance to that with nonoverlapped windows and without segmentation.  To further validate the superiority of the proposed method, we compare with the benchmark specular detection methods in [8,13], as shown in Figure 4. Figures 4A and B-C are the original images from clinical data and CVC-ClinicSpec, respectively. As it illustrated in Figures 4D,E,F, it is seen that method of [8] tends to detect a larger area than the actual highlight because of the dilation operation. Figure 4G shows the method in [13] is unable to detect large specular highlight region due to the application of local surrounding information. From Figures 4J-L, we can observe that the proposed method can obtain accurate highlight detection results and can take both large and small highlight regions into account. To further compare the performance of different methods, we perform quantitative evaluation on images of Figure 4. As shown in Table 1, although the proposed method performs inferior to that of [8,13] in terms of Recall due to missing specular pixels, which are marked in blue in Figures 4K,L, our method obtains the best performance in terms of Precision, Accuracy, and F1-score.
Finally, to validate the robustness of the proposed method, we test 24 images from CVC-ClinicSpec and compare the proposed method with the benchmark methods from different perspective. Figure 5 illustrates the statistical comparison on Precision, Recall, Accuracy, and F1-score using boxplots. It can be observed that the proposed method outperforms the benchmarks in terms of Precision, Accuracy, and F1-score. Theoretically, Precision and Recall are negatively correlated. Although the proposed method yields a lower averaged Recall value, the efficiency of the method can be verified by the highest F1-score, which is the harmonic mean of Precision and Recall.

CONCLUSION
In this paper, an adaptive specular highlight detection method for endoscopic images is proposed. Taking the color distribution characteristics of endoscopic images into account, the criteria with adaptive thresholds for highlight detection are developed.
The experimental results demonstrate that the proposed method outperforms conventional schemes based on color space and can achieve averaged Precision, Accuracy, and F1-score of 88.76%, 99.60% and 72.56%, respectively.