Endoscopic images typically sufer from various image degradation such as specular reflection and motion blur due to camera motion. In this paper, we propose a new matrix decomposition method to eliminate the presence of specular reflections (highlights) from endoscopic images. The proposal combines the characteristics of specular reflections and the features of dimensionality reduction through singular value thresholding (SVT) to obtain a highlight free image. Addition of specular reflections with an image alters the singular values of the image matrix. The algorithm iteratively eliminates the changes that occurred to various singular values until all the highlight pixels are eliminated. In order to avoid any loss of significant information, slicing operation is performed on the residue image matrix remained after the initial process. The useful information is extracted from this residue matrix and reinserted with the semi-low-rank component obtained in SVT operation. The experiments and tests reveal that this new matrix decomposition method eliminates the specular reflections from the endoscopic images.
Over the years, endoscopy has evolved as one of the necessary tools used in the medical
diagnostic domain. It spreads its branches into colonoscopy, bronchoscopy, esophagoscopy,
etc.[
The images obtained from the endoscopy may not be ideal as they contain numerous artefacts
like the presence of specular reflections (also called highlights), blurred out regions,
overexposed and under-exposed pixels, presence of bubbles and debris, chromatic aberrations, etc.
[
Highlight removal has been a widely researched topic over the last few years, with several
diferent approaches proposed by many researchers [
Many of the early algorithms considered the highlight removal as a two-stage process. The
ifrst stage tries to identify the highlight pixels, and in the next stage, the identified pixels are
removed and replaced with new information derived from the image itself. In [
Highlight removal can be accomplished either by first performing the highlight detection followed by highlight removal or by performing removal without the prior detection.
(i). ℎ : The whole image can be split into reflection and difusion
components. The difusion component varies very slowly, whereas the reflection component
varies rapidly. In [
(ii). ℎ ℎ : In this approach [
(iii). ℎ : Generally, these techniques are used after highlight
detection. Once the highlight pixels are identified, these are inpainted with proper values [
(iv). ℎ : Supervised learning approaches show higher
eficiency in specularity removal [
(v). ℎ : Robust Principal Component Analysis (RPCA) is a powerful
tool to be used with unlabelled data. The highlight regions are considered a sparse component,
and the highlight free image is the low-rank component embedded in the endoscopic image
[
This paper proposes a matrix decomposition method to remove the specular reflections from endoscopic images to obtain high-quality images. The proposed method overcomes a lot of demerits of conventional highlight removal algorithms. The merits of the proposed algorithm are listed below.
• The proposed approach eliminates the need for a large labelled dataset as it is not training
driven.
• The proposed approach does not require pre-detection of highlighted pixels.
• The approach does not demand scene-wise tuning of parameters, unlike other works
[19][
In addition to these merits, the algorithm provides a fast rate of convergence by applying the characteristics of highlight pixel in the update equations.
The remainder of the paper is structured as follows. Section 2 discusses the mathematical preliminaries required to better understand the proposed method. The proposed new matrix decomposition method for specular reflection removal is outlined in Section 3. The experimental results are presented in Section 4. Finally, the conclusions and future scope are drawn in Section 5.
In this section, we briefly explain some of the mathematical ingredients which are necessary for the development of the proposed method.
It is an important mathematical tool used to factorize an M matrix into three sub-matrices as shown in Fig. 1. The SVD of an M× N matrix is given as, × N matrix [22]. It decomposes a
XM× N = [U]M× r[Σ] r× r[V]r× N, = min{M, N} are the square roots of the eigenvalues of A A. where, [U]M× r is the matrix of orthonormal eigenvectors of AA . [V]r× N is the matrix of orthonormal eigenvectors of A A. [Σ] r× r is the diagonal matrix of the singular values which (1) (2) (3)
SVD
Soft thresholding of a matrix is used to discard insignificant elements of the matrix with reference to some threshold. Along with discarding these elements, it also reduces the strengths of other elements also. The operator is defined as [23],
(X) = [ (), ∀ ∈ X] () = () × max{ − , 0 }
SVT operator works in two stages. In the initial stage, it performs the SVD operation on the image matrix. Then, the soft thresholding is performed on the singular value matrix. The reconstructed matrix using the modified singular values is the result of this operation. Mathematically, we can represent the operation as follows [23]: (X) =
U 1 (Σ) V where, X = UΣ V is the singular value decomposition of the matrix X, with Σ being the singular value matrix.
In the proposed method, the specular reflection components embedded in the endoscopic image is removed by applying a new matrix decomposition technique. The original endoscopic image (M) is decomposed into a semi-low-rank component (L) and a highlight component (H) as follows.
M = L + H (4)
If we attempt to decompose the matrix into a truly low-rank component and a sparse component as in RPCA [24], many of the highlight pixels will be removed from the original image M. But the highlight component of the image cannot be perfectly sparse in nature. It contributes significantly to the low rank component of the image. In addition, there can be some sparse non-highlight information making the component L a semi-low-rank component. ~ ~ ~ ~ (a) (b) (c)
The key idea to obtain the semi-low-rank component is to extract details out of the original endoscopic image iteratively until all the specular components are removed completely. Removing all specular components in a single step needs to subtract the exact contribution of the specularity in each of the singular values. Extracting this information from the original image is practically not possible. Further, the contribution difers from image to image. So the solution becomes an iterative procedure that can converge when all the specularities are removed. Thus, the algorithm can be summarized as below:
Algorithm 1: Basic steps followed in the proposed method 1 Perform the singular value decomposition of the original image. 2 Apply the ‘soft thresholding operator’ on the singular values to reduce the contribution of specularity on them. 3 Reconstruct the image to obtain the semi-low-rank component. 4 Check for convergence. If not converged, repeat from step 1.
The steps 1, 2 and 3 can be performed by the use of ‘singular value thresholding operator’ defined in Eq. (3). Thus the extraction operation is given by,
The part of M remaining after the formation of L is the residue matrix represented as H, which is expected to contain only the highlight component.
L = (M) H = M −
L (5) (6)
While the algorithm outlined appears to result in highlight component alone, we not that it does not always hold true. This observation can be argued with the reasons listed below: i. There is useful information in an endoscopic image that may be considered sparse. These components have contributions towards the least significant singular values. Applying soft thresolding technique removes this information and will be accumulated on H. ii. In order to remove specularity completely, proper selection of the parameter is essential.
If is selected high, 1 becomes less and only purely sparse components will be removed and it takes more time. So the value of must be very low. But in that case, there is a possibility of relevant information also get removed due to the ‘higher step size’.
This implies that in each iteration we remove both highlight pixels and some relevant information using a very small value of . The decomposition is given in Fig. 3.
Original Image
Equation 5 and 6
L
Semi Low-Rank Component
H Semi Sparse Component
Specular Useful Component Information
Now the H component contains both highlight pixels and useful information. We proceed to extract this useful information from H and reinsert the same into L, so that the highlight component will be available in H and the remaining in L.
The component remaining after the initial step is, H = M - L. Our method of extracting the specular components from H is as follows. All the pixels in the image H are made zero if they are not following the characteristics of specular reflections. Then the remaining component represents only the specular reflection, i.e., for ∀ ℎ ∈ H,
S’(ℎ) = {︃ℎ,
whenever ℎ is a highlight pixel 0, otherwise
The slicing operation is defined as, where,
H = Slicing(M - L) Slicing(H) = [ S’(ℎ),
∀ ℎ ∈ H ]
S’(ℎ) = ishighlight(ℎ) × ℎ That is, S’(ℎ) is thresholding operation.
Each pixel ℎ is decided to be a highlight pixel by applying the characteristics of highlight
pixels [
ishighlight(ℎ = (ℎ , ℎ, ℎ )) = {︃+1, if ℎ > 0, otherwise
The threshold for each channel is obtained from the image under processing using the following equation.
= (channel) + × (channel), ∀ ∈ [ R, G, B ] where, and are the statistical mean and standard deviation of the channel in image H respectively. Applying Eq. (9) and (10) in Eq. (7) yields the operation required to extract highlight pixels from the image H. Clearly the operation obtained is simply the intensity level slicing operation. The thresholds are obtained from Eq. (10).
Now the operation performed for obtaining only the highlight pixels from M - L is, (7) (8) (9) (10) (11) (12)
Once the highlight-only image H is obtained, the remaining component is given by M - L H. Let it be Y. This is the useful information that must reside in the semi-low-rank component. Now this term is added back to L, so that in the next iteration, new value of L is obtained by the singular value thresholding operation on (L + Y) = (M - H). The update equations for the semi-low-rank component and the highlight component can be summarised as,
L+1 = (M − H) H+1 = Slicing(M − L+1) (13)
The above mentioned iterative procedure is repeated until (M - L - H) becomes insignificantly small. That is ‖M − L − H‖2 <= , where ‖.‖2 is the Frobenius norm. Resulting L contains all the features other than specular reflections and H contains all the highlight regions. The entire procedure is outlined in Fig. 4 and steps are enumerated in Algorithm 2.
Slicing
Is Converged ?
Yes No
Output/ Highlight Free Image
Algorithm 2: Removing Highlight components from WCE images
To evaluate the results obtained from the proposed algorithm, we have performed experiments on two diferent publicly available datasets - the Kvasir-colonoscopy 1 dataset and the CVCClinicSpec2 dataset 3.
The results obtained are validated objectively using the percentage of remaining highlight. We design two experiments to validate the proposed method. In the first experiment, images from two datasets are processed using the proposed method and the percentage of highlight remaining in each image after processing is calculated as given by Eq. 14:
Percentage of highlight =
No. of pixels satisfying the conditions for highlight
Total No. of pixels in the original Imge (14)
A pixel satisfying the conditions for highlight is decided by Eq. (9). Low value of percentage indicates a high eficiency for the algorithm.
From the Kvasir-colonoscopy dataset, 500 images are selected for testing our algorithm, the resolution of which ranges from 720× 576 to 1920× 1072. All of the images contain significant specular reflections. Fig. 5 shows the result of the first experiment, and only two images from each dataset are shown for the sake of illustration. The first two rows are the images from the Kvasir-colonoscopy dataset, and the remaining two rows are the images from the CNC-ClinicSpec dataset. Despite the images from diferent datasets, the output of the proposed approach is observed to be consistent. However, it is observed that the boundaries of the highlight pixels in the original image remain. The boundaries are defined by the shadow regions created by the illumination, and these dark regions are clearly visible in the enlarged view of the original image.
As part of the first experiment, we have calculated the percentage of highlights remaining in diferent images from two datasets using Eq. (14). The reduction in highlight is shown in Fig. 6 for 5 sample images.
In the second experiment, the proposed method is compared with the state of the art matrix decomposition methods based on singular values. The methods compared are (a) 1/2 Regularized RPCA [25], (b) AS-RPCA [26], (c) PCP [24] and (d) FPCP [27]. The comparison is shown in Fig. 7. The corresponding statistical distribution of the percentage of highlights that remained in the ifnal image is presented in Table 1, for the 500 images tested from Kvasir- colonoscopy dataset. As seen from the Table 1, the proposed approach results in a 0.0052% (standard deviation 9.7714e-06) of highlights remaining as compared to next best performing approach with 0.0276% highlights intact. The obvious advantage of the proposed approach can be seen against PCA based approaches introspected from the Table 1. 2http://www.cvc.uab.es/CVC-Colon/index.php/cvc-clinicspec/ 3The experiments are conducted on a computer with Intel(R) Xeon(R) CPUE5-1620 2 with 3.7GHz and 8GB RAM and using Python 3.9.
In addition to Table 1, we also conduct the statistical significance analysis using the Box-plots as shown in Figure 8. The obtained results indicate the superiority over the other approaches.
In addition to quantitative results, we also conduct an analysis for time complexity of the proposed approach as against other state-of-the-art approaches. As seen from the results presented in Table 2, the average time complexity of the proposed approach is significantly lower compared to next best performing approach (i.e.,FPCP). The results indicate that superiority of the proposed approach both in terms of highlight removal and time complexity.
The employed Kvasir-colonoscopy dataset consists of images with padded boundaries due to the nature of imaging. These boundary regions result in pseudo-artefacts and cannot be avoided if the whole image is considered. Better strategies for handling such boundaries are not investigated in this work and will be studied in future works. Representation 56 (2018) 188–200. [18] M. S. Subodh Raj, S. N. George, 1/2 regularized rpca technique for 3d human action recovery, in: 2020 IEEE 17th India Council International Conference (INDICON), 2020, pp. 1–5. [19] F. d. S. Queiroz, I. R. Tsang, Automatic segmentation of specular reflections for endoscopic images based on sparse and low-rank decomposition, in: 2014 27th SIBGRAPI Conference on Graphics, Patterns and Images, 2014, pp. 282–289. [20] G. Fu, Q. Zhang, C. Song, Q. Lin, C. Xiao, Specular highlight removal for real-world images,
Computer Graphics Forum 38 (????) 253–263. [21] J. Guo, Z. Zhou, L. Wang, Single image highlight removal with a sparse and low-rank reflection model, in: Computer Vision – ECCV 2018, 2018, pp. 282–298. [22] V. Klema, A. Laub, The singular value decomposition: Its computation and some applications, IEEE Transactions on Automatic Control 25 (1980) 164–176. [23] J.-F. Cai, E. J. Candès, Z. Shen, A singular value thresholding algorithm for matrix completion, SIAM Journal on Optimization 20 (2010) 1956–1982. [24] E. J. Candès, X. Li, Y. Ma, J. Wright, Robust principal component analysis?, J. ACM 58 (2011). [25] S. Raj M S, S. George, l 1/2 regularized rpca technique for 3d human action recovery, 2020, pp. 1–5. [26] G. Liu, S. Yan, Active subspace: Toward scalable low-rank learning, Neural computation 24 (2012). [27] P. Rodríguez, B. Wohlberg, Fast principal component pursuit via alternating minimization, in: 2013 IEEE International Conference on Image Processing, 2013, pp. 69–73.