High Dynamic Range Microscopy for Color Selective Virtual De-Staining of Immunocytological Specimens

Abstract

Immunocytochemical markers are increasingly applied for diagnosis of diseases. Usually two or more marker stains are applied at once, together with a counterstain for a reliable microscopic investigation of cell specimens. As a preprocessing step for the detection of marker-positive cells, other stains should be removed by image processing techniques. This virtual de-staining can be achieved by color separation algorithms, thus removing the undesired stain and reconstructing an image containing only the desired marker component. Known algorithms for color separation however show significant color artifacts, which are caused by inevitable non-linearities during image acquisition. In this paper we develop high dynamic range (HDR) microscopy color separation, which removes non-linearities, dynamic range limitations, as well as quantization effects and enables accurate color separation and virtual de-staining. Color accuracy in the virtual de-stained images is provided by the ∆E00 measure. Our simulations demonstrate that the perceivable color error is reduced from 86% to 0.65%. Finally, we provide results for HDR-based virtual destaining on cell samples from cytopathological routine which confirm the performance of our approach.

-

In recent years proteins within cells have been identified that are directly related to particular diseases, e.g. cancer. Many of these proteins can be visualized with immunocytochemical marker stains. For example cells infected by a virus can be visualized by attaching a stain to antibodies which in turn bind to a specific up-regulated protein after virus infection. The antibody, docked to the affected cell, is stained by a chromogene which then can be identified in a microscopic inspection. Human papilloma viruses (HPV), e.g., are responsible for almost all cervical cancers [ 1 ], and the infection can be detected with the specific p16-antibody in cervical brush smears with high sensitivity [ 2 ]. An immunocytological investigation can thus non-invasively identify high risk patients for this type of cancer. The automated scanning of slides followed by computer aided detection of marker-positive cells will help physicians to cope with the high load of specimens in a screening scenario and to reach a more reliable diagnostic result.

In general the cells are stained with a so-called counterstain (e.g. hematoxylin) and one or more immunocytochemical marker stains. The counterstain allows to investigate the cell morphology and secures robust and reliable autofocus for automated image acquisition even in slide areas without marker positive cells [ 3 ]. Moreover, since the counterstain allows to identify a nucleus inside a marker-positive cell, these can be differentiated from artifacts with similar color than the marker stain. By algorithmically removing the counterstain with a color separation algorithm, the slide can be virtually destained. Thus markerpositivity and nuclei morphology can be individually investigated. Therefore this virtual de-staining is an important preprocessing step for the automated detection of marker-positive cells in immunocytochemically stained specimens.

Color separation in microscopic images has been proposed by Ruifrok [ 4 ]. This algorithm is based on the assumption of a linear sensor response of the camera. Many cameras do not obey this linear behavior. Moreover, non-linearities occur due to quantization steps and clipping effects at the dynamic range limits of the camera. These inevitable effects in the imaging process in turn cause perceivable color errors in the individual channels of the color separation.

In this paper we quantify the error of the colors in the separated images. To this end we calculate the ∆E00 measure, which directly allows to rate whether or not a color error is perceivable by a human. We then demonstrate that high dynamic range (HDR) microscopy [ 5 ] removes the imaging limitations and consequently enables the image acquisition of the full range of biological variations. We show that this improved imaging almost completely removes the color errors in the separated images. Finally, we demonstrate results on clinical images. 2

Materials and Methods

Leica DMLA (10x) and JAI CVM90 camera were used. Epithelial cells of colon cancer in peritoneal effusion exfoliation were marked with pan-Cytokeratin (antibody: DAKO, Denmark) and stained with red AEC chromogen (DAKO). HPV positive squamous epithelial cells in PAP smears were marked with anti-p16 (antibody: Santa Cruz) and stained with brown DAB chromogen (DAKO). Slides were coverslipped after counterstain with hematoxilin of nuclei of the cells (blue). 2.1

Color Separation

The Lambert-Beer Law describes the attenuation of light of intensity I0 passing through a semi-transparent of optical density OD

Im = I0 e OD (1) The optical density, in turn, is linearly related to the concentration of the matter. Assuming a linear behavior of the camera transfer function, the optical density can be estimated from the measured intensity Im[ 4 ]. RGB values in photomicroscopic images can be transferred to the OD-space by inverting (1) for each color channel separately. On the other hand, the OD vectors d (desired stain) and u (undesired counterstain) can be estimated from the RGB values of these stains. The counterstain component of an OD vector x can be removed by applying the basis transformation A := (d, u, n) 1 (with n = d u), removing the counterstain component and applying the inverse of A again xd = A 1  0 0 0  A x    1 0 0  0 0 1 Applying this procedure pixelwise and transferring the results back to RGB space with (1) yields an image where the undesired component has been removed. 2.2

HDR Imaging

Algorithmic approaches to increase the dynamic range of the imaging system are based on the acquisition of an exposure set, i.e. a set of images of the same scene acquired under different exposure time [ 6 ]. The imaging model for the camera is

Ij = f (qj + n1) + n2 = f (tj ϕ + n1) + n2 (3) where qj = tj ϕ is the incident radiant energy, tj the exposure time of exposure j 2 [1..N ], ϕ the radiant power, and n1 and n2 are noise sources in the imaging process. The acquired image I, however is a non-linear mapping of the incident radiant energy by the non-linear camera transfer function (CTF) f . Saturation and black level are covered by the CTF as well.

To recover the full dynamic range, all images from the exposure set have to be combined. To this end the non-linearities have to be removed by applying the inverse f 1 of the CTF first. The estimates ϕˆj = t1j f 1(Ij ) are combined by a weighted sum according to ϕˆ = ∑N

j=1 wj ϕˆj ∑N j=1 wj (2) (4) The weights wj account for the reliability of the individual estimates. For clipped (white- or black-level) values the reliability weights are 0. 2.3

Simulation

For the quantitative assessment of the errors due to quantization, non-linearities and clipping effects we simulate the image acquisition process of an OD vector x = αd + βu + γn. The OD vectors are transferred back to RGB space via (1) and the non-linearities of the camera are modeled by applying clipping, quantization and the non-linear CTF from the camera. The scalar multiples of x are chosen such that the OD increases from translucent material to opaque and that all quantization steps of the RGB space in between are traversed. An artificial 250 200 150 B100 50 0 250

Ground truth LDR LDR linearized HDR 200 150 100 G 50 image containing all these RGB values is created. Likewise, such an image is produced after correcting the non-linearities of the camera with the inverse CTF and from a simulated HDR image (exposure times 1, 2, 4, 8 and 16 ms). We apply the color separation on these images as described in section 2.1 and compare the results with the ground truth. 3

Results

To depict the deviations in RGB space after color separation, Figure 1 compares the RGB values of the ground truth, conventional imaging, correction for nonlinearities and HDR imaging. Furthermore, we use the ground truth from the simulations to quantify the errors introduced during image acquisition process. Firstly, the corruption of the original signal is measured with the signal to noise ratio. Secondly, the perceptual distance is quantified by computing the ∆E00 error [ 7 ]. A ∆E00 > 1 indicates a noticeable perceptual error, therefore we compute the fraction of points where this error is larger than one. The quantitative results are summarized in Table 1. Figure 2 shows cells from specimens where the virtual destaining technique was used to remove the counterstain or marker. The RGB plot from Figure 1 clearly reveals that a systematic error is caused if the non-linearities of the camera are not considered. This systematic error can be removed by correcting the non-linear camera transfer function. Still, the accuracy of the color separation is degraded as the quantization error is propagated, especially for low RGB values during the conversion from RGB to OD space. Even more, the limited dynamic range of the conventional imaging is not able to cope with the high variation in biological specimens. As soon as the black level of the camera is reached in at least one camera channel, the color separation cannot decompose the color information correctly and major color artifacts corrupt the virtually destained slide.

HDR imaging is capable to overcome these shortcomings by combining images with different exposure times. Quantization effects are reduced by averaging the information from this set of images and thus computing an image in floating point arithmetic. We showed that the perceivable error can be significantly reduced by HDR and that the quality of virtually destained slides is improved. Acknowledgement. The project was supported by the excellence initiative of the German federal and state governments. We thank Prof. R.H. Jansen, german director of the Thai-German Graduate School of Engineering, KMUTNB, for fruitful discussion and cooperation.

References

1. Schiffman M , Castle PE . The promise of global cervical cancer prevention . N Engl J Med . 2005 ; 353 : 2101 - 4 .

2. Denton KJ , Bergeron C , et al. The sensitivity and specificity of cytology vs HPV testing for detecting high-grade cervical disease in the triage of ASC-US and LSIL pap cytology results . Am J Clin Pathol . 2010 ; 134 ( 1 ): 12 - 21 .

3. Bell A , Schneider T , Mu¨ ller-Frank D , et al. Fully automatic screening of immunocytochemically stained cytopathological specimens for early cancer detection . In: Proc SPIE; 2007 . p. 651431 - 1 -11.

4. Ruifrok AC , Johnston DA . Quantification of histochemical staining by color deconvolution . Anal Quant Cytol Histol . 2001 ; 23 ( 4 ): 291 - 9 .

5. Bell AA , Brauers J , Kaftan JN , et al. High dynamic range microscopy for cytopathological cancer diagnosis . IEEE J Sel Top Sig Proc . 2009 ; 3 ( 1 ): 170 - 84 .

6. Mann S , Picard RW . On being undigital with digital cameras: extending dynamic range by combining differently exposed pictures . In: Proc IS&T; 1995 . p. 442 - 8 .

7. Sharma G , Wu W , Dalal EN . The CIEDE 2000 color-difference formula: implementation notes, supplementary test data, and mathematical observations . Color Res Appl . 2005 ; 30 ( 1 ): 21 - 30 .