=Paper=
{{Paper
|id=None
|storemode=property
|title=High Dynamic Range Microscopy for Color Selective Virtual De-Staining of Immunocytological Specimens
|pdfUrl=https://ceur-ws.org/Vol-715/bvm2011_8.pdf
|volume=Vol-715
}}
==High Dynamic Range Microscopy for Color Selective Virtual De-Staining of Immunocytological Specimens==
https://ceur-ws.org/Vol-715/bvm2011_8.pdf
High Dynamic Range Microscopy for Color
Selective Virtual De-Staining of
Immunocytological Specimens
David Friedrich1 , André Bell1 , Kraisorn Chaisaowong1 , Till Braunschweig2 ,
Ruth Knüchel-Clarke2 , Til Aach1
1
Institute of Imaging & Computer Vision, RWTH Aachen University
2
Institute of Pathology, University Hospital Aachen
david.friedrich@lfb.rwth-aachen.de
Abstract. Immunocytochemical markers are increasingly applied for
diagnosis of diseases. Usually two or more marker stains are applied
at once, together with a counterstain for a reliable microscopic inves-
tigation of cell specimens. As a preprocessing step for the detection of
marker-positive cells, other stains should be removed by image processing
techniques. This virtual de-staining can be achieved by color separation
algorithms, thus removing the undesired stain and reconstructing an im-
age containing only the desired marker component. Known algorithms
for color separation however show significant color artifacts, which are
caused by inevitable non-linearities during image acquisition. In this
paper we develop high dynamic range (HDR) microscopy color separa-
tion, which removes non-linearities, dynamic range limitations, as well
as quantization effects and enables accurate color separation and virtual
de-staining. Color accuracy in the virtual de-stained images is provided
by the ∆E00 measure. Our simulations demonstrate that the perceiv-
able color error is reduced from 86% to 0.65%. Finally, we provide results
for HDR-based virtual destaining on cell samples from cytopathological
routine which confirm the performance of our approach.
1 Introduction
In recent years proteins within cells have been identified that are directly related
to particular diseases, e.g. cancer. Many of these proteins can be visualized with
immunocytochemical marker stains. For example cells infected by a virus can
be visualized by attaching a stain to antibodies which in turn bind to a spe-
cific up-regulated protein after virus infection. The antibody, docked to the
affected cell, is stained by a chromogene which then can be identified in a mi-
croscopic inspection. Human papilloma viruses (HPV), e.g., are responsible for
almost all cervical cancers [1], and the infection can be detected with the specific
p16-antibody in cervical brush smears with high sensitivity [2]. An immunocy-
tological investigation can thus non-invasively identify high risk patients for this
type of cancer. The automated scanning of slides followed by computer aided de-
tection of marker-positive cells will help physicians to cope with the high load of
specimens in a screening scenario and to reach a more reliable diagnostic result.
High Dynamic Range Microscopy 29
In general the cells are stained with a so-called counterstain (e.g. hema-
toxylin) and one or more immunocytochemical marker stains. The counterstain
allows to investigate the cell morphology and secures robust and reliable autofo-
cus for automated image acquisition even in slide areas without marker positive
cells [3]. Moreover, since the counterstain allows to identify a nucleus inside a
marker-positive cell, these can be differentiated from artifacts with similar color
than the marker stain. By algorithmically removing the counterstain with a
color separation algorithm, the slide can be virtually destained. Thus marker-
positivity and nuclei morphology can be individually investigated. Therefore
this virtual de-staining is an important preprocessing step for the automated
detection of marker-positive cells in immunocytochemically stained specimens.
Color separation in microscopic images has been proposed by Ruifrok [4].
This algorithm is based on the assumption of a linear sensor response of the cam-
era. Many cameras do not obey this linear behavior. Moreover, non-linearities
occur due to quantization steps and clipping effects at the dynamic range limits
of the camera. These inevitable effects in the imaging process in turn cause
perceivable color errors in the individual channels of the color separation.
In this paper we quantify the error of the colors in the separated images. To
this end we calculate the ∆E00 measure, which directly allows to rate whether
or not a color error is perceivable by a human. We then demonstrate that high
dynamic range (HDR) microscopy [5] removes the imaging limitations and con-
sequently enables the image acquisition of the full range of biological variations.
We show that this improved imaging almost completely removes the color errors
in the separated images. Finally, we demonstrate results on clinical images.
2 Materials and Methods
Leica DMLA (10x) and JAI CVM90 camera were used. Epithelial cells of colon
cancer in peritoneal effusion exfoliation were marked with pan-Cytokeratin (an-
tibody: DAKO, Denmark) and stained with red AEC chromogen (DAKO). HPV
positive squamous epithelial cells in PAP smears were marked with anti-p16 (an-
tibody: Santa Cruz) and stained with brown DAB chromogen (DAKO). Slides
were coverslipped after counterstain with hematoxilin of nuclei of the cells (blue).
2.1 Color Separation
The Lambert-Beer Law describes the attenuation of light of intensity I0 passing
through a semi-transparent of optical density OD
Im = I0 · e−OD (1)
The optical density, in turn, is linearly related to the concentration of the matter.
Assuming a linear behavior of the camera transfer function, the optical density
can be estimated from the measured intensity Im [4]. RGB values in photo-
microscopic images can be transferred to the OD-space by inverting (1) for each
30 Friedrich et al.
color channel separately. On the other hand, the OD vectors d (desired stain)
and u (undesired counterstain) can be estimated from the RGB values of these
stains. The counterstain component of an OD vector x can be removed by
applying the basis transformation A := (d, u, n)−1 (with n = d × u), removing
the counterstain component and applying the inverse of A again
100
xd = A−1 0 0 0 A · x (2)
001
Applying this procedure pixelwise and transferring the results back to RGB space
with (1) yields an image where the undesired component has been removed.
2.2 HDR Imaging
Algorithmic approaches to increase the dynamic range of the imaging system are
based on the acquisition of an exposure set, i.e. a set of images of the same scene
acquired under different exposure time [6]. The imaging model for the camera
is
Ij = f (qj + n1 ) + n2 = f (tj ϕ + n1 ) + n2 (3)
where qj = tj ϕ is the incident radiant energy, tj the exposure time of exposure
j ∈ [1..N ], ϕ the radiant power, and n1 and n2 are noise sources in the imaging
process. The acquired image I, however is a non-linear mapping of the incident
radiant energy by the non-linear camera transfer function (CTF) f . Saturation
and black level are covered by the CTF as well.
To recover the full dynamic range, all images from the exposure set have to
be combined. To this end the non-linearities have to be removed by applying
the inverse f −1 of the CTF first. The estimates ϕ̂j = t1j f −1 (Ij ) are combined
by a weighted sum according to
∑N
j=1 wj ϕ̂j
ϕ̂ = ∑N (4)
j=1 wj
The weights wj account for the reliability of the individual estimates. For clipped
(white- or black-level) values the reliability weights are 0.
2.3 Simulation
For the quantitative assessment of the errors due to quantization, non-linearities
and clipping effects we simulate the image acquisition process of an OD vector
x = αd + βu + γn. The OD vectors are transferred back to RGB space via (1)
and the non-linearities of the camera are modeled by applying clipping, quanti-
zation and the non-linear CTF from the camera. The scalar multiples of x are
chosen such that the OD increases from translucent material to opaque and that
all quantization steps of the RGB space in between are traversed. An artificial
High Dynamic Range Microscopy 31
image containing all these RGB values is created. Likewise, such an image is
produced after correcting the non-linearities of the camera with the inverse CTF
and from a simulated HDR image (exposure times 1, 2, 4, 8 and 16 ms). We ap-
ply the color separation on these images as described in section 2.1 and compare
the results with the ground truth.
3 Results
To depict the deviations in RGB space after color separation, Figure 1 compares
the RGB values of the ground truth, conventional imaging, correction for non-
Ground truth
LDR
LDR linearized
HDR
250
200
150
B
100
50
250
0 200
250
200 150
150 100
100 50
50
0 0
R
G
Fig. 1. Simulation results: RGB values after color separation. A systematic error is
inherent in the values for the conventional LDR imaging. Even if the non-linearities are
corrected, errors remain, especially for low RGB values. They are resolved by HDR.
(a) (b) (c) (d) (e) (f) (g)
Fig. 2. DAB (top) and AEC (bottom) examples from clinical routine (a); after virtual
counterstain removal (b-d); after virtual marker de-staining (e-g). Input images are
(b,e) from conventional imaging, (c,f) after correction through linearization and (d,g)
from HDR imaging.
32 Friedrich et al.
linearities and HDR imaging. Furthermore, we use the ground truth from the
simulations to quantify the errors introduced during image acquisition process.
Firstly, the corruption of the original signal is measured with the signal to noise
ratio. Secondly, the perceptual distance is quantified by computing the ∆E00
error [7]. A ∆E00 > 1 indicates a noticeable perceptual error, therefore we com-
pute the fraction of points where this error is larger than one. The quantitative
results are summarized in Table 1. Figure 2 shows cells from specimens where
the virtual destaining technique was used to remove the counterstain or marker.
Table 1. Results of simulation: SNR ∆E00 > 1
quantitative assessment of error.
conventional imaging 21.3 86.0 %
correct non-linear CTF 30.3 5.6 %
HDR imaging 39.5 0.65 %
4 Discussion
The RGB plot from Figure 1 clearly reveals that a systematic error is caused
if the non-linearities of the camera are not considered. This systematic error
can be removed by correcting the non-linear camera transfer function. Still,
the accuracy of the color separation is degraded as the quantization error is
propagated, especially for low RGB values during the conversion from RGB to
OD space. Even more, the limited dynamic range of the conventional imaging
is not able to cope with the high variation in biological specimens. As soon as
the black level of the camera is reached in at least one camera channel, the color
separation cannot decompose the color information correctly and major color
artifacts corrupt the virtually destained slide.
HDR imaging is capable to overcome these shortcomings by combining images
with different exposure times. Quantization effects are reduced by averaging the
information from this set of images and thus computing an image in floating
point arithmetic. We showed that the perceivable error can be significantly
reduced by HDR and that the quality of virtually destained slides is improved.
Acknowledgement. The project was supported by the excellence initiative of
the German federal and state governments. We thank Prof. R.H. Jansen, german
director of the Thai-German Graduate School of Engineering, KMUTNB, for
fruitful discussion and cooperation.
References
1. Schiffman M, Castle PE. The promise of global cervical cancer prevention. N Engl
J Med. 2005;353:2101–4.
High Dynamic Range Microscopy 33
2. Denton KJ, Bergeron C, et al. The sensitivity and specificity of cytology vs HPV
testing for detecting high-grade cervical disease in the triage of ASC-US and LSIL
pap cytology results. Am J Clin Pathol. 2010;134(1):12–21.
3. Bell A, Schneider T, Müller-Frank D, et al. Fully automatic screening of immuno-
cytochemically stained cytopathological specimens for early cancer detection. In:
Proc SPIE; 2007. p. 651431–1–11.
4. Ruifrok AC, Johnston DA. Quantification of histochemical staining by color decon-
volution. Anal Quant Cytol Histol. 2001;23(4):291–9.
5. Bell AA, Brauers J, Kaftan JN, et al. High dynamic range microscopy for cytopatho-
logical cancer diagnosis. IEEE J Sel Top Sig Proc. 2009;3(1):170–84.
6. Mann S, Picard RW. On being undigital with digital cameras: extending dynamic
range by combining differently exposed pictures. In: Proc IS&T; 1995. p. 442–8.
7. Sharma G, Wu W, Dalal EN. The CIEDE 2000 color-difference formula: implemen-
tation notes, supplementary test data, and mathematical observations. Color Res
Appl. 2005;30(1):21–30.