A combination of several classi¯ers using global features for the content description of medical images is proposed. Beside well known texture histogram features, downscaled representations of the original images are used, which preserve spatial information and utilize distance measures which are robust regarding common variations in radiation dose, translation, and local deformation. These features were evaluated for the annotation task and the interactive query task in ImageCLEF 2005 without using additional textual information or query re¯nement mechanisms. For the annotation task, a categorization rate of 86.7% was obtained, which ranks second among all submissions. When applied in the interactive query task, the image content descriptors yielded a mean average precision (MAP) of 0.0751, which is rank 14 of 28 submitted runs. As the image deformation model is not ¯t for interactive retrieval tasks, two mechanisms are evaluated regarding the trade-o® between loss of accuracy and speed increase: hierarchical ¯ltering and prototype selection.
ImageCLEFmed 2005 consists of several challenges for content-based retrieval[
The annotation task consists of 9,000 images grouped into 57 categories and 1,000 images to be automatically categorized. It should be noted that the category de¯nition is based solely on the aspects of 1. imaging modality, i.e. identi¯cation of the imaging device (three di®erent device types) 2. imaging direction, i.e. relative position of the body part towards the imaging device 3. anatomy of the body part examined, and 4. biological system, which encodes certain contrast agents and a coarse description of the diagnostic motivation for the imaging.
Thus, the category de¯nition does not incorporate any diagnosis information, e.g. the detection of pathologies or their quantitative analysis. 2.1
Based on earlier experiments conducted on a similar image set, three types of features and
similarity measures were employed[
Tamura et al. proposed a set of texture features to capture global texture properties of an
image, namely coarseness, contrast, and directionality[
Hm(q) log
2Hm(q) Hm(q) + Hm(r) + Hm(r) log
2Hm(r) Hm(q) + Hm(r) ¸
To keep spatial information about the image content, down-scaled representations of the original images are used and the accompanying distance measures work directly on intensity values. It is therefore possible to incorporate a priori knowledge into the distance measure by modelling typical variability in the image data, which does not alter the category that the image belongs to. The cross-correlation function (CCF) from signal processing determines the maximum correlation between two 2D image representations, each one of size h £ h: sCCF(q; r) =
max jmj;jnj·d 8 <
h h
Px=1 Py=1(r(x ¡ m; y ¡ n) ¡ r) ¢ (q(x; y) ¡ q) :qPxh=1 Pyh=1 (r(x ¡ m; y ¡ n) ¡ r)2 Pxh=1 Pyh=1 (q(x ¡ m; y ¡ n) ¡ q)2; (2)
Here, q(x; y) and r(x; y) refer to intensity values at a pixel position on the scaled representations of q and r, respectively. Note that sCCF is a similarity measure and the values lie between 0 and 1. CCF includes robustness regarding two very common variabilites among the images: translation, which is explicitly tested within the search window of size 2d + 1, and radiation dose, which is normalized by subtracting the average intensity values q and r. For the experiments, downscaling to 32 £ 32 pixels and a translation window of size d = 4 was used, i.e. translation can vary from ¡4 to +4 pixels in both x- and y-direction.
While sCCF considers global displacements, i.e. translations of entire images, pathologies,
implants and normal inter-patient variability suggest to model local deformations for medical images.
This can be done with an image distortion model (IDM)[
X Y dIDM(q; r) = X X min X x=1 y=1 jx0j;jy0j·W1 :jx00j;jy00j·W2 ; (3)
Again, q(x; y) and r(x; y) refer to intensity values of the scaled representations. Note that each pixel of q must be mapped on some pixel in r, whereas not all pixels of r need to be target of a mapping. Two parameters steer dIDM: W1 de¯nes the size of the neighborhood when searching for a corresponding pixel. To prevent a totally unordered pixel mapping, it is useful to incorporate the local neighborhood as context when evaluating a correspondence hypothesis. The size of the context information is controlled by W2. For the experiments, W1 = 2, i.e. a 5 £ 5 pixel search window, and W2 = 1, i.e. a 3 £ 3 context patch are used. Also, better results are obtained if the gradient images are used instead of the original images, because the correspondence search will then focus on contrast and be robust to global intensity di®erences due to radiation dose. It should be noted that this distance measure is computationally expensive as each window size in°uences the computation time in quadratic manner. The images were scaled to a ¯xed height of 32 pixels, i.e. the original aspect ratio was preserved. 2.2
To obtain a decision q 7! c 2 f1 : : : Cg for a query image q, a nearest neighbor classi¯er evaluating k nearest neighbors according to a distance measure is used (k-NN). It simply votes for the category which accumulated the most votes among the k reference images closest to q. This classi¯er also allows visual feedback in interactive queries. 2.3
Prior experiments showed that the performance of the single classi¯ers can be improved
significantly if their single decisions are combined[
For comparability, the distance values (dTamura, dIDM ) are normalized at ¯rst over all distances d(q; ri); i = 1 : : : N between sample q and each reference ri: d0(q; ri) =
d(q; ri) PnN=1 d(q; rn) Afterwards, a new distance measure can be obtained by a weighted sum of distance measures d1, d2.
dc(q; r) = ¸ ¢ d01(q; r) + (1 ¡ ¸) ¢ d02(q; r); ¸ 2 [0; 1] For a similarity measure s, d(q; r) := 1 ¡ s(q; r) is used and the normalization is performed afterwards. Thus, the parallel combination of the three classi¯ers results in dcombined(q; r) = + + ¸Tamura ¢ d0Tamura(q; r) ¸CCF ¢ d0CCF(q; r)) ¸IDM ¢ d0IDM(q; r) with ¸Tamura; ¸CCF; ¸IDM ¸ 0 and ¸Tamura + ¸CCF + ¸IDM = 1. (4) (5) (6)
The combined classi¯cation process relies on three parameters: ¸Tamura, ¸CCF and k for the number of nearest neighbors to be evaluated (¸IDM is linear dependent). To obtain suitable values for them, the reference set of 9,000 images was split at random into a training set of 8,000 images and a test set of 1,000 images. The best parameter values found for this con¯guration are then applied to the 1,000 query images. For practical reasons, the matrices DTamura = (dTamura(qi; rj ))ij , SCCF = (sCCF(qi; rj ))ij , and DIDM = (dIDM(qi; rj ))ij are computed once. Afterwards, all combination experiments can be performed rather quickly by processing the matrices. 2.5
Since the distance computations for the scaled representations are rather expensive, there is -in general- great interest for prototype selection which allows to save computation time, storage space and might even improve the categorization rate by removing possible outliers in the reference set.
Prototype sets can be obtained in various ways [
[ c=1:::C : 8 8 99 <arg min < X dcombined(r; r0)==
r2Rc :q02Rc ;;
These elements frc0g; c = 1::C yield the smallest sum of distances to all members of their respective category.
The prototypes are used to obtain a dissimilarity-space representation of the reference images and the unknown images:
The retrieval task uses 50,024 images for reference and consists of 25 queries, which are given as a combination of text information and query images, with some queries specifying both positive and negative example images. While the image data for the annotation task only contains grayscale images from mostly x-ray modalities (plain radiography, °uoroscopy, and angiography), the image material in this task is much more heterogeneous: It also contains photographs, ultrasonic imaging and even scans of illustrations used for teaching. Note that the interactive task demands a higher level of image understanding, since several of the 25 queries directly refer to the diagnosis of medical images, which is often based on local image details, e.g. bone fractures or the detection of emphysema in computed tomography images (CT) of the lungs. 3.1
The content representations described in the previous section only use grayscale information, i.e. color images are converted into grayscale by using standard color weighting:
Y = 6969 ¢ R + 23434 ¢ G + 2365 ¢ B
32768 In general, however, color is the single most important discriminate feature type on stock-house media and the image corpus used for the interactive query task contains many photographs, color (7) (8) (9) (10) Content representation Tamura texture histogram, Jensen-Shannon divergence 32£32, CCF (9 £ 9 translation window) X£32, IDM (gradients, 5 £ 5 window, 3 £ 3 context) X£32, IDM (as above, 32x32 Euclid 500-NN as ¯lter) scans of teaching material, and microscopic imaging. Therefore, a basic color feature was employed to compute mean, variance and third order moments for each color channel red, green, and blue. This yields a 9-dimensional feature vector. Euclidean distance with equal weights for each color component is used to compute the distance between two vectors.
Some of the queries do not consist of a single example image, but use several images as a query pool Q: positive and negative examples. For such queries, a simple summation scheme is used to obtain an overall distance:
jQj d(Q; r) = X wi ¢ d0(qi; r); Q = [f(qi; wi)g; wi = i=1 i ½
¡1 : qi negative ex. (11) 4
All results were obtained non-interactively, i.e. without relevance feedback by a human user, and without using textual information for the interactive query task. 4.1
Table 1 shows the categorization results obtained for the 1,000 unknown images using single classi¯ers. As IDM is very expensive (see running times below), a serial combination with a faster, but more inaccurate classi¯er as a ¯lter was also tested. For this, Euclidian distance on 32 £ 32 scaled representations was used and only the 500 closest references were passed to IDM. This cuts computation time down to 1=18, as the costs for the ¯ltering step are neglectable.
To obtain the optimal empirical weighting coe±cients ¸ for the parallel combination, an exhaustive search would have been necessary. Instead, a more time-e±cient two-step process was employed: First, a combination of the two spatial representations was considered. Afterwards, a combination of the combined spatial representations with the Tamura texture feature was investigated. Both runs tested values for ¸ increasing at a stepsize of 0:1. The results for these two steps on the testing set, i.e. 1,000 images of the 9,000 original reference images, are shown in Tab. 2.
This results in the parameter con¯guration shown in Tab.3. When using this parameter set for the classi¯cation of the 1,000 images to be categorized, a categorization rate of 86.7% for the 1-NN is obtained.
Using the 57 prototypes obtained via (7) as a representation set, a dissimilarity-space representation for the reference set and the unknown set was computed. The dissimilarity representations were then compared using a Euclidian distance. In addition, not only the elements with minimum sum of distances were used, but also the ones with the best n; n = 2 : : : 5 elements per category. This yields 114, 171, 228, and 285 components for the representation vectors. For comparison, experiments were also done for a random pick of 1 : : : 5 elements per category, resulting in representation vectors of the same size. The results are shown in Fig.1.
Weights ¸IDM ¸CCF
Categorization rate k=1 k=5
¸IDM;CCF ¸Tamura
Categorization rate k=1 k=5 0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0 Since no ground truth for the automatical optimization of the parameters is available, only a short visual inspection was done and two runs were submitted. The results are listed in Tab.4. The result quality is measured by mean average precision (MAP).
These results are 19th and 14th place among 28 submitted runs in the \visual only, automatic" category of this task, reaching half the MAP of the leading competitor in this category. 4.3
Concerning running times, texture features by Tamura and CCF are ¯t for interactive use. By pre¯ltering with a computationally inexpensive distance measure, computation time can be severly reduced without sacri¯cing too much accuracy. In the experiments, pre-¯ltering clearly outperformed dissimilarity space approaches for both random prototype selection and 1centres. However, further evaluation of algorithms for prototype selection is necessary. The parallel combination of single classi¯ers proved very useful, as it improves the categorization results considerably and can also be performed as an easy post-processing step on the distance matrices. 5.1
The results obtained for the annotation task verify the results obtained on a smaller corpus using
leaving-one-out[
Note that the rather high weight ¸Tamura overemphasizes the role of the texture features in the experiments, as the actual improvement of the categorization rate is statistically insigni¯cant for the 1-NN. It marginally improves the quality of the next nearest neighbors a bit as seen in the results for the 5-NN, which produces slightly better results for interactive queries which list a set of nearest neighbors. While results were satisfactory for queries based on grayscale radiographs, other queries, especially from photograpy imaging, had rather poor results. It should also be noted that several queries demand a high level of image content understanding, as they aim at diagnosis-related information, which is often derived from local details in the image (see Tab.6).
The methods used in this work to describe the image content either preserve no spatial
information at all (texture features by Tamura) or capture it at very large scale, omitting local details
important for diagnosis-relevant questions. Using only the image information, such queries cannot
be processed with satisfactory quality of the results with a one-level approach. Refering to the
concepts described in [
This work is part of the IRMA project, which is funded by the German Research Foundation, grant Le 1108/4.