In this article, we describe the used technologies and experimental results for the medical retrieval task at ImageCLEF 2005. The topics of competition this year contain both semantic queries and visual queries. The content-based approach containing four image features and the text-based approach using word expansion are developed to accomplish the mission. The experimental results show that the text-based approach has higher precision rate than content-based approach. Further, the results of combining both the content-based and text-based approaches are better than those using only one of the approaches. We summarize that the consideration on the image of visual queries can provide more human semantic perception and improve the efficiency for medical image retrieval. ACM Categories and Subject Descriptors: Information Storage and Retrieval; Artificial Intelligence; Free Keywords: Medical Image retrieval; content based image retrieval;
In this paper, we present the research experience of the NCTU group at ImageCLEFmed 2005. The dataset of medical image retrieval task contains about 50,000 images totally from the Casimage, MIR, PEIR, and PathoPIC datasets. Each image of collection contains annotations in XML format. The majority of the annotations are in English but a significant number is also in French and German, with a few cases that do not contain any annotation at all. The queries of this task were formulated with example images and a short textual description explaining the research goal.
The participants were requested to accomplish the task in either fully automatic retrieval or retrieval with manual feedback. The used query information can consider the example images or the textual description only, or combine the images and textual together. The main purpose is to evaluate the retrieval of medical images from heterogeneous and multilingual document collections containing images as well as text.
For handling the medical image retrieval task, the NCTU group gives two primitives: content-based approach and text-based approach. The combination of the two approaches using similarity weight was also discussed. The content-based approach in this work uses four image features, Facade scale image feature, Gray Histogram layout, Coherence Moment and Color histogram, extracted from the images directly. The text-based approach processes the annotations based on the vector space model and word expansion using Wordnet[Miller95]. The mixed retrieval of visual and textual is done by combining the content-based approach and the text-approach with different weights adjusting.
In Section 2, the image features for the content-based approach are described. Section 3 illustrates the text-based approach that processes the multilingual annotations and translation. Section 4 describes our submissions at ImageCLEF 2005 and the ranking results. Here, we give an explanation and a discussion on our experimental results. Finally, section 5 provides concluding remarks and future direction are for medical image retrieval.
This section describes the features used in the paper for the ImageCLEF 2005 evaluation. In an image retrieval system, image features are extracted from pixels of image. For fast response time, image features used must be concise, and for precision, image features is used must contain meaningful information to represent the image itself. Image feature can retrieval the image in visual. In this paper we adopt several image features we have proposed [Cheng 04] which image features have good performance in medical image application. In designing the image features, to emphasize the contrast of an image and handle images with less illuminative influence, we normalize the value of a pixel before quantization. In [Cheng 04] we proposed a relative normalization method. First, we cluster a whole image into four clusters by the K-means clustering method [Han01]. We sort the four clusters ascendant according to their mean values. After clustering, we shift the mean of the first cluster to value 50 and the fourth cluster to 200; then, each pixel in a cluster is multiplied by a relative weight to normalize. Let mc1 be the mean value of cluster 1 and mc4 is the mean value of cluster 4. The normalization formula of pixel p(x,y) is defined in Eq. (1).
p( x, y)normal = ( p( x, y) − (mc1 − 50)) ×
200 (mc4 − mc1)
The pixel values of an image are trivial and straight-forward features. For computational efficiency, images are always scaled to a common small size and compared using the Euclidean distance. [Keysers04] has shown that in the optical character recognition and medical image retrieval based on facade image features have obtained excellent results. In this work we scale down an image into 8×8 pixels to form a 64 feature vectors as facade scale image feature.
Histogram [Swain91] is a prime image feature for image information retrieval. Histogram method is invariant in image rotation, it is easy to implement and have good result in color image indexing. Because radiotherapy medical image only consists of gray level, the spatial relationship becomes very important. Medical images always contain particular anatomic regions (lung, liver, head, and so on); therefore, similar images have similar spatial structures. We divide an image into nine sections and calculate their histogram respectively. After normalization, the gray values are quantized into 16 levels for computational efficiency.
In the gray histogram, the gray value may be quantized into several bins to improve the similarity between adjacent bins. We set an interval range δ to extend the similarity of each gray value. The histogram layout feature estimates the probability of each gray level that appears in a particular area. The probability equation is defined in Eq. (2), where δ is set to 10, where pj is a pixel of the image, and m is the total number of pixels. The gray histogram layout of an image has a total of 144 bins.
hc (I ) = j=1 i m [ p j − ∑ δ 2 , p j +
δ m δ 2 ] ∩ ci
(1) (2)
One of the problems to design an image representation is the semantic gap. The state-of-the-art technology still cannot reliably identify objects. The coherence moment feature attempts to describe the features from the human’s viewpoint in order to reduce the semantic gap.
We cluster an image into four classes by the K-means algorithm. After clustering an image into four classes, we calculate the number of pixels (COHκ), mean value of gray value (COHμ) and standard variance of gray value (COHρ) in each class. For each class, we group connected pixels into eight directions as an object. If an object is bigger than 5% of the whole image, we denote it as a big object; otherwise it is a small object. We count how many big objects (COHο) and small objects (COHν) are in each class, and use COHο and COHν as parts of image features.
Since we intend to know how the reciprocal effects among pixels, so we use the smooth method on the image. If the spatial distribution of pixels of two images is similar, they will also be similar after smoothing. If their spatial distributions are quite different, they may have a different result after smoothing. After smoothing, we cluster an image into four classes and calculate the number of big objects (COHτ) and small objects (COHω). Each pixel will be influenced by its neighboring pixels. Two close objects of the same class may be merged into one object. Then, we can analyze the variation between the two images before and after smoothing. The coherence moment of each class form a seven-feature vector, (COHκ, COHμ, COHρ, COHο, COHν, COHτ, COHω). The coherence moment of an image is a 56-feature vector that combines the coherence moments of the four classes.
Color histogram [Swain91] is a basic method and has good performance for representing image content. The color histogram method gathers statistics about the proportion of each color as the signature of an image. In our work, the colors of an image are represented in the HSV (Hue/ Saturation/ Value) space, which is believed closer to human perception than other models, such as RGB (Red/ Green/ Blue) or CMY (Cyan/ Magenta/ Yellow). We quantize the HSV space into 18 hues, 2 saturations, and 4 values, with additional 4 levels of gray values; as a result, there are a total of 148 (i.e., 18×2×4+4) bins. Let C (|C| = m) a set of colors (i.e., 148 bins), PI (QI) is represented as Eq. (3), which models the color histogram H(PI) (H(QI)) as a vector, in which each bucket hci counts the ratio of pixels of PI (QI) in color ci.
PI =< hc1 (PI ),..., hcm (PI ) > , QI =< hc1 (QI ),..., hcm (QI ) > (3) In many previous studies, each pixel is only assigned a single color. Consider the following situation: I1, I2 are two images, all pixels of I1 and I2 fall into ci and ci+1 respectively; I1 and I2 are indeed similar to each other, but the similarity computed by the color histogram will regard them as different images. To address the problem, we set an interval range δ to extend the color of each pixel and introduce the idea of a partial pixel as shown in Eq.(4),
∑ hci (PI ) = p∈PI |α p − β p |
δ | PI | Let ci-1, ci, and ci+1 stand for a color bin, a solid line indicate the boundary of ci, p is the value of a pixel, δ δ δ δ [ p − , p + ] denotes the interval range δ, the shadow part, [α p ,β p ] , is the intersection of [ p − , p + ] 2 2 2 2 and ci. The contributions of the pixel to ci and ci-1 are computed as |α p − β p | and | ( p −δ 2) −α p | , δ respectively. It is clear that a pixel has its contributions not only to ci but also to its neighboring bins. δ Using the modified color histogram, the similarity of two color images q and d is defined in Eq. (5): (4) (5) n ∑min(hi (q),hi (d )) SIMcolor(H(q), H(d)) = H(q) ∩ H(d) = i=1 | H(q) |
. n ∑hi (q) i=1
The medical image collection of the ImageCLEF 2005 evaluation contains gray and color images. In color images, users are usually attracted by the change of colors more than the positions of objects. Thus, the effective feature in query a color image is different from query a gray image. The image have an obvious feature is whether the image is color or gray value image. When the user queries an image by example, the system first determines whether the example is color or grayscale. We calculate the color histogram, if the four bins of gray values occupy more than 80% of the whole image, we decide that the query image is gray; otherwise it is color. If the input is a color image, then we set the weight parameter denoted by “C”; if the query image is detected gray valued image, we use the weight parameter denote by “G” as show in Table 1 and Table 2.
In the ImageCLEFmed collections of annotations are in English, French and German. The overall multilingual search process is show in Fig. 1. Given an initial query Q, the system performs the cross-language retrieval, and returns a set of relevant documents to user. We use the representation expressing a query as a vector in the vector space model [Salton83]. The Textual Vector Representation is defined as following. Let W (|W| = n) be the set of significant keywords in the corpus. For a document D, its textual vector representation (i.e., DT) is defined as Eq. (6),
Query
Q
Query expansion
translation
wti (DT ) = mtfati x,DtTf × log nNti tf ti ,DT max tf In Eq.(7),
stands for the normalized frequency of ti in DT, max tf is the maximum number of occurrences of any keyword in DT, N indicates the number of documents in the corpus, and nti denotes the number of documents in whose caption ti appears.
In the above, we introduce the textual vector representation for documents. As for q query Q, one problem is that since QT is given in English, it is necessary to translate QT into French and German, which is the languages used in the document collection.
A short query usually cannot cover as many useful search terms as possible because of the lack of sufficient words. We perform the query expansion process to add new terms to the original query. The additional search terms is taken from a thesaurus – WordNet [Miller95]. For each expansion English term, it is then translated into one or several corresponding French and German words by looking it up in a dictionary1.
Assume AfterExpansion(QT ) = {e1,..., eh} is the set of all English words obtained after query expansion and query translation, it is obvious that AfterExpansion(QT ) may contain a lot of words which are not correct translations or useful search terms. To resolve the translation ambiguity problem, we define word co-occurrence relationships to determine final query terms. If the co-occurrence frequency of ei and ej in the corpus is greater than a predefined threshold, both ei and ej are regarded as useful search terms.
So far, we have a set of search terms, AfterDisambiguity(QT ) , which is presented as Eq.(8), AfterDisambiguity(QT ) = {ei , e j | ei , e j ∈ AfterTranslation(QT )
& ei , e j havea significant co - occurrence}
After giving the definition of AfterDisambiguity(QT ) , for a query Q, its textual vector representation (i.e.,
QT =< wt1 (QT ),..., wtn (QT ) > where wti (QT ) is the weighting of a keyword ti in QT, which is measured as Eq. (10), wci (QT ) indicates whether there exists an e j ∈ AfterDisambiguity(QT ) .
In Eq. 1 http://www.freelang.net/ (7) (8) (9) images in whose caption ti appears.
⎪⎧ tfti ,QT × log N wti (QT ) = ⎨ ⎪⎩ max tf nti (10) (10), W is the set of significant keywords as defined before, stands for the normalized frequency of ti in
AfterDisambiguity(QT ) , maxtf is the maximum number of occurrences of any keyword in AfterDisambiguity(QT ) , N indicates the number of images in the corpus, and nti denotes the number of tfti ,QT max tf
The entire ImageCLEFmed library consists of multiple collections (e.g., Casimage, PEIR, MIR, PathoPIC). Each collection is organized into cases that represent a group of related images and annotations. Each case consists of a group of images and an optional annotation. Each image is part of a case and has optional associated annotations, which consist metadata (e.g., HEAL tagging), and/or a textual annotation.
In the practice for medical images retrieval system, the doctor query by semantic sentence that is more convenience. After textual query, the system retrieves related images by annotation for user to browse. The user can further more rank the images in visually. Combine the textual and visually features will help the user find the desire image more precise. In ImageCLEF 2005, medical image retrieval task contains 25 queries for evaluation, the queries mixed visual image and semantic textual to retrieve desire images. The visually queries use image example to find similar images, each topic contain at least one image example. The semantic textual queries allow user query by a sentence, which some semantic concept are hard to derive from images directly. The goal of this task is to examine how the visual feature can improve the query result.
All submissions of participants in this task were classified into automatic runs and manual runs. The automatic runs means that the system at the query process without human manual intervened. In the automatic category, the methods can be classified into three sub-categories: Text only, Visual only and Mixed retrieval (visual and textual) according to the feature used. The category “Text only” means that systems use textual feature only to retrieve relevant images. “Visual only” category means that systems only use visual image feature without combine textual annotation to retrieve similar images. Mixed retrieval means the systems combine the visual and textual feature to retrieve images.
In this task, we have submitted ten runs for the mixed retrieval of automatic runs and six runs for the visual only of automatic runs. In the content-based approach, we combine four proposed image features by weighted adjusting to retrieve related images. The weight of features we set at the system initial and do not have any further user intervention while query is processing. Table 1 lists the query result of visual only runs and the setting weight of four image features. Table 2 lists the result of mixed retrieval runs and the setting weight of image features and textual features. The different of each runs is the weighted setting of features. The query topics contain color and gray images. We first examine the queries image is color or gray image by color/gray feature. According to the image is color or gray set different weight for image features. In the Table 1, “C” denotes that query image is color image and “G” denotes that query image is gray image. We submit six runs for visual only category. The run, “nctu_visual_auto_a8”, has the better result in our experiment. The weight of each feature are set equal, it means that four image features have the same importance. The result also shows that visual only approach has a bottleneck because the query topics contain semantic queries. illustrate that combine the visual and textual feature will get better results than single features. Run8 assume that the significant of visual and textual feature are equal. Run9 emphasizes the weight of visual features and Run10 emphasizes the weight of textual features. The result shows that text-based approach is better than content-based approach, but the content-based approach can improve the textual result.
Table 2: The result of mixed retrieval runs and the weight of visual image features and textual features
Submission runs The weight of Image features
ImageCLEF 2005 medical image retrieval task offers a good test platform to evaluate the ability of image retrieval technologies. There are totally 112 runs submitted for this task. The results of the evaluation show that the method we proposed is excellent. Our best result rank by MAP is 3, there are one better system than us. In the experiment result, we find that content-based approach retrieving similar images rely on visual feature, which has less semantic expansion. The text-based approach has better performance than content-based approach. Combine the textual and visual features will get best result.
The results in the medical retrieval task show that weighted setting between the features is very important. The variation between different settings of weight is extreme. Suitable weight adjusting will improves the results. [Boser92] B. Boser, I. Guyon, and V. Vapnik A training algorithm for optimal margin classifiers. In Proceedings of the Fifth Annual Workshop on Computational Learning Theory, 1992. [Salton83] G. Salton and M. J. McGill, Introduction to Modern Information Retrieval. New York: McGraw-Hill, 1983.