In this paper we describe the methods we used in the 2005 ImageCLEF content-based image retrieval evaluation. For the medical retrieval task, we combined several low-level image features with textual information retrieval. Combining these two information sources, clear improvements over using one of these sources alone are possible. Additionally we participated in the automatic annotation task, where we used FIRE, our content-based image retrieval system, on the one hand and a subimage based method for object classification on the other hand. The results achieved are very good. In particular, we obtained the first and the third rank in the automatic annotation task out of 44 submissions from 12 groups.
content-based image retrieval, object recognition, textual information retrieval It is known that in content-based image retrieval (CBIR) benchmarking of systems is a major problem. ImageCLEF, as part of the Cross language evaluation forum, is one major step in the direction of creating standard benchmarking tasks and setting up competitions to compare content-based image retrieval systems. One of the main conclusions that can be drawn from the 2004 ImageCLEF image retrieval evaluation is that textual information and user feedback can strongly improve the results if available. Especially if the queries are of semantic nature it is intrinsically difficult to solve them using visual information alone.
Especially in real life applications, as e.g. in medicine, where textual information is available and pictures alone are not sufficient to describe a medical case, any information available should be used. If, for example, the query image is a microscopic photo of a bacteria culture, a standard image retrieval system will easily find other pictures of bacteria cultures, but it will hardly be able to distinguish between different kinds of bacteria. With additional, textual query information, like ”E. Coli bacteria”, the query, and thus the result, is more precise.
As we obtained the best score in the category “visual information only, no user interaction” in the 2004 ImageCLEF evaluation, it was an interesting challenge to extend our FIRE system1 towards using textual information.
Other groups already proposed their approaches of combining textual information retrieval and
content-based image retrieval, e.g. [
In this paper, we describe the techniques we used for the 2005 ImageCLEF evaluation, in particular how we combine textual information retrieval and content-based image retrieval.
In 2005 ImageCLEF, four tasks were available: a) automatic annotation, b) medical image retrieval, c) bilingual information retrieval, and d) interactive retrieval. We participated in the automatic annotation task and the medical image retrieval task. Our approach to the medical retrieval task is described in Section 2, the two approaches to the automatic annotation task are described in Section 3. 2
For the medical retrieval task in the 2005 ImageCLEF Image Retrieval Evaluation, 25 queries
were given. Each query was defined by a short textual query description and one to three example
images. One query contained a negative example image, all other example images were positive.
A more detailed description of the task and an overview on the results can be found in [
Given a set of positive example images Q+ and a (possibly empty) set of negative example images Q− a score S(Q+, Q−, X) is calculated for each image X from the database:
S(Q+, Q−, X) =
X S(q, X) + q∈Q+
X (1 − S(q, X)). q∈Q− where S(q, X) is the score of database image X with respect to query q and is calculated as S(q, X) = e−γD(q,X) with γ = 1.0. D(q, X) is a weighted sum of distances calculated as (1) (2)
M D(q, X) := X wm · dm(qm, Xm).
m=1 Here, qm and Xm are the mth feature of the query image q and the database image X, respectively. dm is the corresponding distance measure and wm is a weighting coefficient. For each dm, PX∈B dm(Qm, Xm) = 1 is enforced by re-normalization.
Given a query (Q+, Q−), the images are ranked according to descending score and the K images X with highest scores S(Q+, Q−, X) are returned by the retriever.
Due to the lack of suitable training data, weights wm were chosen heuristically based on experiences from earlier experiments with other data. 2.2
To incorporate textual information in FIRE, we decided to use an existing textual information
retrieval engine [
To incorporate textual information in FIRE, we decided to use an existing textual
information retrieval engine that was developed at Lehrstuhl fu¨r Informatik VI. The text retrieval engine
implements a variant of the Smart-2 retrieval metric, which is based on the well-known term
frequency inverse document frequency (tf-idf) metric. First, the textual information is preprocessed
by removing function words that are considered to be of no importance for the actual retrieval
process (so called stopping). The stop word list used for this purpose comprises 319 of the most
frequently occurring function words in the English language. Subsequent to the stopping process,
the remaining words are reduced to their stems using Porter’s stemming algorithm [
K
n(t)
stemmed words form the index terms that are used in order to index the text documents provided
with the image data. In our implementation of the Smart-2 retrieval metric we use the following
definition of the inverse document frequency:
Here, t denotes an index term, and K is the number of text documents. Due to the floor operation
in Eq. (3) a term weight will be zero if it occurs in more than half of the documents. According
to [
1 + log n(d) 0 n(d) = PPt∈T n(t, d) if t ∈ d if t ∈/ d with log 0 := 0 and g(t, d) :=
t∈T :n(t,d)>0 1 The logarithms in Eq. (4) prevent documents with high term frequencies from dominating those with low term frequencies. In order to obtain the final term weights, g(t, d) is divided by a linear combination between a pivot element c and the number of singletons n1(d) in document d: with λ = 0.2 and ω(t, d) :=
g(t, d) (1 − λ) · c + λ · n1(d) k=1
K c = 1 X n1(dk) and n1(d) :=
K
X
1 t∈T :n(t,d)=1 Unlike tf-idf, only query terms are weighted with the inverse document frequency idf(t): ω(t, q) = 1 + log n(t, q) · idf(t)
f (q, d) = X ω(t, q) · ω(t, d) The Smart-2 retrieval function is then defined as the product over the document and query specific index term weights:
t∈T
To use the textual information for image retrieval, each image has to be attached to at least one (possibly empty) text document. These text documents are used in the above described image retrieval process. To determine the distance dtext(qm, Xm) between a query image q with query text qm and a database image X with attached text Xm, first the textual information retriever is queried using the query text. Then, the textual information retriever returns the list of all relevant documents from the database. These documents are ranked by the retrieval status values (RSV) R which is is high for documents similar to the query and low for dissimilar documents. The distance d(qm, Xm) is then calculated as dtext(qm, Xm) = (
Rmax − RX ρ if X is in the list of relevant documents otherwise where Rmax is the maximum of all returned RSVs, RX is the RSV attached to image X, qm and Xm are the query text and the text attached to image X, respectively, and ρ is a constant chosen sufficiently large to make sure that images whose texts do not appear in the list of relevant objects have high distances. Note, the case where ρ = Rmax corresponds to assigning RSV of 0 to all non-relevant texts.
The resulting distances dtext(qm, Xm) are used in the retrieval process described in the previous section. (3) (4) (5) (6) (7) (8) (9) (10) 2.3
In the following we describe the image features we used in the evaluation, these features are extracted offline from all database images. 2.3.1
The most straight-forward approach is to directly use the pixel values of the images as features.
For example, the images might be scaled to a common size and compared using the Euclidean
distance. In optical character recognition and for medical data improved methods based on image
features usually obtain excellent results [
In this work, we used 32 × 32 versions of the images, these were compared using Euclidean distance. It has been observed, that for classification and retrieval of medical radiographs, this method saves as a not-top-bad baseline. 2.3.2
Color histograms are widely used in image retrieval [
In [
In [
A feature is called invariant with respect to certain transformations if it does not change when
these transformations are applied to the image. The transformations considered here are
translation, rotation, and scaling. In this work, invariant feature histograms as presented in [
Method B is a general object recognition method using histograms of image patches and
discriminative training of log-linear models [
The parameters of method A were optimized using 1,000 images from the 9,000 training images
as development set and the remaining 8,000 images for training. The parameters of method B
were chosen as they work best on the Caltech Database [
A more detailed description of the task and a detailed analysis of the results can be found
in [
Method A uses our CBIR system FIRE, and a subset of the above described features consisting of thumbnails of the images of the sizes 32×32 and X × 32 and Tamura Texture Histograms. Error rates for using these features alone are given in Table 1.
Some experiments with different weightings of Tamura features and thumbnails on our
development corpus have shown that using the image distortion model alone outperforms the
combinations. In particular the combination of image distortion model (weighted 5) and Tamura texture
features (weighted 2) is interesting, as this performed best in previous experiments on smaller
versions of the IRMA database [
For method B we used the object recognition and classification approach using histograms of image
patches and maximum Entropy training to classify the 1000 test images as described in [
To reduce the time and memory requirements for the clustering process, we used only 4000 images for estimating the Gaussian mixture model and created the histograms for the remaining 5000 training images using this mixture model. For the discriminative training of the log-linear model, we used all training histograms.
The model submitted used multi-scale features where the first PCA component was discarded to
account for brightness changes and 4096-dimensional histograms. This combination was reported
to work best on the Caltech database [
In the following we describe the exact setup of the submitted runs to the automatic annotation task and the medical retrieval task and discuss the results. Furthermore, we discuss our methods, points to errors we made, and present results with errors corrected. 4.1
Our submission using model A obtained the first rank in the automatic annotation task. The submission following the object recognition approach obtained the third rank. In total, 44 runs were submitted by 12 groups. The second rank was obtained by the IRMA group2 using a similar approach to our model A and the fourth rank was obtained by University of Li`ege, Belgium using an approach with image patches and boosted decision trees. A clear improvement over the baseline result of 36.8% error rate cam be observed. This baseline results is obtained by a nearest neighbor classifier using 32x32 thumbnails of the images and Euclidean distance. 4.2
For the medical retrieval task, we used the features described in Section 2.3 with different weightings in combination with text features. In total, we submitted 10 runs which are shortly described here.
Runs using textual information only: We submitted two full-automatic runs, where only textual information was used. These runs were labelled En and EnDeFr. In En only the English texts were used, for EnDeFr the English, the German, and the French texts were used and combined with equal weights.
Runs using visual information only: We submitted three full-automatic runs, where only visual information was used. The runs 5000215, 0010003, and 1010111 only differ in the weighting of the image features. The exact weightings can be seen in Table 2. The run labelled 5000215 uses exactly the same setting as our submission to the 2004 ImageCLEF evaluation which had the best score from all 23 submissions in the category “visual features only, no user interaction”. From the bad score of 0.06, it can be seen that this years tasks differ strongly from last year’s task.
two runs with relevance feedback where textual and visual information was used. For the run i6-3010210111, the features were combined in exactly the way described above. For the runs i6-3(1010111-min(111)) and i6-3(3030333)-min(111) before combining the textand the visual features, first the minimum distance of all three text distances was taken for each image, to better account for images that have texts in one language only. The runs i6-vistex-rfb1 and i6-vistex-rfb2 used relevance feedback from the first 20 results of the automatic run i6-3(1010111-min(111)) and differ only in the user feedback. In both cases the feedback was given by a computer scientist familiar to the FIRE system with little background in medicine. Furthermore, the textual information was not available for the user feedback, thus the feedback is based on visual information only. In this paper we presented the methods we used in the 2005 ImageCLEF CBIR evaluation. We participated in the automatic annotation task, where we obtained the first and the third rank, and we participated in the medical image retrieval task, where our results were quite bad due to wrong settings, results with correct settings are presented in this work, and it can be seen that the method of combining textual information retrieval and content-based image retrieval performs very well. In particular, the result obtained would have been ranked 3rd in the medical retrieval task in the category “full automatic runs using textual and visual information”.