This paper describes the participation of INAOEs research group on machine learning, image processing and information retrieval (TIA) in the photographic retrieval task at ImageCLEF2007. Many experiments were performed comprising most of the query and target languages proposed for this years competition. A web-based query expansion technique was proposed for introducing context terms in the query topics. An intermedia blind relevance feedback technique was adopted in some runs for query refinement. Furthermore, experiments were performed with a novel technique for query/document expansion based on automatic image annotation methods. Initial experimental results give evidence that this idea could help to improve effectiveness of retrieval methods, though several issues should be addressed.
Technology innovations have provided humans with more information than we are able to manually analyze.
Currently, many huge repositories with textual, visual and auditive information are available. In addition
there are collections of mixed data, for example images accompanied with textual descriptions. These last
type of collections are referred to as annotated collections (one data type, text, is the annotation of another
data type, images). Since documents in annotated collections are described in two modalities, document
retrieval from these collections is a more challenging task than that of traditional information retrieval. This
due to the problem of the lack of correspondence between modalities (the semantic gap) [
Current solutions for image retrieval from annotated collections are based on a single modality (text or
image). Though retrieval accuracy of such systems is limited, mainly, because of the omitted modality and
the querying complexity. In consequence there exist an increasing interest on the development of methods
that can take advantage of all of the available information, that is text + image. In this research line, an effort
is being carried out by the organizers of ImageCLEF1; the cross-language image retrieval track of the CLEF 2
forum, whose goal is to investigate the effectiveness of combining text and image for retrieval, to collect and
provide resources for benchmarking image retrieval systems and to promote the exchange of ideas which will
lead to improvements in the performance of retrieval systems. Organizers provide participants with image
collections, query topics and relevance assessments of retrieval systems’ runs [
This paper describes the participation of the INAOE’s research group on machine learning, image processing and information retrieval (TIA) in the photographic retrieval competition at ImageCLEF2007. This is the first time TIA participates on ImageCLEF. A total of 95 runs were submitted comprising all of the target languages and the following query ones: English, Spanish, German, Italian, French, Swedish, Japanese, Portuguese, Russian and Visual. Experiments were carried out with different techniques, like intermedia feedback and fusion of independent retrieval models. Other experiments were carried out with an approach based on web query expansion and with a new method based on automatic image annotation. We propose a web-based technique for expanding ImageCELF2007 queries, with which promising results were obtained. Furthermore, we propose a new research direction not explored yet for image retrieval from annotated collections. The approach consist of using automatic image annotation methods for obtaining text from the visual content of the images. This text is then combined with the original annotations of the images (and/or with the queries) and then standard strategies are adopted for retrieving documents. Experimental results show that runs based on text and image improve those based on text only. The best results were obtained with a combination of intermedia feedback and our Web-based query expansion technique. Relatively good results were obtained with the annotation based expansion. However, several issues should be addressed in order to obtain better results with this technique.
The rest of the document is organized as follows. In the next section we briefly introduce the ad-hoc
photographic retrieval task, for further information we refer the reader to the overview paper by Grubinger
et al [
The ImageCLEF2007 is a track running as part of the CLEF campaign. It comprises four tasks related
to image retrieval, namely: ad-hoc photographic retrieval, object retrieval, medical image retrieval and
automatic annotation for medical images. This paper presents developments and contributions for the first
task, that of retrieving images from a collection of annotated photographs. The goal of this task is the
following: given a multilingual statement describing an user information need, find as many relevant images
as possible from the given document collection [
1http://ir.shef.ac.uk/imageclef/2007/ 2http://clef-campaign.org/
The predominant approach for image retrieval from annotated collections consist of using text-only retrieval methods. This mainly because in some collections annotations describe effectively the visual and semantic content of images, for example the collection used for ImageCLEF2006. However in realistic collections images are described by only a few words indicating the semantic content of each image, for example the collection used for ImageCLEF2007. In these scenario information extracted from the images could be helpful for improving retrieval performance. For this reason we performed experiments with methods based on both text and images, text-only runs were also submitted for measuring the gain we have by using visual information. Independently of the type of run all of our submissions used a textual retrieval system (for the mixed runs text was extracted from images and used with the textual retrieval system), which is described in Section 3.1. Furthermore, in Section 3.2 the proposed technique for expansion of textual queries is presented. 3.1
Since text was used in all of the TIA’s runs a textual retrieval system was implemented. For this purposed
the TMG MatlabR toolbox was used, kindly provided by Zeimpekis et al [
After removing meta-data and useless information, the text of the captions was indexed on the four target
languages (English, Spanish, German and Random), resulting on four indexed collections. For indexing we
used a tf-idf weighting, English stop words were removed and standard stemming was applied [
The Web is the largest repository of information ever existed. There are millions of documents available in the Web that can be used to extract semantic knowledge. In this work we propose a web-based query expansion technique in order to incorporate context terms into the original queries, with the hope that expanded queries are able to reach (through a retrieval model) relevant documents containing other terms than the ones contained in the original queries.
The proposed approach is a very intuitive method that uses the GoogleR search engine. For each topic, we take the textual description and submitted a search to GoogleR; the top−k snippets returned by the search engine are considered for expanding a query. We tried two approaches that we called naive and repetition. The naive approach consist of taking the snippets as they are returned by GoogleR with no preprocessing. On the other hand, the repetition approach consist of retaining the terms that most co-occurred among the snippets. For the experiments reported here the top−20 snippets were considered. Queries expanded with the naive method resulted on larger queries, while the ones expanded with the repetition method included only a few extra terms. 4
In order to consider both textual and visual information in the retrieval process, two strategies were adopted:
intermedia feedback and late fusion of independent retrieval systems. We decided to use these techniques
due to their simplicity, and because both methods have obtained relatively good results on the retrieval task
[
Relevance feedback is a way of allowing user interaction in the retrieval process. It was first proposed for
retrieval from text collections and then used on the content-based image retrieval (CBIR) task [
A novel technique based on PRF has been proposed for image retrieval from annotated collections [
For some runs we adopted a late fusion strategy for merging the output of independent visual and textual systems. This merging method consists of running two independent retrieval systems using a single (different) modality each. Then the relevant documents returned by both systems are combined.
For this work we proposed the following fusion strategy. Assuming that TR is the list of relevant documents (ranked in descending order of relevance) according to a textual retrieval system applied to the images’ captions; similarly, VR is the list of ranked relevant documents according the a CBIR system that only uses the images. We combined and re-ranked the documents returned by both retrieval systems, generating a new list of relevant documents LFR = {TR ∪ VR}; where each document di ∈ LFR is ranked according to the score formula given by Equation (1) score(di) = α × RTR (di) + (1 − α) × RVR (di) 1TR(di) + 1VR(di) (1) where RTR (di) and RVR (di) is the position in the ranked list of document di according to the textual and visual retrieval systems, respectively. 1TR(di) and 1TR(di) are indicator functions that take the value 1 if document di is in the list of relevant documents according to the textual and visual retrieval systems respectively, and zero otherwise. The denominator accounts for documents appearing in both lists of relevant documents (TR and VR). Documents are sorted in ascending order of their score. Intuitively with this score documents appearing in both sets (visual and textual) will appear at the top of the ranking, regarding their position on the independent lists of relevant documents. We tried several values for α and the best results were obtained with α = 0.9. 5
The task of automatic image annotation (AIA) consists of assigning textual descriptors (words, labels) to images, starting from visual attributes extracted from them. There are two ways of facing this problem, at image level and at region level. In the first case, labels are assigned to the entire image as an unit, not specifying which words are related to which objects within the image. In the second approach, which can 4Note that the process can start from text, obtaining query images for a CBIR system, as well. be conceived as an object recognition task, the assignment of labels is at region level; providing a one-to-one correspondence between words and regions5. The last approach can provide more semantic information for the retrieval task, although it is more challenging than the former.
In this work we decided to use region-level AIA methods for obtaining text from images and then using these textual labels for expanding topics and images’ annotations. We believe that by taking both modalities to a common representation and then using standard retrieval strategies can be helpful for improving singlemodality approaches. A graphical description of the proposed approach is shown in Figure 1. As we can see the process involves several steps that can be summarized in the following tasks: segmentation, annotation, and expansion. 5.1
The first step for obtaining words from regions within images is obtaining regions from images. This task is
known as segmentation and consists of discovering partitions within a given image with the restriction that
each of the partitions contains a single object. Many segmentation methods have been proposed, although
this is still an open problem in vision. For segmenting the IAPR-TC 12 Benchmark collection we decided
to use a state of the art algorithm named normalized cuts [
Using a recently developed segmentation-annotation interface [
5Note that with region-level annotation we have also an annotation at image level, which is the set of labels assigned to the regions within the image.
5 Clouds
17 Vehicle
6 Grass 18 Boat
AIA methods start from visual attributes extracted from the image or regions and assign labels based on
some training examples. Each training example is a pair composed of a region and its corresponding label.
In order to create a training set of region-label pairs an annotation interface was used [
The set of labels that can be assigned to regions (that is, the annotation vocabulary) was defined subjectively by the authors, by looking at the ImageCLEF2007 textual topic descriptions. The vocabulary of allowed words is shown in Table 1, and the number of regions, in our training set, annotated with each label is shown in Figure 3. Some labels are supposed to represent several concepts, for example the label water was used to label regions of rivers, ocean, sea, and streams. While other labels represent specific objects, such as swimming-pool and tower. From Figure 3 we can see that there are several labels that have many training examples (for example, Sky, Person), though several other labels have only a few. This fact together with poor segmentation made difficult the process of annotation. Sample images with their corresponding annotations are shown in Figure 2.
The training set of region-label pairs is used with a knn classifier for annotating the rest of un-annotated
images-regions on the IAPR-TC 12 Benchmark collection for document expansion, or the topic’s images
for query expansion. Note that the training set size is very small6 for achieving good results with the
knn algorithm; even when knn have obtained good results on this task, outperforming other state of the art
annotation methods [
Candidate labels and confidence values are obtained from knn, the annotation method. Semantic
association between labels is obtained by measuring co-occurrences of labels at document level in an external
6The training set was created for applying a recently proposed semi-supervised learning algorithm based on unlabeled data
[
For document expansion the annotations assigned to each region in each image in the IAPR-TC 12 Benchmark are added to the original annotation. For query expansion the sample topic images were segmented and annotated, then the set of annotations was used for expanding (or creating) textual queries, in Figure 5 an expanded topic is shown. As we can see some labels are repeated on the resulting expanded topic (sky, people and tree); we decided to consider repeated labels because in this way repeated terms are considered representative terms of the expanded query. If we would not considered label’s repetitions the word people, for example, would be considered equally important (for representing the query) if it appeared as annotation of an unique region in a single image or as 4 annotations of 4 different regions within the three topic’s images. 6
A total of 95 runs were submitted to the ImageCLEF2007, comprising all of the above described methods: intermedia feedback (IMFB ), naive web-based query expansion (NQE), repetition web-based query expansion (WBQE ), late fusion of independent retrievers (LF ) and annotation based expansion (ABE ). Note that some runs are combination of these methods. The top ranked entries for each language configuration together with a description of the methods used are shown in Table 2.
As we can see, most of the entries are ranked near the top one. The best performance overall runs were obtained by using IMFB together with NQE s. We can observe that the runs with IMFB+NQE for target language English have exactly the same MAP value, independently of the query language. This is due to the fact that all of the queries were translated to English, then the snippets returned by GoogleR were used for expanding the translated query. Snippets were similar because the English search engine was used, and translations were very similar for all the query languages. The expanded queries were then combined with the captions of the top−10 relevant documents according to the baseline FIRE run, which were the same for all the query languages.
Other entries ranked high were those using NQE. Actually the NQE is present in all of the top ranked runs. NQE outperformed WBQE in all of the language configurations. This can be graphically appreciated in Figure 6 (left). This is a surprising result because with the naive approach several noisy terms are added to the queries. While with the repetition approach only the terms that most appear among all the snippets are added. Therefore, the good results of NQE are due to the inclusion of many highly related terms, while the insertion of some noisy terms does not affect the performance of the retrieval model. From Figure 6 (left) we can also clearly appreciate that IMFB outperformed the LF approach.
Six runs based on annotation expansion were submitted to the ImageCLEF2007. In these runs document and query expansion was combined with the other techniques proposed in previous sections. The descriptions of the annotation based expansion (ABE ) runs submitted to ImageCLEF2007 are shown in Table 3, together with their overall ranking position. Results with ABE are mixed. The two top ranked runs with ABE
ID
INAOE-EN-EN-AN-EXP-5
INAOE-VISUAL-EN-AN-EXP-3 INAOE-EN-EN-NaiveWBQE-ANN-EXP-LF-2
INAOE-EN-EN-AN-EXP-2 INAOE-EN-EN-DQE-ANN-EXP-LF
INAOE-EN-EN-AN-EXP-1
Methods Baseline,IMFB Baseline,IMFB
NQE,LF, NQE,Baseline
NQE,LF Baseline
ABQE
X X X X X X
ABDE X X X X
We have described the first participation of INAOE’s TIA research group at the ImageCLEF2007 photographic retrieval task. A total of 95 runs were submitted comprising most of the query languages and all of the target ones. Experiments were performed with two widely used approaches on image retrieval from annotated collections: intermedia feedback and late fusion of independent retrievers. A Web-based query expansion technique was proposed for introducing context into the topics and a novel annotation based expansion technique was also developed. The best results were obtained with the intermedia feedback combined with a version of the web-based query expansion technique. With the naive expansion technique several relevant related terms are added to the original query, as well as many noisy terms. However noisy terms do not affected the performance of the retrieval system. The intermedia feedback technique outperformed late fusion by a large margin.
Relatively good results were obtained with the annotation based runs, however this can be due to the techniques combined with the ABE methods. Results with this technique give evidence that automatic image annotation methods could be helpful on image retrieval from annotated collections. This because promising results were obtained even when segmentation was poor, the training set was extremely small and imbalanced, and annotations did not covered the objects present within the image collection. In consequence for future work we will address all of these issues by creating a larger training set of annotated images, using other segmentation algorithms, defining labels objectively trying to cover most of the objects present in images within the collection and keeping balanced the training set. Also we intent to use other annotation methods.
Acknowledgements. We would like to thank M. Grubinger, A. Harbury, T. Desealers, P. Clough and all other organizers of ImageCLEF2007 because of their important effort. This work was partially supported by CONACyT under grant 205834.