This paper presents the work of the MMIS group at ImageCLEF 2008. The results for three tasks are presented: Visual Concept Detection Task (VCDT), ImageCLEFphoto and ImageCLEFwiki. We combine image annotations, CBIR, textual relevance and a geographic filter using our generic data fusion method. We also compare methods for BRF and clustering. Our top performing method in the VCDT enhances supervised learning by modifying probabilities based on a matrix that shows how terms appear together. Although it occurred in the top quartile of submitted runs, the enhancement did not provide a statistically significant improvement. In the ImageCLEFphoto task we demonstrate that evidence from image retrieval can provide a contribution to retrieval; however we are yet to find a way of combining text and image evidence in a way to provide an improvement over the baseline. Due to the relative performances of difference evidences in ImageCLEFwiki and our failure to improve over a baseline we conclude that text is the dominant feature in this collection.
In this paper we describe the experiments of the MMIS group at ImageCLEF’08. We participated in three tasks: Visual Concept Detection Task (VCDT), ImageCLEFphoto and ImageCLEFwiki.
All experiments were performed in a single framework of independently testable and tuneable modules. The framework is described in detail in Section 2. The experiments conducted and individual runs submitted for each task are described in Sections 3, 4 and 5. Finally we present our conclusions in Section 6. 2
Our system framework is shown in Figure 1. The text elements of the corpus are indexed as a bag-of-words, analysed geographically and stored in a geographic index. Texture and colour features are extracted from images to form feature indexes, these features are further analysed to automatically annotate the images. This allows us to compare query images to our index using both semantic annotations and low-level features.
Blind Relevance Feedback (BRF) is employed across media types similarly to [
Our intermediate format is the standard TREC format. This allows us to evaluate and tune each module independently. The results of all the independent modules are combined in the data fusion module. The data fusion module combines both the ranks provided by the image and text query engine, and the filters provided by the geographic and annotation query engines. The difference between a rank and a filter is all elements in a filter are considered of equal relevance.
In the ImageCLEFphoto task we are evaluated on the novelty of our top results and provided with a subject which this novelty will be judged with respect to. We have clustered our top results using the geographic index, image annotations and text index.
Further details on the individual modules and tuning are described in the following sections. 2.1
Content-Based Image Retrieval (CBIR) provides a way to browse or search images from large image collections based on visual similarity. CBIR is normally performed by computing the dissimilarity between the object images and query images based on their multidimensional representations in content feature spaces, for example, colour, texture and structure. In this section, we are going to introduce the key issues of the image search applied to our three tasks. 2.1.1
ImageCLEFphoto Task. Colour feature is the most commonly used visual feature e.g. HSV,
RGB [
Figure 1: Our Application Framework VCDT. The features used in the VCDT experiments are a combination of colour feature feature, CIELAB, and texture feature, Tamura.
CIE L ∗ a ∗ b∗ (CIELAB) [
The Tamura texture feature is computed using three main texture features called “contrast”,
“coarseness”, and “directionality”. Contrast aims to capture the dynamic range of grey levels in
an image. Coarseness has a direct relationship to scale and repetition rates and it was considered
by Tamura et al. [
The process for extracting each feature is as follows, each image is divided into nine equal rectangular tiles, the mean and second central moment feature per channel are calculated in each tile. The resulting feature vector is obtained after concatenating all the vectors extracted in each tile. 2.1.2
ImageCLEFphoto Task. The χ2 statistic is a statistical measure that compares two objects in a distributed manner and basically assumes that the feature vector elements are samples. The dissimilarity measure is given by dχ2 (A, B) =
Xn (ai − mi)2
,
mi
mi =
i=1
ai + bi
2
where A = (a1, a2, ..., an) and B = (b1, b2, ..., bn) are the query vector and test object vector
respectively. It measures the difference of the query vector (observed distribution) from the mean
of both vectors (expected distribution)[
n dp(A, B) = (X |ai − bi|p) p1 ,
i=1 ImageCLEFphoto Task. In the ImageCLEFphoto task, every query topic includes three independent example images a, b, c. The probability of an object image x being a relevant match for a query topic is determined by the joint probability of the relevance between the images a, b, c in the query topic and the object image x. According to probability theory, the joint result is given by
D(abc, x) = d(a, x) × d(b, x) × d(c, x), where D(abc, x) is the distance between a query topic including images a, b and c with an image x. d(a, x), d(b, x) and d(c, x) are distances between the three images a, b, c and the image x, respectively. (1) (2) (3) (4) 2.1.4
We employed Blind Relevance Feedback (BRF) for CBIR in a similar fashion to [
• The first BRF method takes the top seven results from the text retrieval results as new query examples, and the seven ranked result are combined by the search method introduced in Section 2.1.3. • The second BRF method was employed in both the ImageCLEFphoto and ImageCLEFwiki tasks. The final ranked result of this BRF is the sum of the ranks of the top five examples from the text result with equal weights. The detail of this combining method are described in Section 2.6. 2.2
The Image Annotator is the core part of the Visual Concept Detection Task (VCDT) which aims to create a model able to detect the presence or absence of 17 visual concepts in the images of the collection. The input is a training set of 1825 images that have already been annotated with words coming from a vocabulary of 17 visual concepts. The output is the annotations.
We use as a baseline for this module the framework developed by Yavlinsky et al. [
The process can be described as follows. First, images are segmented into nine equal tiles, and then, low-level features are extracted. The features used to model the visual concept densities are a combination of colour CIELAB and texture Tamura, as explained in Section 2.1.
The next step is to extract the same feature information from an unseen picture in order to compare it with all the previously created models (one for each concept). The result of this comparison yields a probability value of each concept being present in each image.
Then we modify some of these probabilities using additional knowledge from the image context in order to improve the accuracy of the final annotations.
The context of the images is computed using a co-occurrence matrix where each cell represents the number of times two visual concepts appear together annotating an image of the training set.
The underlying idea of this algorithm is to detect incoherence between words with the help of this correlation matrix. Once incoherence between words has been detected, the probability of the word associated to the lowest probability will be lowered, as well as all the words which are semantically similar.
Among the many uses of the concept “semantic similarity,” we refer to the definition by Miller
and Charles [
Thus, by modifying the probability values of some concepts, annotations are produced by selecting the concepts with the highest probability. The output of this module, the annotations, will be the input for the Annotation Query Engine and the Document Cluster Generator. 2.2.1
The input for this module is textual queries which are the concepts of our vocabulary and the annotations produced by the Image Annotator module. Given a query term, we represent the top n images annotated by it following the standard TREC format. This constitutes one of the inputs for the Data Fusion module. Retrieval performance is evaluated with the mean-average precision (MAP) on the whole vocabulary of terms, which is the average precision, over all queries, at the ranks where recall changes (where relevant items occur). 2.3
Our text retrieval system is based on Apache Lucene [
We use only the English meta-data and queries (monolingual retrieval). In ImageCLEFphoto both the title and location fields are searched as text. We do not use the notes field as in previous training experiments we have found this gives worse results. 2.4
We process the text fields with Sheffield University’s General Architecture for Text
Engineering (GATE) [
In ImageCLEFwiki we compare two different disambiguation methods, both based on a
geographic co-occurrence model mined from Wikipedia [
The second method (Neighbourhoods), builds neighbourhoods of trigger words from Wikipedia. Depending which trigger words occur in the vicinity of an ambiguous placename dictates which location it will be disambiguated as. For example if Oxford occurs in the context of Cambridge, Cambridge will be disambiguated as Cambridge, Cambridgeshire as Oxford is a trigger word of Cambridge, Cambridgeshire.
As the ImageCLEFphoto corpus contains minimal referrences to ambiguous placenames we only use the MR method.
To query our geographic index we extract locations from the query and based on the topological data contained in the TGN return a filter of all the documents mentioning either the query locations or locations within the query locations. For example a query location of the “United States” will return all the documents mentioning the United States (or synonyms such as “USA”, “the States” etc.) and all documents mentioning states, counties, cities and towns within the United States. 2.5
We only employ clustering in ImageCLEFphoto. We propose a simple method of re-ordering
the top of our rank based on document annotations. We consider three sources of annotations:
Automated annotations assigned to images, words matched to WordNet and locations extracted
from text (described in Section 2.4). WordNet is a freely available semantic lexicon of 155,287
words mapping to 117,659 semantic senses [
In Image clustering all the images have been annotated with concepts coming from the Corel
ontology. This ontology was created using SUMO (Suggested Upper Merged Ontology) [
For example, if the cluster topic is “animal” we will split the ranked list of results into sub ranked lists, one corresponding to every type of animal and an additional uncategorised rank. These ranks will then be merged to form a new rank, where all the documents at rank 1 in a sub list appear first, followed by the documents at rank 2, followed by rank 3 etc. The documents of equal rank in the sublists are ranked amongst themselves based on their rank in the original list. This way the document at rank 1 in the original list remains at rank 1 in the re-ranked list. We only maximise the diversity of the top 20 documents, after the 20th document the other documents maintain their ordering in the original list.
Similarly in WordNet clustering we build clusters of sub-categories of animal, bird, sport, vehicle and weather. We match these to the bag-of-words for each document contained in the text index. For example if the cluster topic is sport we will have a sub-ranked list containing every document that mentions “tennis”. If for image clustering the cluster topic is not animal , vehicle or weather, or for WordNet clustering the topic is not animal, bird, sport, vehicle or weather, we default to location clustering. In this case we have a sublist for every different location a document is annotated with. 2.6
With multiple sources of evidence being provided by the different query engines, a method of combining these data was required. Each query engine produced an output rank of data set images ordered with respect to their relevance to the input query. Not all engines gave values for the entire data set—some gave ranks of relevant sub-sets of the main data set.
We adopted a group consensus function based on the Borda Count method [
Data were split into two categories which were treated differently; ranks and filters. Ranks were processed without adjustment and directly compared to each other during the combination process. Filters were used to filter out non-relevant results from an intermediate combined rank stage. This was used, for example, with the output of the geographic query engine, where it was judged more appropriate to ‘filter in’ explicitly determined relevant results than to treat it like a rank.
The final output of the data fusion algorithm was the combined result of the multiple input ranks and filters. where n is the number of pure ranks to combine.
This function then defined our parameter space within which we had to search to find the more appropriate weights for the rank combination process. We performed a brute force search of the entire parameter space to find the optimum weights. We used past years’ data for training and optimum sets of weights were considered those that gave the maximum mean average precision (MAP). The final weights are given in Table 1 for the two tasks that used rank combination; the ImageCLEFwiki and the ImageCLEFphoto tasks.
The parameter values used by the filter stage were also important. Unlike the pure rank weights that were used to multiply an entire rank, the filters were used by penalising those values that were not present in the filter. These penalisation weights pi, making up weight vector P , were subject to the same constraints and were derived in a similar way to the pure rank weights by exhaustively searching all possible values up to a limit which was defined as when the variation in the parameter yielded no change in the final rank’s MAP value greater than 0.0001.
The overall process is described here as a five stage algorithm: 1. Read in rank data
Data was provided by the query engines in ordered ranked lists with those elements that were judged to be most relevant by the engine appearing first, with each rank denoted Ri. Each of the m elements was assigned a value based on its position in the list, so the first element was given ‘1’, and increased consecutively down the list until the last element had a value of m. 2. Multiply by rank weights
Each query engine output list was scaled by multiplying every rank value in the list by its parameter value wi. 3. Sum ranks
The scaled ranks were then combined by producing a new rank R0 where every element r in each Ri was replaced by value r0. Stages 2 and 3 can be summarised thus: (5) (6) r0 = m X wiri i=1 where m is the number of ranks to combine and wi is an element of the weight vector W . 4. Filter rank
The newly combined intermediary rank R0 was then subjected to the output of each filter engine to produce rank R00. For each element in the rank R0, any element that was not found in the filter data had its rank value penalised by that rank’s parameter value as described by filter function f (r). This pushed less relevant elements further down the ordered rank. f (r0i) = ( r0i r0i present in filter r0ipi r0i not present in filter (7) 5. Sort rank
The filtered rank R00 was then sorted to ensure that the rank values were in ascending order.
The sorted rank is then the output of the combination stage of the overall system. 3 3.1
The objective of the Visual Concept Detection Task (VCDT) is to detect the presence or absence of 17 visual concepts in the 1000 images that constitute the test set. In addition to that, some confidence scores are provided once an object is detected. The higher the value the greater the confidence of the presence of the object in the image.
For the VCDT task, we submitted four different algorithms, all of them correspond to automatic runs dealing with visual information. The second run uses statistical information about visual concepts co-occurring together in addition to the visual information, and the final run is a combination of the other three. 3.1.1
This algorithm corresponds to the work carried out by Yavlinsky et al. [
This second algorithm is described in detail in Section 2.2. The submitted run is based on an enhanced version of the algorithm described in the previous section. The input is the annotations achieved by the algorithm developed by Yavlinsky et al. together with a matrix that represents the probabilities of all the words of the vocabulary being present in the images. This algorithm was also tested on the Corel collection of 5,000 pictures and a vocabulary of 374 words obtaining statistical significant results (5%). 3.1.3
The third algorithm follows a simple approach based on dissimilarity measures. Given one test image, we compute its global distance (Cityblock distance) to the mean of the training images which share one common keyword. The smaller the distance value the higher the probability for a test image to be annotated by the keyword of that category. Our submitted result is based on the combination of two results from two single feature spaces. One is the CIELAB colour feature. For each pixel, we compute the CIELAB colour values. Within each tile the mean and the second moment are computed for each channel. The other is the Tamura texture feature. The combination here is a simple additive combination of the probability for each test image for each category.
Algorithm Enhanced automated image annotation Automated image annotation Combined algorithm Dissimilarity measures algorithm This submitted run is based on the combination of all the other runs submitted by this team, in addition to one extra combined result formed from the output of a feature extraction algorithm for the Tamura and CIELAB features. This was a simple additive combination which, through testing was shown to be useful when used in combination with the other algorithm outputs. By testing the output of each individual algorithm on subsets of the training data they were produced with, a rough indication of performance of the algorithm per concept was derived. These figures then allowed an automated system to pick for each concept the best performing algorithm’s output and combine it into a new result set. The individual algorithms’ outputs were not adjusted or scaled in any way (other than the pre-combined result set mentioned above).
In our testing routines based on splitting the available training data into training and
evaluation sets, the combination performed marginally better than the individual component algorithm
outputs. This was due to selecting those algorithms which performed better at certain concepts
and classes of concepts. Further work will be carried out to more robustly take advantage of
concept classification when combining algorithm results.
The evaluation metric followed by the ImageCLEF organisation is based on ROC curves. Initially,
a receiver operating characteristic (ROC) curve was used in signal detection theory to plot the
sensitivity versus (1 - specificity) for a binary classifier as its discrimination threshold is varied.
Later on, ROC curves [
Our best algorithm was previously tested with the Corel dataset, a collection of 5,000 images
and 374 terms obtaining a statistically significant (5%) improvement over the baseline approach
followed by Yavlinksy et al. [
In this section, we describe the nine submitted runs. Txt. Our text only run uses Lucene as a base with TF•IDF term weighting and comparing queries to documents using the vector space model. Stemming is performed using the snow ball stemmer. Stop words are removed and text is lowercased and stripped of diacritics. Img. This run is a pure CBIR. The dissimilarity value between an image example and an object image is computed by χ2 Statistics measure based on their HSV colour feature space. Probability theory is adapted for combining the three dissimilarity values of the image examples in each topic with an object image. The final result is the Mean Average Precision (MAP) of 39 query topics. ImgBrfWeightingIterative. This run is a combination of image evidence and text evidence. BRF is employed in this run. The top seven examples from ranked text results of iteration one were taken as new image examples for each query topic. The same search method as with “Img” is used in this run. The final result is the combination of the result ranks of this run and the ranks of “Img” using weight 0.6 and 0.4.
ImgBrfWeighting. The differences between this run and “ImgBrfWeightingIterative” is that the BRF rank for this run was produced by summing the ranks of the first five results of the text engine’s results when the original query was used as input.
ImgTxt / TxtGeo / ImgTxtGeo. These three runs are the combination of image evidence and text evidence, text evidence and geographic evidence, image evidence and text evidence and geographic evidence, respectively. These evidences were merged by the data fusion process as described in Section 2.6. The ranks’ and filters’ weights are derived through exhaustive search of their parameter spaces. The final combination is made with optimal weights (see Table 1). ImgTxtGeo-ImageCluster. This run combines the text, image and geographic data as with the ImgTxtGeo run and then clusters the result using the Image Clustering method described in Section 2.5.
Run ID MAP Txt 0.0923 Img 0.0256 ImgBrfWeightingIterative 0.0326 ImgBrfWeighting 0.0217 ImgTxt 0.0715 TxtGeo 0.0696 ImgTxtGeo 0.078 ImgTxtGeo-ImageCluster 0.0465 ImgTxtGeo-MetaCluster 0.0461 ImgTxtGeo-MetaCluster. This run combines the text, image and geographic data as with the ImgTxtGeo run and then clusters the result using the WordNet Clustering method described in Section 2.5.
The ImageCLEFwiki task aims to investigate retrieval approaches in the context of a larger scale
and heterogeneous collection of images. The dataset of this task is a wikipedia image collection,
which contains 151,518 images created and employed by the INEX Multimedia track in 2006-2007
[
In this section, we describe the six submitted runs.
SimpleText. This run is our baseline – pure text based search. The detailed description can be found in Section 2.3.
TextGeoNoContext. This run is a combination of text evidence and geographic evidence. It uses the MR method or placename disambiguation described in Section 2.4, which is not context aware. The geographic filter and text rank were combined by multiplying the rank of each document in the text rank not appearing in the geographic filter by a penalisation value of 3.0 (detailed in Section 2.6).
Run ID MAP SimpleText 0.1918 TextGeoNoContext 0.1896 TextGeoContext 0.1896 Image 0.0037 ImageText 0.1225 ImageTextBRF 0.1225 TextGeoContext. This run is a combination of text evidence and geographic evidence. It uses the context aware Neighbourhood method or placename disambiguation described in Section 2.4. Text and geographic evidence are combined in the same way as the TextGeoNoContext run. Image. This run is a pure content based image search. Gabor texture features were extracted from the test collection and used to form a high dimensional space. The query images were compared to the corpus images using the Cityblock distance.
ImageText. This run is a combination of text evidence and image evidence. The two were combined using a convex combination of ranks (Section 2.6) using weights 0.3 and 0.7. ImageTextBRF. This run is a combination of text evidence and image evidence. The same text and image retrieval system are used as in the previous section. We use the top five results from the text retrieval results as query images for our BRF system. The three were combined using a convex combination of ranks using weights 0.25, 0.1 and 0.65 for Image, BRF and Text relevance respectively. 5.2
Our conclusions from ImageCLEF’08 are limited as none of our experiments achieved a statistically significant improvement over a baseline. Discussions of the results are provided below. VCDT. The enhanced automated image annotation method performed well appearing in the top quartile of all methods submitted, however it failed to provide significant improvement over the automated image annotation method. An explanation for this can be found in the small number of terms of the vocabulary that hinders the functioning of the algorithm and another in the nature of the vocabulary itself, where instead of incoherence we have mutually exclusive terms and almost no semantically similar terms.
ImageCLEFphoto. As mentioned in Section 4.2, despite the combination of Text and Image evidence performing worse than the text baseline with respect to MAP, the superior GMAP shows that Image evidence is improving performance in the worst case. In future work we would like to explore further data fusion methods and find a way to take full advantage of this additional evidence without undermining text retrieval where it is performing well.
ImageCLEFwiki. Our text baseline outperformed all other retrieval methods. From this we conclude that text is by far the dominant feature on retrieval for this heterogeneous collection. The fact that blind relevance feedback made negligible positive or negative difference to image retrieval combined with the very low retrieval results for image retrieval alone, implies simple features can contribute little to retrieval on this collection. Geographic retrieval offered some improvement in some measures (p@10 and R-prec.), but not statistically significant. We can only conclude that a geographically aware system could provide some improvement, but due to the short length of the documents, context based placename disambiguation will be unlikely to provide a significant improvement.
WordNet, online lexical