For this year's Image CLEF Photo Retrieval task, we have prepared 5 submission runs to help us assess the e ectiveness of 1) image content-based retrieval, and 2) textbased retrieval. We investigate whether the clustering of results can increase diversity by returning as many di erent clusters of images in the results as possible. Our image system uses the FIRE engine to extract image features such as color, texture, and shape from a database consisting of more than half a million images. The text-retrieval backend uses Lucene to extract texts from image annotations, title, and cluster tags. Our results reveal that among the three image features, color yields the highest retrieval precision, followed by shape, then texture. A combination of color extraction with text retrieval has the potential to increase precision, but only to a certain extent. Clustering also improves diversity in one of our clustering runs.
The goal of this year's Image CLEF event is to promote the diversity of search results through
presenting relevant images from as many clusters as possible. Such cluster may focus on the
location where the image was taken, the subject matter, the event, the time, etc. Our database
consists of an unprecedented 498,920 newspaper images, courtesy of the Belgium news agency,
each containing a picture, a title, a short description of the image, and a time stamp. Handling
a database of such size is already a feat on its own. Each of our 50 queries consists of up to 3
sample images (each having a description and a picture). We may use the text, the image, or both
parts as query for the retrieval task. In addition, our queries are divided into two parts: part 1
(25 queries) provides the cluster titles for each query to help us cluster the results; part 2 does
not provide any cluster hints. For more details about the task see [
The University of Ottawa team has developed a system for text-based retrieval, and image content-based retrieval. In the sections that follow, we describe each system, compare their retrieval e ectiveness, and investigate whether or not clustering helps increase the diversity of results. We have used the k-means clustering algorithm. Then we describe two ways to incorporate clusters into the resulting ranking.
This system is running on the Lucene search engine that searches through a document collection based on the frequency of query terms in the document (tf-idf measure). In our system, the image annotations, titles, and tags are indexed. To further improve the search results, we undertook two additional steps: stemming and query expansion. 2.1.1
It has been shown that stemming can slightly improve retrieval scores. Stemming means removing
the su xes of words, such as -ly, -ing, -ed, -s, etc, which sometime maybe overlooked by the
system. As a pre-processing step prior to building the index, we converted all words into their
stemmed form by running the Porter Stemming algorithm [
In addition to using the image title as our query, we also expanded the query using the terms that
appear in the description section of the 3 sample images. This was used in last year's competition,
and has been shown to work well [
X d~j 1 jDj d~jinD where: q~0 is the original query vector (i.e., the image title); D is the set of known relevant documents (i.e., the description of sample images); d~j is the frequency vector for a relevant document j in D.
We used the following parameters from Rocchio's method: = 1:0, = 0:75. 2.2
The wealth of image data provides us with an excellent opportunity to assess di erent image retrieval methods. The image database is the largest we have tested to date, and we shall see how our system performed under such a heavy load. Our system extracted 3 image features from each image: color, Tamura texture, and scale invariant feature transform (SIFT) [5].
Of particular interest is the SIFT feature, which is a feature related to shapes. This local image
feature extracts particular interest points from the image which are highly distinctive, relatively
stable to scale, and invariable to rotations and minor changes in illumination and noises. Images
are rst applied a Gaussian-blur lter at di erent levels, producing successively blurred images.
The di erences between the blurred images are calculated based on the Di erence of Gaussians
(DoG) technique. And from the extremes of DoG, local interest points are derived. We have
found a front-end SIFT extraction tool (called extractsift ) from the FIRE image retrieval package
[
Because feature extraction was a very lengthy process, some time-saving tricks were needed. In particular, the SIFT extraction takes 10 sec/image on an Athlon 64 3.0GHz dual-core system, which is simply too long. So we have reduced the size of all images by 50% to allow us to nish the tasks in reasonable time. 2.3
To investigate the e ect of clustering documents, we employed the k-means clustering algorithm
on the documents retrieved from the query-expanded text retrieval system. The version of the
algorithm we used can be found in [
It is important to mention that clustered documents are re-inserted into the ranking in a way that increases the diversity of results. Two ways of doing this are proposed:
1) Cluster-by-cluster: Clusters are ranked in descending order by the average similarity score of documents in the cluster. Then, documents in the top scored cluster are all inserted to the ranking, followed by the next top score cluster, etc.
2) Interleaved: Again clusters are ranked in descending order. Di erently from above, only one document from each cluster is inserted into the ranking at a time. When all clusters have contributed at least one document to the ranking, the method begins inserting the second document into the ranking. This is what we hope to be the better way. 3
Due to the limitation of 5 submission runs per participant, we were not able to try out the combination of text and image retrieval systems. But we anticipate that the combination run will generate improved scores, because of the following evidences.
First, it is evident that content-based image retrieval alone cannot achieve good performance, because the precision values are simply too low. However, we notice that the precision at depth 5 is the highest for image retrieval, suggesting that perhaps these documents could be added to the text results to boost its performance.
Second, we often have the same text retrieval score for two or more retrieved documents, which
makes ranking di cult. If the image retrieval score is combined with text retrieval using careful
weighting, it becomes much easier to assign the ranking. Previous work shows that a weighting
scheme of 85% text score + 15% image score raises the precision by about 0.03, comparing with
text-only retrieval system [
Among the 3 image features tested, color is the best feature, followed closely by SIFT, and then by the Tamura texture. The striking similarities between the precision of SIFT and Tamura runs seem to re ect a baseline performance that might resemble random retrieval. Alternatively, their low precision scores might be explained by the reduced sizes of image, which might have eliminated too many details needed for extractions.
Lastly, clustering of documents can tip the ranking to favor either precision (all documents of a cluster are located at top positions) or diversity (interleaving insertion of cluster documents). This is clearly seen in the Clusters Interleaved and Clusters NonInterleaved runs. [5] Lowe, D: Object recognition from local scale-invariant features. Proceedings of the International Conference on Computer Vision 1999, 2:1150{1157.