This paper describes our diversity featured retrieval system which are built for the task of ImageCLEFPhoto 2008. Two existing tools are used: Solr and Carrot2. We have experimented with different settings of the system to see how the performance changes. The results suggest that the system can indeed increase diversity of the retrieved results and keep the precision about the same.
In this paper, we describe how to quickly set up a diversity featured search system by combining existing tools that are available to the public, namely Solr [2] and Carrot2 [3]. We further describe how to tune the system for better performance in the context of ImageCLEFPhoto 2008. We use Solr for text search and Carrot2 for increasing diversity of results.
Solr is a text search server based on the popular search library Lucene [1]. Solr provides some very useful and convenient features. The web-services like API allows users to index documents via XML over HTTP, query it via HTTP GET and receive results in XML format. The fields and field types of documents can be easily defined in the schema.xml file. In this file, users can also specify the Solr “out of the box” tokenizers and token filters to use for indexing and query. In addition, the HTML administration interface gives users comprehensive insight into the system.
Carrot2 is a open source search results clustering engine. It provides five different algorithms for automatically organising search results into thematic categories. Carrot2 works as a pipeline of three kinds of components: input components, filter components and visualisation components. Input components obtain search results from a source of choice (e.g. YahooAPI, GoogleAPI, Lucene, Solr, etc.), then filter components apply clustering algorithms to the search results, and finally visualisation components render the clustered results to the user. 3
In ImageCLEFPhoto 2008, each image comes with 9 fields of data: DOCNO, TITLE, DESCRIPTION, NOTES, LOCATION, DATE, IMAGE, THUMBNAIL and TOPIC. We decided that TITLE, DESCRIPTION, NOTES, LOCATION are the fields that would provide useful information for text based image retrieval. Therefore, we constructed a new field named TEXT by combining the text from the four fields, and specified it as the default search field in Solr. Solr’s default configuration of tokenizers and token filters is used in our experiments, namely WhitespaceTokenizer, SynonymFilter, StopFilter, WordDelimiterFilter, LowerCaseFilter, EnglishPorterFilter and RemoveDuplicatesTokenFilter. More details on each tokenizer and token filter can be found on [2]. After feeding the Solr server with all the 20,000 documents in XML format, we have a running search engine that is able to return a ranked list of documents based on the query submitted by the user. Note that all the fields are indexed by Solr.
In Carrot2, we construct an input component for acquiring results from the Solr server. Then,
a filter component is used for clustering the results, and a visualisation component is used for
displaying the clusters and their members. Since ImageCLEFPhoto 2008 assesses the S-recall [
Basically, the above pseudo code chooses documents based on two criteria: 1. appear as early as possible in the input/ranked list, 2. cover as many different clusters as possible. The 20 documents chosen by the above procedure form the list to be submitted for assessment. 4
We have submitted 32 runs, all of which are EN-EN-AUTO-TXT, meaning that all the submissions are English-English monolingual runs using fully automated text clustering methods. Among them, there are 8 groups, each of which includes 4 runs with the same system configuration except that different number of documents are used for clustering. In other words, for each group, we vary the number of documents (40, 60, 80 and 100) that are used from the top of the input list for clustering by Carrot2, in order to get different output lists. For example, a run with top 40 documents applies clustering on the 40 documents and chooses 20 for submission based on the procedure in Section 3.
We increasingly change another 3 kinds of settings of the system to examine how the performances would change. Firstly, for topics whose CLUSTER field is Country, City, State or Location, we specify Carrot2 to cluster the documents by the LOCATION filed, otherwise by the TEXT field. Secondly, we change the parameters of the clustering algorithm used in Carrot2. Finally, we apply expansion to the indexing and query stage. In the following, we describe each group of runs. 4.1
This group uses the default settings of Solr and Carrot2. In terms of text retrieval, expansion
is not applied during indexing or query. Clustering is applied to the TEXT field for all topics.
Carrot2’s default clustering algorithm (Lingo [
It seems that for topics that have been specified to be clustered by Country, City, State and Location, the LOCATION filed in each document contains the essential information for clustering. Therefore, such topics are clustered based on the LOCATION field. This group is different from the Baseline group in that clustering on some topics are based on the LOCATION field, while others based on the TEXT field. 4.3
Lingo [
This group is the same as group 4.3 except that it uses (0.10, 0.90). 4.5
This group is the same as group 4.3 except that it uses (0.15, 0.85). 4.6
This group is based on group 4.3, but applies indexing and query expansion in Solr. Specific domain ontologies have been a popular choice for expansion. Cyclopedia websites such as Wikipedia have also been adopted for expansion. Due to limited time, we used neither of the approaches. Instead, we examined the topics and data-set, and then manually built an indexing expansion list and query expansion list, as shown in Appendix. These lists are used as the synonym lists in Solr for indexing and query. In the expansion lists, for lines containing “>=”, any of the words before “>=” are replaced by words after “>=” during expansion. For example, “ship, ships => ship, vehicle” will replace any “ship” or “ships” by “ship, vehicle”. For lines without “>=”, any word from the line will be replaced by the whole set of words from the same line. For example, “USA” or “United States of America” or “US” will be replaced by “USA, United States of America, US”.
Precision@20 0.4 0.35 0.3 0.25 0.2 0.15 0.05 0
This group is the same as group 4.6 except that it uses (0.10, 0.90) in Lingo. 4.8
This group is the same as group 4.6 except that it uses (0.15, 0.85) in Lingo. 5
As shown in Figure 1, group 6, 7 and 8 have clearly higher precisions than the other groups. This is due to indexing and query expansion using the expansion lists in Appendix. As have been mentioned, the expansion lists are built based on an examination of the data-set. For example, in Topic 48 “vehicle in South Korea”, “South Korea” normally means the country, so there is not much ambiguity. However, “vehicle” can mean many things, e.g. car, bus, boat. Intuitively, expanding names of different types of vehicle with the word “vehicle” during indexing will boost the precision, because many images are only annotated with specific vehicle names rather than the word “vehicle”. Therefore, after indexing expansion, “car” becomes “car vehicle”. Similarly, we expanded specific animal names with the word “animal”, so “fish” becomes “fish animal”.
In terms of cluster recall, we can see in Figure 2 that different parameters of the clustering algorithm (Lingo) have led to different performances. It is a little surprising that group 2 (Section 4.2) performed worse than group 1 (Section 4.1), the reason of which needs to be examined. In addition, it seems that low Cluster Assignment Threshold (i.e. more documents are clustered) and high Candidate Cluster Threshold (i.e. more clusters are created) give better cluster recall. In our experiments, (0.05, 0.95) gives the best results: group 3 is better than group 4 and 5; group 6 is better than group 7 and 8. 0.45 0.4 0.35 0.3 0.25 0.2 0.15 0.1 0.05 0 40 60 80 100 1 2 3 4 5 6 7 8
Group
By comparing the two figures, it can also be noticed that groups with the same settings of Solr have very similar precisions, no matter what settings of Carrot2 were used. For example, with different parameters of Lingo, the precisions of group 6, 7 and 8 are relative stable, but the values of cluster recall vary. This can be seen as an evidence that the diversity featured retrieval system can make the results more diverse while maintaining the precision. 6
This paper has described our submissions to ImageCLEFPhoto 2008. We have changed 4 kinds of settings: the field used for clustering, the number of images used for clustering, indexing and query expansion, and parameters of the clustering algorithm. The results suggest that indexing and query expansion can fairly improve precision. Appropriately chosen clustering method can increase diversity of the results while keeping precision almost the same.
As we have mentioned, the performance of group 2 is a little against our initial anticipation. It would be interesting to find out why. On the other hand, we plan to build an automatic expansion approach using resources such as ontologies, rather than using the manually built expansion lists. 7
Work undertaken in this paper is supported by the EU-funded TrebleCLEF project (Grant agreement: 215231) and by Tripod (Contract No. 045335). [1] Apache lucene project. http://lucene.apache.org. [Visited 23/07/08]. [2] Apache solr project. http://lucene.apache.org/solr/index.html. [Visited 23/07/08]. [3] Carrot2 project. http://project.carrot2.org/. [Visited 23/07/08].
A A.1
Indexing Expansion List # vehicles ship, ships => ship, vehicle cutter, cutters => cutter, vehicle train, trains => train, vehicle rail, rails => rail, vehicle locomotives, locomotive => locomotive, vehicle wagon, wagons => wagon, vehicle tractor, tractors => tractor, vehicle bus, buses => bus, vehicle car, cars => car, vehicle forklift, forklifts => forklift, vehicle boat, boats => boat, vehicle Penguin, Penguins => penguin, animal condors, condor => condor, animal monkey, monkeys => monkey, animal bird, birds => bird, animal iguana, iguanas => iguana, animal snail, snails => snail, animal toucan, toucans => toucan, animal lion, lions => lion, animal llama, llamas => llama, animal snake, snakes => snakes, animal Tortoise, Tortoises => Tortoise, animal parrot, parrots => parrot, animal Booby, Boobies => Booby, animal horse, horses => horse, animal sea lion, sea lions => seal, animal
# sports football => football, sport surf => surf, sport motorcycle => motorcycle, sport race => race, sport
# people fans, fan => fan, people
# water river => river, water lake => lake, water
# stone, rock rock => rock, stone brick, bricks => brick, stone A.2
# countries USA, United States of America, US
# synonym church, churches, cathedral, cathedrals oxidised, rusty observing, watch, see football, soccer match, game accommodation, room
# common sense swimming => swimming in the water drawings, drawing => drawing, line, petroglyph prize, prizes => prize, medal water => water, sea