In our participation in the 2009 edition of the ImageCLEF task we pursued two objectives, first to expand the number of subtasks in which we evaluate our proposed multimodal fusion presented in previous works, MultiModal Local Context Analysis (MMLCA). Furthermore, we evaluated three new proposals: a subquery generation technique based on clustering, a new variant of Multimodal Re-ranking TF-IDF (MMRR TF-IDF), finally we evaluated a term filtering technique for the medical domain which is based on WordNet Domains. [4]. From the experiments conducted: On the one hand, we have confirmed that MMLCA performs better than the other local expansion techniques evaluated. On the other hand, the results show that our proposal of subquery generation based on clustering combined with PRF obtains especially good results (the 5th best textual run of the Photo Retrieval subtask in terms of F-Meassure).
Historically in VIR area there were two approaches used to carry out the IR of images: In
the beginning of the VIR in the late 70s, VIR systems were based on the image annotations,
therefore, these were Text-Based VIR (TBIR) systems. Later in the early 90’s, in an attempt to
overcome the dependence of TBIR systems from the existence of textual annotations to perform
the indexing of an image, the image Content Based VIR (CBIR) systems appear [
Finally, in recent years as the technologies used by CBIR systems matured, a third approach to tackle the problem of the VIR emerged, these systems combine textual and image based technologies. In this context are organized competitions like ImageCLEF1 which is a specific VIR task which takes place within the framework of the annual competitions of the CLEF2 campaigns. These competitions aim the development of multimodal systems using image collections with their related short annotations.
In our participation in the 2009 edition of the ImageCLEF task we pursued two objectives,
first to expand the number of subtasks in which we evaluate our proposed multimodal fusion
presented in previous works, MultiModal Local Context Analysis (MMLCA) [
This paper is structured as follows: Firstly, it presents the main characteristics of the IR-n system focusing on the techniques used, and then it moves on to describe the experiments and the results obtained. Finally, it presents conclusions and future work. 2
To perform the experiments we used IR-n, an information retrieval system based on passages.
Such systems treat each document as a set of passages; each passage defines a portion of text from
the document. Unlike systems based on documents, passages based systems give greater relevance
to those documents where the query terms appear in closer positions to each other [
The IR-n architecture allows us to use local query expansion based on either the most relevant
passages or the most relevant documents. Furthermore, it allows choosing between two different
term selection strategies for the query expansion. These strategies are Probabilistic Relevance
Feedback (PRF)[
In a previous work we evaluated these techniques on different generic domain image collections.
We concluded that MMLCA is the best of these techniques in terms of precision and the only one
of the four which does not hurt the diversity of the results returned by the VIR system. [
This strategy involves the merging of the list returned by a TBIR system and the list returned by a CBIR system. Following a standard re-ranking strategy this is done by giving a different weight to the normalized relevance value or ranking position for a document in each list, the Textual List (TL) and the Visual List (VL).
IR-n allows to use the standard re-ranking strategy and also a variation of our MMRR TF-IDF
proposal used in previous works [
1http://www.imageclef.org 2http://www.clef-campaign.org
Our previous TF-IDF Re-ranking proposal was based on the following assumptions: on the one hand, the list based on image annotations is more confident than the list based on images and on the other hand, we assumed that TF-IDF is a suitable way to measure the quantity and the quality of a text. Thus, we used a TF-IDF threshold to decide whether for the final relevace value of an image we only use the relevance value returned by the TBIR or we take the risk of adding also the relevance value returned by the CBIR.
These assumptions proved to be effective but were not enough. There were two problems with this approach: The first one is that we consider that any relevance value obtained from a CBIR system always have the same level of risk, independently of the position of its related image in the VL. Nervertheless, this is not true since that although CBIR systems obtain lower MAP results than TBIR systems, in general the first ones obtain not bad P20 results (see Photo Retrieval 2008 results).
The second problem in our previous approach was that the TF-IDF threshold was very dependent of the images returned for each query. The reason is that the system used a percentage value as parammeter for the threshold. Later for each query it obtains the value of the threshold applying the percentage parammeter to the maximum TF-IDF value obtained from the documents in the two lists. This dependence from the image subset retrieved could cause erratic results.
In order to solve these problems our proposal should use a new threshold formula. This formula should allow us to manage different levels of risk in order to achieve an optimal performance and should depend from the characteristics of the entire colletion instead of only depend on the image subset retrieved for each query.
The new threshold (1) used is based on two points: firstly, on to work out the average of the TF-IDF value per sentence (T F IDF perSentence) for the whole collection. We use this value for avoiding the dependence of the characteristics of the image set returned for a query. Secondly, on to use a simple lineal function for decreasing the threshold value as the position of the image increases in the VL.
threshold(pos) = numSentences ∗ T F IDF perSentence ∗ ( −pos maxP osCBIR + 1) (1) • T F IDF perSentence is the average TF-IDF value per sentence for the whole collection, this value is worked out automatically by the system. • numSentences, this parameter is a multiplier which represents the maximum number of sentences that the annotations related to an image can contain in order to take into account the relevance value returned by the CBIR for this image. • pos is the position of the image in the VL. • maxP osCBIR, this parameter indicates the limit position in the visual ranking to use the relevance values returned by the CBIR.
Reviewing the TF-IDF MMRR formula (2) we can observe that the threshold is used to avoid the risk of use the CBIR relevance values for those images which are low ranked in the VL and for those ones which have image annotations with enough quantity and quality to perform a suitable TBIR.
T L(d) + V L(d), if 0 <= pos(d) <= maxP osCBIR F L(d) = and threshold(pos(d)) > T F IDF (d) T R(d), else (2) • T F IDF (d) is the TF-IDF value of the text document d related to an image. • pos(d) returns the position of the document d if it exists in the visual list else it returns -1.
It is important to point out that when the TF-IDF MMRR strategy is enabled it is executed for each retrieval iteration. It implies that if the system is using a local expansion technique the TFIDF MMRR will be executed for the two retrieval iterations involved. Firstly, in the first iteration of the retrieval it will modify the document list used to feed the local expansion strategy. Finally, in the second iteration it will modify the document list returned after retrieve the expanded query. 2.3
Usually, when the user uses a VIR system, they find that there are several similar images between the first results returned by the system. Thus, if they want to find different relevant images they have to navigate through the next pages of the returned results.
Our approach is an attempt to solve this problem. It is based on Lingo [
Opposite to the clustering approach we used in our previous participation, instead of using the documents returned by the clustering tool, our system uses the set of terms which represent each one of the clusters returned by Carrot2, the label of the clusters. The system builds a subquery per each term set representing a cluster. The new subqueries are compounded by the terms of the original query jointly with those terms representative of the cluster, discarding those terms which already exist in the original query and that are not repeated in other cluster labels returned by Carrot2.
Finally, the system performs the retrieval for each one of the new generated subqueries obtaining a ranking list per subquery. These ranking lists are fused using an standard re-ranking strategy in order to produce the final ranking result.
It is important to point out that when the clustering module is used to create subqueries, the system uses the same local expansion and re-ranking configuration for obtaining the ranking used to feed this clustering module and for the retrieval of each one of the subqueries generated. 2.4
In our previous experiments using generic domain image collections we observed that especially
when the precision of the initial ranking used for the expansion has a poor precision, in general
LCA shows better precision results than PRF. [
However, this behavior was not observed in the experiments performed at the 2008 Medical Retrieval subtask. Reviewing the behavior of the system, we concluded that a reason for this different behavior could be that in the medical domain the terms used in the collections follow a different distribution of significance in comparison with their distribution in a generic domain. This different distribution could hurt the performance of LCA, an strategy based on term coocurrence. In fact, we believe that while some terms of the query are especially relevant under a medical viewpoint other of its terms are not so important as their frequency figures in the collection indicate. Thus, a possible solution could be to filter the terms returned by the local expansion strategy for the expansion, in order to only allow terms with enough significance under the medical domain.
For this purpose, we generated a stopword list specifically for the medical domain. In order to generate it we used all the WordNet terms which not pertain to the following medical WordNet Domains: medicine, dentistry, pharmacy, psychiatry, radiology, surgery, purescience, chemistry, biology, biochemistry, zoology, anatomy, physiology and genetics. 2.5
The system uses a query expansion module based on MeSH. It is the same used in our past participation in the Medical Retrieval task at ImageCLEF 2008. 3
This section shows the results of the system and describes the configuration used for each one of the tasks in which we have been involved. Below there is the description of each one of the IR-n configuration parameters used in our different participations: • Relevance Feedback (relF B): Indicates which relevance feedback is used PRF, LCA,
MMPRF or MMLCA. • Relevance Feedback parameters (exp/num/ncbir/term): If exp has value 1, this denotes we use relevance feedback based on passages. But, if exp has value 2, the relevance feedback is based on documents. Moreover, num denotes the number of passages or documents that the local expansion strategy will use from the textual ranking, ncbir denotes the number of documents that the multimodal local expansion strategy will use from an image based list and finally, term indicates the number of terms that the local expansion strategy will use for the query expansion. • Subqueries Generation based on Clustering (clust): Indicate if this module is used or not. • Multimodal Re-ranking Strategy (rrtfidf ): Indicate if the system uses the MMRR
TFIDF or not. • Automatic query expansion based on MeSH (mesh): Indicates if it is used or not. • Medical Stopwords based on WordNet Domains(stopWN): Indicates if the system uses medical stopwords filtering for the terms selected by the local expansion technique.
For all the experiments we have use DFR as the weighting schema. We based this decision on our training results at ImageCLEF 2007 edition. 3.1
For our participation in the Photo Retrieval subtask the CBIR system used was FIRE [
Further information regarding the collections and the topic sets used in this task can be found
at [
We divide the results achieved in the competition in two parts. The Table 1 shows the Part 1 query set results ordered by F-Meassure. It shows the results of our submitted runs jointly with the best TXT run and the best TXTIMG run of the subtask for this topic set. The characteristic of the queries within this query set is that they provide a number of subqueries and images, per query, which define the different clusters that should be retrieved in order to return a diverse list of images to the user.
Using the extra information provided for this query set IR-n replaces the cluster labels returned by Carrot2 with the subqueries provided for each query. Furthermore, the system obtains the VL for each subquery using the image provided with each subquery of the topic.
For this topic set our group was the 5th best group in terms of F-Meassure for a total of 19 participants.
Finally, point out that our Alicante2 run was the 5th best textual run of the competition based on its results for the two subsets. 3.2
run deuceng
TXT lach TXTIMG Alicante7 TXTIMG Alicante5 TXTIMG Alicante4 TXTIMG Alicante1
TXT Alicante2
TXT Alicante6 TXTIMG Alicante8 TXTIMG Alicante9 TXTIMG yes yes no yes yes yes yes no yes
Our group was ranked at the 13th position in terms of MAP for a total of 16 participant groups. 4
Our conclusions related with the generic domain subtasks are the following ones: On the one hand, taking into account that for the Photo Retrieval subtask we have used an irregular procedure for obtaining the visual lists and that for the WikipediaMM we do not have used all the images provided with the queries, we have confirmed that even with this problems MMLCA performs better than the other local expansion techniques evaluated (see Photo Retrieval Part 1 and WikipediaMM results).
On the other hand, regarding our subqueries generation based on clustering technique, we observe that it works better when it is combined with PRF than when it is combined with MMLCA. It explains the good results achieved by our Alicante2 run, which using PRF and out clustering based technique achieved the 5th best textual run of the Photo Retrieval subtask. In future works we will study deeply the reasons for this profitable relationship.
Point out that after reviewing the results we found an important bug in the MMRR TF-IDF strategy used that has hurted all the runs which used it. Thus, we are forced to delay for future works the evaluation of this technique.
Finally, for the Medical Retrieval results we conclude that our proposal of Medical stopwords based on WordNet Domains did not improve the MMLCA results as we expected. Further analysis should be performed in the future to understand why MMLCA does not work as well in medical domain as it does for generic domain. This research has been partially funded by the Spanish Government within the framework of the TEXT-MESS (TIN-2006-15265-C06-01) project and by European Union (EU) within the framework of the QALL-ME project (FP6-IST-033860).