We provide a technical description of our submission to the CLEF 2006 Cross Language Image Retrieval(ImageCLEF) Photo Collection Standard Ad Hoc task. We performed monolingual and cross language retrieval of photo images using photo annotations with and without feedback, and also a combined visual and text retrieval approach. Topics are translated into English using the Babelfish online machine translation system. Our text runs used the BM25 algorithm, while our visual approach used simple low-level features with matching based on the Jeffrey Divergence measure. Our results consistently indicate that the fusion of text and visual features is best for this task, and that performing feedback for text consistently improves on the baseline non-feedback BM25 text runs for all language pairs.
Dublin City University’s participation in the CLEF 2006 ImageCLEF Photo Collection Ad Hoc
task adopted standard text retrieval using image metadata and text search topics, with and without
pseudo relevance feedback (PRF) and a combination of text retrieval with low-level visual feature
matching. The underlying text retrieval system is based on a standard Okapi model for document
ranking and PRF [
The results of our experiments demonstrate that PRF improves on the baseline in all cases with respect to both average precision and the number of relevant documents retrieved. Combined text retrieval with visual feature matching gives a further improvement in both of these retrieval effectiveness measures in all cases.
This remainder of this paper is organised as follows: Section 2 briefly outlines the details of our standard retrieval system and describes our novel PRF method, Section 3 details our submitted runs, Section 4 gives results and analysis of our experiments, and finally Section 5 concludes the paper. 2
The introduction of a new collection for the CLEF 2006 Ad Hoc Photo retrieval task meant that there was no previous retrieval test collection to use for system development and tuning for these documents. We thus used the previous ImageCLEF St Andrew’s collection and related experiments on the TRECVID datasets to guide our selection of fusion methods and retrieval parameters for our experiments. 2.1
The contents of the structured annotation for each photo (TITLE, DESCRIPTION, NOTES,
LOCATION and DATE fields) were collapsed into a flat document representation. Documents
and search topics were processed to remove stopwords from the standard SMART list [
Based on these development experiments, the text feature was matched using the BM25
algorithm with parameters: k1 = 1.0, k2 = 0, and b = 0.5. When using relevance feedback, the top
15 documents were assumed pseudo-relevant and the top scoring 10 expansion terms calculated
using the Robertson selection value [
The visual features were matched using the Jeffrey Divergence (a.k.a. Jensen-Shannon distance)
matching function [
Three following visual features were used: HSV colour histogram with 16x4x4 quantisation
levels on a 5x5 regional image grid, Canny 8 edge + 1 bin for non-edges feature for each region of
a 5x5 regional image grid, and a DCT histogram feature based on the first 5 coefficients quantised
each into 3 values for a 3x3 regional image grid. More details on these features can be found in
[
The results for the three visual features were combined using the weighted variant (i.e. linear
interpolation) of the CombSUM fusion operator [
The results from the two visual examples for each topic were fused using the CombMAX fusion
operator, which took the maximum of the normalised scores from each separate visual result list
[
Text and visual runs were fused using the weighted CombSUM fusion operator with weights 0.70 and 0.30 for text and image respectively. Scores were again normalised to lie between 0 and 1 in the separate text and visual result sets before fusion. 3
We submitted two mono-lingual runs for German and English and cross language runs for both
these languages. For English photo annotations we submitted runs where the queries were in
Russian, Portuguese, Dutch, Japanese, Italian, French, Spanish, German, Chinese. While for
German photo annotations we only ran cross language queries for French and English queries. The
queries were translated to the respective document target language using the online Babelfish[
For each language pair including the monolingual runs we evaluated three different approaches: text only queries with no feedback and with feedback, and a combined visual and text (with text feedback) run. This gave us a total of 39 runs submitted. 4
Results for our runs are shown in Table 1 From the table we can see that our multilingual fused text and visual submission performed very well, achieving either the second or top rank of submissions for each language pair in terms of Mean Average Precision (MAP). Text feedback increased the text only results in terms of MAP by on average 16.0%. Fusion with visual results increased these results on average by a further 9.2%. Our experiments produced consistent evidence across language pairs that text feedback and fusion with visual features is beneficial in Image Photo search.
Our monolingual English runs with text feedback and fused visual results performed relatively poorly, achieving 8th of 50 submissions in terms of MAP. The increased competition in the more popular monolingual English category relative to the multilingual categories as well as our limited approach in tuning a single parameter selection for all our submitted runs probably accounts for the relative decrease in effectiveness of our approach in the English monolingual category. Our system parameters were tuned for multilingual topics and in future we should tune separately for the monolingual case. For monolingual searches we would expect the effectiveness of the initial query to be higher than in the multilingual case, and therefore both the appropriate feedback and fusion parameters for combining text and visual results may differ significantly compared to the multilingual case. In our case, the initial English monolingual text query under-performed, but was increased by 20.3% by the text feedback approach we employed. This result was further improved by only 4% through fusion with visual results. We suspect that the optimal parameters for BM25 may differ significant between the monolingual and multilingual cases.
Our results consistently show that our fused text and image retrieval submission outperforms our text-only methods. The average relative improvement in MAP was 9.2%, maximum was 18.9% and minimum was 4.0%. The evaluation measures Mean Average Precision, total relevant documents retrieved and precision at document cut-offs 10, 20, 30 are improved for all tested language pairs when the text results are fused with image results compared to text alone. This indicates that our fusion approach is stable and produces reliable results. We suspect that our fusion parameters were a bit conservative in the importance given to the visual results for this task. But on the other hand, if we increase the importance of visual results, we may sacrifice some of the stability and consistency of our results. This will be investigated in followup experiments.
Our results also consistently show that our text runs are improved for all language pairs tested when using text feedback compared to without it. This is true for all language pairs and evaluation measures in Table 1, except for precision at a cut-off of 10 documents for English queries against German documents. The decrease in precision is small and insignificant in this case and goes
Media
MAP
Rank 4.0% 20.3% against the overwhelming trend established by our results. At an average relative increase in MAP of 16.0%, a maximum of 28.8% and a minimum of 9.9% for the language pairs, the importance of text feedback is well established for text-based photo search by our results.
The lack of relevant tuning data because of the significant difference between this year and previous years ImageCLEF photo content and descriptions may have led to a less than optimal choice of parameter for fusing visual and text results. Post-ImageCLEF experiments should be able to quantify the improvements that can be made with better tuning or alternative fusion strategies. 5
Our results for the ImageCLEF 2006 photo retrieval task show that fusing text and visual results achieves better effectiveness than text alone. We also demonstrated that PRF is important for improving the effectiveness of the text retrieval model with consistent improvement in results across language pairs. Future experiments will investigate fusion of text and visual features more deeply, since we believe this still has more to offer than we have shown in our current experiments.