For the CLEF 2004 ImageCLEF St Andrew's Collection task the Dublin City University group carried out three sets of experiments. We carried out standard cross-language information retrieval (CLIR) runs using topic translation using machine translation (MT), combination of this run with image matching results from the VIPER system, and a novel document rescoring approach based on automatic MT evaluation metrics. Our standard CLIR approaches works well in comparison on this task. Encouragingly combination with image matching lists can produce small positive changes in the overall retrieval output. However, rescoring using the MT evaluation metrics in their current form significantly reduces retrieval effectiveness.
2.1
Prec. 10 docs Av Precision Rel. Ret.
For the all the experiments report here the Okapi parameters were set using the provided training topics as follows; K1 = 1.0 and b = 0.5 for baseline runs and K1 = 1.5 and b = 0.6 for PRF runs. The 20 top ranked PRF expansion terms were selected from the summaries of the top 5 ranked documents. The original topic terms were upweighted by a factor of 3.5 relative to terms introduced by PRF. There are a total of 829 relevant images available in the collection.
Topics were translated into English, the document language, using the following web-based MT systems: Systran (http://www.systransoft.com/), SDL (http://www.freetranslation.com/) and InterTrans (http://www.intertrans.com/). Results are show for CLIR using each separate translation and a term union merged translation.
Baseline Runs Table 1 shows baseline retrieval runs for Systran without application of PRF. Results in all languages appear reasonable with little apparent correlation between precision and recall figures.
Feedback Runs The text annotations of the images are typically very short, typically comprising only a few sentences. In developing our system for the PRF compared our summary-based approach developed for use with newspaper archives with a standard PRF approach selecting terms from complete documents. We were a little surprised to find that selecting terms from summaries of even these short documents worked better on the development topics than the whole document approach.
Table 2 shows feedback results for each topic language with the three MT systems and the merged translated topics. Separate results are shown for the original and later released revised Spanish topics. Comparing all these runs we can see that for Systran, PRF on on average produces
SDL 0.480 0.394 638 0.472 0.407 666 0.604 0.501 763 0.400 0.369 633 0.472 0.441 767
In recent years, several automatic MT evaluation methods have been proposed as a supplement to,
or, in certain cases, a replacement for costly human MT evaluations [
In order to be able to use these metrics to calculate the similarity between a user query and a topic document in IR, we regard the original topic document and the MT-translated user query as translations of an unknown source text, as is shown in Figure 1.
The same three sets of topic translation were used as in the previous experiments. The topic translations and documents were pre-processed to remove stopwords, capitalisation and punctuation.
If we think of the query translations as human reference translations, it is possible to measure the accuracy of the would-be ’machine translations’ (the documents) using automatic MT evaluation metrics. The best ’machine translation’ is the translation with the lowest word-error score with regard to the reference translations. The goal of our experiment was to find out to what extent the best ’machine translation’ corresponded with a relevant document.
Experiments with development topics showed that best results were obtained with a combination of 2 existing MT evaluation methods (NIST and GTM) and an adaptation of the BLEU evaluation metric.
BLEU ranks different MT output texts based a combination of an N-gram similarity score and a sentence brevity penalty with respect to a corpus of human reference translations. The BLEU evaluation script was adapted in two ways. First, we eliminated the sentence brevity penalty. The original BLUE metric penalizes short sentences to avoid the possibility that very short segments such as ’the’ would receive a maximum score when compared to any sentence containing ’the’. This penalty is clearly not relevant for the retrieval task at hand. A second modification to the script consisted in allowing a non-zero BLEU score, regardless of the fact that for one or more of the N-gram categories (unigram to 4-gram) no positive matches were found between the MT output and human reference translations.
NIST differs from BLEU with respect to both the co-occurrence score and the sentence brevity penalty. NIST alters the co-occurrence score in favour of lower order N-grams (i.e. low trigrams or quadrigram matches play less a role in the overall score) and more informative N-grams (i.e. N-grams that occur less frequently receive a higher weight). The sentence brevity penalty used by NIST is less severe than the one used by BLEU for sentences with small variations with respect to the reference translation.
GTM allows the calculation of standard precision and recall scores for automatically produced translations. It also calculates an f-measure score, which combines both the precision and recall scores for a given translation. It is this f-measure score, along with the NIST and adapted BLEU scores, that we used in our automatic ranking of the documents.
During our experiments we ranked the translated queries against the top 1000 documents retrieved for each topic using the PRF approach described in the previous section. We used a summation of the NIST, f-measure and adapted BLEU scores. We ran two sets of experiments. In the first set of experiments we evaluated the retrieved document list against only one reference translation, as produced by one of the three online MT systems, giving us three resulting ranking lists of documents for each topic. In a second set of experiments we merged the translated queries, using the three different translations of the topic as three different reference translations.
Table 4 shows results of document rescoring using MT evaluation metrics. Comparing these results to those using standard PRF methods in the earlier tables, it can be seen that the MT evaluation metrics are not effective for IR scoring in their present form. The main goal of our experiments was not to substantially improve the best available Image Retrieval methods, but
to investigate the novel idea for IR of treating topic documents and translated user queries as comparable translations of an unknown source text. Clearly based on the results shown here we need to explore further whether this approach can be adapted successfully for IR applications. 4
Our experiments for ImageCLEF have demonstrated that our standard CLIR method works effectively for the short text documents in the St Andrew’s collection, and further that there is potential for improvement in retrieval effectiveness from the use of image matching in CL image retrieval. Our experiments using MT evaluation metrics for scoring CLIR have so far not been successful, but we will be analysing our results to better understand the results and to seek alternative means of applying this approach.