In this paper, we use the Systran machine translation system for translating queries for cross language image retrieval in a pilot experiment at CLEF 2003, called ImageCLEF. The approach we have taken is to assume we have little experience in CLIR, few available resources and a limited time in which to create a working CLIR system for this task. In this preliminary study, we investigate the effectiveness of Systran on short queries by comparing a manual assessment of translation adequacy with an automatic score derived using NIST's mteval evaluation tool for machine translation output. We discuss the kinds of translation errors encountered during this analysis and show the impact on retrieval effectiveness for individual queries in the ImageCLEF task.
ImageCLEF is a pilot experiment run at CLEF 2003, dealing with the retrieval of images by their captions in
cases where the source and target languages differ (see [1] for further information about ImageCLEF). Because
the document to be retrieved is both visual and textual, approaches to this task may involve the use of both
multimodal and multilingual retrieval methods. The primary task at this year’s ImageCLEF is an ad hoc retrieval
task in which fifty topics were selected for retrieval and described using a topic title and narrative. Only the title
is translated into Dutch, Italian, Spanish, French, German and Spanish, and therefore suitable for CLIR. As
coordinators of this task, we found that assessors used both the image and the caption during their judgment for
relevance, and therefore we know that this task involves more than just CLIR. Further challenges include: (
There are essentially three stages to Systran: analysis, transfer and synthesis. The first stage, analysis, preprocesses the source text and performs functions such as character set conversion, spelling correction, sentence segmentation, tokenisation, and POS tagging. Also during the analysis phase, Systran performs partial analysis on sentences from the source language, capturing linguistic information such as predicate-argument relations, major syntactic relationships, identification of noun phrases and prepositional phrase attachment using their own linguistic formalism and dictionary lookup.
After analysis of the source language, the second process of transfer aims to match with the target language through dictionary lookup, and then apply rules to re-order the words according to the target language syntax, e.g. restructure propositions and expressions. The final synthesis stage tidies up the target text and determines grammatical choice to make the result coherent. This stage relies heavily on large tables of rules to make its decisions. For more information, consult [2] and [6]. 3.
Assessing the quality of the output produced by a machine translation (MT) system offers a challenging problem to researchers. Organisations such as DARPA and NIST have established the necessary resources and framework in which to experiment with, and evaluate, MT systems as part of managed competitions, similar to the TREC (see, e.g. [7]) and CLEF (see, e.g. [4]) campaigns. For manual evaluation1, three dimensions upon which to base judgment include translation adequacy, fluency and informativeness. Translation quality is normally assessed across an entire document when measuring fluency and informativeness, but adequacy is assessed between smaller units (e.g. paragraphs or sentences) which provide a tighter and more direct semantic relationship. To assess adequacy, a high quality reference translation and the output from an MT system are divided into segments to evaluate how well the meaning is conveyed between the versions. Fluency measures how well the translation conveys its content with regards to how the translation is presented and involves no comparison with the reference translation. Informativeness measures how well an assessor has understood the content of a translated document by asking them questions based on the translation and assessing the number answered correctly.
Given titles from the ImageCLEF test collection in Chinese, Dutch, French, Spanish, German and Italian; we first passed these through the on-line version of Systran to translate them into English, the language of the ImageCLEF document collection. We then asked assessors to judge the adequacy of the translation by assuming the English translation would be that for submission to a retrieval system for an ad hoc task. Translators who had previously been involved with creating the ImageCLEF test collection were chosen to assess translation quality because of their familiarity with the topics and the collection, each assessor given topics in their native language. Translators were asked to assess topic titles2 in the source language with the Systran English version and make a judgment on how well the translation captured the meaning of the original (i.e. how adequate the translated version would be for retrieval purposes). A five-point scale was used to assess translation quality, a score of 5 representing a very good translation (i.e. the same or semantically-equivalent words and syntax), to very bad (i.e. no translation, or the wrong words used altogether). Assessors were asked to take into account the “importance” of translation errors in the scoring, e.g. for retrieval purposes, mis-translated proper nouns might be considered worse than other parts-of-speech.
Table 1 shows an example topic title for each language and translation score for very good to good (5-4), okay (3) and bad to very bad (2-1) to provide an idea of the degree of error for these adequacy scores. We find that assessment varies according to each assessor; some being stricter than others, which suggest that, further manual assessments may help to reduce subjectivity. In some cases, particularly Spanish, the source language title contains a spelling mistake which obviously affects translation quality. Some assessors allowed for this in their rating, others did not, therefore suggesting the need to manually check all topics for errors prior to evaluation.
language
Adequacy rating 4-5
Although most accurate (and most subjective), manual evaluation is time-consuming and expensive, therefore automatic approaches to assess translation quality have also been proposed, such as the NIST mteval3 tool. This approach divides documents into segments and computes co-occurrence statistics based on the overlap of word n-grams between a reference translation produced manually and an MT version. This method has been shown to correlate well with adequacy, fluency and informativeness because n-grams capture both lexical overlap and syntactic structure [3].
In the latest version of mteval, two metrics are used to compute translation quality: IBM’s BLEU and NIST’s own score. Both measures are based on n-gram co-occurrence, although a modified version of NIST’s score has been shown to be the preferred measure. These scores assume that the reference translation is of high quality, and that documents assessed are from the same genre. Both measures are also influenced by changes in literal form, such that translations with the same meaning but using different words score lower than those that appear exactly the same. This is justified in assuming the manual reference translation is the “best” translation possible and the MT version should be as similar to this as possible. For n-gram scoring, the NIST formula is: N all wi...wn
that co-occur Score = ∑ n=1
∑ (
number of occurrences of w1...wn-1
Info(wi...wn) = log 2 number of occurrences of w1...wn The NIST formula uses info(w1…wn) to weight the “importance” of n-grams based on their length, i.e. that longer n-grams are less likely than shorter ones, and reduces the effects of segment length on the translation score. The information weight is computed from n-gram counts across the set of reference translations. The brevity penalty factor is used to minimise the impact on the score of small variations in the length of a translation. The mteval tool enables control of the n-gram length and maximises matches by normalising case, keeping numerical information as single words, tokenising punctuation into separate words, and concatenating adjacent non-ASCII words into single words.
In our experiments, we make the assumption that the English topic title is the reference translation, rather than ask the translators to produce an English version from an original. For example, the English topic title “North Street St Andrews” was translated into French as “La rue du Nord St Andrews”. We assume that if this were translated into English again, the “best” translation would be “North Street St Andrews”. Given that the translators used in the manual assessment were those who created the non-English translations from the English titles in the first place, we feel this assumption can be justified.
Because manual assessment is based on translation adequacy for retrieval, the Systran version “The street of North St Andrews” (a literal interpretation of the French version) is given a high adequacy rating even though it differs in syntax from the reference translation “North Street St Andrews”. The result is that the NIST score for a larger n-gram length would be low and not correlate with the score given manually (see Table 1 for more examples). Therefore to minimise this we compute the NIST score for an n-gram length of 1 word, reducing the measure to simply counting word overlap. In this case, the weighting function has the effect of reducing the importance of those words occurring frequently, e.g. function words. Table 2 shows example translations and their corresponding NIST score for Chinese translations. To use mteval, we created a reference containing the English versions of the topic titles where each title represents a segment within a document, and a test file containing the Systran versions in the same format.
8.1294 3.3147 1.727 Mountain scenery People dancing Picture postcards by the valentine photographic company
Mountain scenery dances people the Tanzania photography company photographs scenery postcard
At Sheffield, we have implemented our own version of a probabilistic retrieval system called GLASS, based on the “best match” BM25 weighting operator (see, e.g. [5]). Captions were indexed using all 8 fields, which include a title, description, photographer, location and set of manually assigned index categories and the default settings of case normalisation, removal of stopwords and word stemming.
To improve document ranking using BM25, we used an approach where documents containing all query terms were ranked higher than any other. We first identified documents containing all query terms, computed the BM25 score and ranked these highest, followed by all other documents containing at least one query term, again ranked by their BM25 score. The top 1000 images and captions returned for each topic title formed our entry to ImageCLEF. Evaluation was carried out using the set of relevant images for each topic (qrels) which forms part of the ImageCLEF test collection and the NIST information retrieval evaluation program, trec_eval4. We evaluate retrieval effectiveness using average precision for each topic, and across topics mean average precision (or MAP) is used.
Figure 1 shows a stacked bar chart of manual assessment scores obtained across each language for each topic. Each bar represents a topic and a maximum bar height of 30 would represent each assessor rating the translation as very good. As expected, the quality of translation is dependent on the topic title, although the majority of topics do get an overall rating that is less than 50-66% of the maximum possible value. The 6 topics with the highest overall manual rating (over 25) are topics 3 (Picture postcard views of St Andrews), 22 (Ruined castles in England), 43 (British windmills), 45 Harvesting), 47 (People dancing) and 49 (Musicians and their instruments). The 2 lowest scoring topics (an overall score < 15) are topics 34 (Dogs rounding-up sheep) and 48
English: Dogs rounding up sheep Italian: Dogs that assemble sheep German: Dogs with sheep hats Dutch: Dogs which sheep bejeendrijven French: Dogs gathering of the preois Spanish: Dogs urging on ewes Chinese: Catches up with the sheep the dog
Museum exhibits Exposures in museums Museumaustellungssteucke Museumstukken Exposure of objects in museum Objects of museum no translation
Ruined castles in England Ruins of castles in England Castle ruins in England Ruin of castles in United Kingdom Castles in ruins in England Castles in ruins in England Become the ruins the English castle 30 25 20 15 10 5 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 Chinese
Dutch
German
French Italian
Spanish
Chinese appears to exhibit the greatest variation of scores, and from Table 3 has one of the lowest average rating scores (Dutch being the lowest). The Chinese Systran translations are on average the shortest and 14% of the topics get a rating of very bad (3rd highest), and 28% a rating of very good (the lowest). From Table 3, Italian has the highest average manual rating, followed closely by German and Spanish suggesting these are strong bilingual pairings for Systran. French has the highest number of topics rated very poor, followed by Chinese and Italian which is perhaps surprising as French-English is claimed to be one of Systran’s strongest translations. Upon inspection, many of these low scores are from words which have not been translated. Italian has the highest number of topics rated very good, followed by German then French. Spanish has fewest topics given a very poor rating.
Avg manual score
Avg
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 Chinese
Dutch
German
French
Italian
Spanish
Table 4 shows the translations with a zero NIST score where the reference and Systran translations have no words which overlap. In many cases, however, this is simply because different words are used to express the same concept, or lexical variations of the word (such as plurals) are used instead. For information retrieval, this is important because if a simple word co-occurrence model is used with no lexical expansion, the queries may not match documents (although in some cases the lexical variations will recover these). This highlights one of the limitations of using mteval for assessing translation quality in CLIR, particularly when the queries are short.
Woodland scenes Scottish marching bands Tea rooms by the seaside Portraits of Mary Queen of Scots Boats on Loch Lomond Culross abbey Road bridges Ruined castles in England Portraits of Robert Burns Glasgow before 1920 Male portraits The mountain Ben Nevis Churches with tall spires Men holding tennis racquets A coat of arms British windmills Waterfalls in Wales Harvesting Museum exhibits Woodland scenes Waterfalls in Wales Harvesting Mountain scenery Glasgow before 1920 Male portraits Harvesting Welsh national dress Museum exhibits Woodland scenes Tea rooms by the seaside Wartime aviation People using spinning machines British windmills Harvesting Welsh national dress People dancing Museum exhibits Woodland scenes Wartime aviation Male portraits Museum exhibits Mountain scenery Woodland scenes Road bridges Men cutting peat Harvesting Museum exhibits Mountain scenery
Forest scenery no translation Seashore teahouse no translation In Luo river Mongolia lake ships Karohs overhaul Daoist temple Highway bridge Becomes the ruins the English castle no translation no translation Men’s portrait Nepali Uygur peak no translation çº¹ç« England’s windmill Well’s waterfall Harvests translation Has the high apex the churches Scenes of forests Water falls to the country of Scales Harvest Panorama mountaineer No translation Portraets of men Harvests Walisi tract Museumaustellungsstuecke Scene of a forest It knows it from te’ on lungomare Air in time of war Persons who use a filatoio English flour mills Harvesters Dressed traditional Welshman Persons who dance Exposures in museums A forest Aviators in time military Picture of a man Objects of museum Vista of mountains bunch faces Viaducts Trurfstekers harvest Museumstukken Mount landscapes These differences also contribute to the lack of correlation between the manual and automatic assessments (shown in Table 3). For Chinese Systran sometimes produces no translation (given a manual score of 1), and there appear to be more cases when the translation has gone seriously wrong. For Dutch, erroneous translations are caused also caused by the incorrect translation of compounds (which also occurs in German). The most highly correlated scores are between the assessments for French (using Spearman’s rho) suggesting that topics which receive a high manual assessment also receive a high automatic score, thereby confirming the use of an automatic evaluation tool to assess translation quality for CLIR (particularly for French). The correlation between manual and automatic results is not consistent across languages, however, where, for example, the correlation for Italian is lowest and not significant (at p<0.01). From Table 4, many Italian translations are rated highly by manual assessment and the kinds of translations suggest that the problem derives from the inability of mteval to determine semantic equivalents between translations.
Figure 3 shows a graph of recall versus precision across all topics and for each language using the strict intersection5 set of ImageCLEF relevance judgments. The graph follows a typical pattern showing that as the number of relevant documents found increases (recall), the precision decreases as relevant documents appear in lower rank positions. As with other results from CLIR experiments, the monolingual results are higher than those for translated queries, showing that these do not retrieve as well. Chinese has the lowest precision-recall curve, and is noticeably lower than the rest of the languages which seem to bunch together and follow a similar shape. The French curve is the highest of the languages, which matches with Table 3 where French has the lowest NIST score, the least number of topics with a zero NIST score, and a high proportion of topics with a high manual assessment rating.
1
Topics which perform poorly include 4 (seating inside a church), 5 (woodland scenes), 29 (wartime aviation), 41 (a coat of arms) and 48 (museum exhibits). These exhibit average NIST scores of 2.63, 0.64, 2.80, 3.71 and 3.83 respectively, and manual ratings of 3, 3.7, 4.17, 3.5 and 1.83 respectively. In some cases, the translation quality is high, but the retrieval low, e.g. topic 29, because relevance assessment for cross language image retrieval is 5 Strict intersection is the smallest set of relevance documents including only those which co-occur between assessors and marked as relevant (not including those judged as partially relevant). based upon the image and caption. There are cases when images are not relevant, even though they contain query terms in the caption, e.g. the image is too small, too dark, the object of interest is obscured or in the background, or the caption contains words which do not describe the image contents (e.g. matches on fields such as the photographer, or notes which provide background meta-information).
2
We might expect MAP to correlate well with the NIST score for the GLASS system because both are based on word co-occurrences, but it is interesting to note that retrieval effectiveness is correlated just as highly with the manual assessments, even though correlation between the manual and automatic assessments is not always itself high. This is useful as it shows that as a CLIR task, the quality of translation has a significant impact on retrieval thereby enabling, in general, retrieval effectiveness to indicate the quality of translation. Remaining factors may be due to relevance assessments, the IR system, pseudo relevance feedback or use of other retrieval-enhancing methods.
We have shown that cross language image retrieval for the ImageCLEF ad hoc task is possible with little or no knowledge of CLIR, or requirement of linguistic resources. Using Systran as a translation “black-box” requires little effort, but at the price of having no control over translation or being able to recover when translation goes wrong. In particular, Systran provides only one translation version which may not be correct and would provide better CLIR if several alternatives were output. There are many cases when proper names are mistranslated, words with diacritics not interpreted properly, and words translated incorrectly because of the limited degree of context. Because the task of CLIR does not necessarily require syntactic correctness, we find Systran can be used successfully for translation between a wide range of language pairs where essentially we make use of only the large dictionaries maintained by Systran.
We evaluated the quality of translation using both manual assessments, and an automatic tool used extensively in MT evaluation. We find that quality varies between different languages for Systran based on both the manual and automatic score which is correlated, sometimes highly, for all languages. There are limitations, however, with the automatic tool which would improve correlation for query quality in CLIR evaluation, such as resolving literal equivalents for semantically similar terms, reducing words to their stems, removing function words, and maybe using a different weighting scheme for query terms (e.g. weight proper names highly). We aim to experiment further with semantic equivalents using Wordnet, and also assess whether correlation between the manual and automatic scores can be improved by using longer n-gram lengths.
Using a probabilistic retrieval system, we obtain a mean average precision score which is 75% of the monolingual score. Although Chinese retrieval is lowest at 51%, this would still provide multi-lingual access to the ImageCLEF test collection, albeit needing improvement. Also, given that the task is not purely text, but also involves images, this score may be improved using content-based methods of retrieval. We aim to experiment with pseudo relevance feedback, and in particular improve performance using query expansion based on
As a retrieval task, we have shown that translation quality does affect retrieval performance because of the correlation between manual assessments and retrieval performance, implying that in general, higher translation quality results in higher retrieval performance. We have also shown that for some languages, the manual assessments correlate well with the automatic assessment suggesting this method could be used to measure translation quality given a CLIR test collection.
We would like to thank members of the Natural Language Processing group and Department of Information Studies for their time and effort in producing manual assessments. Thanks also to Hideo Joho for help and support with the GLASS system, and in particular his modified BM25 ranking algorithm, and thanks to NTU for providing Chinese versions of the ImageCLEF titles. This work was carried out within the Eurovision project at
[6] Systran. The SYSRAN linguistics platform: A software solution to manage multilingual corporate knowledge. White paper. 2002. http://www.systransoft.com/Technology/SLP.pdf