We investigated dictionary based cross language information retrieval using lexical triangulation. Lexical triangulation combines the results of different transitive translations. Transitive translation uses a pivot language to translate between two languages when no direct translation resource is available. We took German queries and translated then via Spanish, or Dutch into English. We compared the results of retrieval experiments using these queries, with other versions created by combining the transitive translations or created by direct translation. Direct dictionary translation of a query introduces considerable ambiguity that damages retrieval, an average precision 79% below monolingual in this research. Transitive translation introduces more ambiguity, giving results worse than 88% below direct translation. We have shown that lexical triangulation between two transitive translations can eliminate much of the additional ambiguity introduced by transitive translation.
2.1 (2000)).
The underlying IR system used in the Sheffield submission was the GLASS system (Sanderson
The translation resources were derived from the German, Spanish, Dutch, and English
components of EuroWordNet (
Given that the intention of this work is to examine CLIR using simulated Machine Readable
Dictionaries, the choice of EuroWordNet (
The intention of the EuroWordNet project was to develop a database of WordNets for a number
of European languages similar to, and linked with, the Princeton WordNet 1.5 (
By developing a series of WordNets for European languages, and linking them to the original
Princeton 1.5 WordNet for English, EuroWordNet has created a structure similar to the controlled
vocabulary thesaurus used by Salton as described by
In EuroWordNet, the relationships between the synsets of the various component languages and
the Princeton 1.5 WordNet synsets3 can take many forms. These include, for example, the eq_hyponym4
relation, which relates more general to more specific concepts (
Our work used EuroWordNet to generate structures to simulate a Machine Readable Dictionary. The only relationships used in the construction of the dictionary tables, were the eq_synonym and eq_near_synonym relationships. These are by far the most restrictive and precise of the possible relationships.
The eq_synonym relationship records the fact that the language synset is synonymous with the
WordNet synset. EuroWordNet introduced the eq_near_synonym relationship to record the fact that
certain terms that share a common hypernym (more general concept) are closer in meaning than others.
In this situation the co-hyponyms (more specific terms) that are closely related are close enough in
meaning that they could be used for translation purposes, but are not synonymous and are therefore not in
the same synset. This closeness is represented by linking the synsets with an eq_near_synonym
relationship (
For each language used from EuroWordNet, two tables were generated. The first mapped lemmas to the synset-ids of the synsets related by eq_synonym or eq_near_synonym. The second maps synset-ids to their constituent lemmas (i.e. related by eq_synonym or eq_near_synonym). As we will explain below, these tables are used to parameterise the translation process.
2 The Sheffield University Computer Science Department was a collaborator in the EuroWordNet project and Wim Peters of that department kindly made extracts from EuroWordNet available for this research.
3 In EuroWordNet terms the Inter Lingual Index or ILI.
4 The relationships in EuroWordNet have names on the form eq_relationship_name the eq_ indicates that the relationship involves some degree of “equality”. 2.3
Query processing was fully automatic and the queries were generated using all parts of the topics. The queries were passed through a series of processes as follows: • • • • • • •
Parsing - The conversion of the topics to queries which makes use of title, description and narrative fields.
Normalisation - all characters were reduced to the lower case unaccented equivalents (i.e. “Ö” reduced to “o” and “É” to “e” etc.) in order to maximise matching in both the lemmatisation and translation processes.
Lemmatisation - The various inflected forms of the query words were reduced to a canonical lemma
form to enable matching with the German EuroWordNet translation resources. A table derived from
the CELEX German database was used to determine the appropriate lemmata5 for a wordform.
German compound words were split using a simple algorithm. The algorithm looks for a series of
wordforms that will match with the whole compound. If such a complete match is found the
corresponding lemmata of the wordforms are returned. The algorithm takes account of the use of “s”
as “glue” in the construction of German compounds. This approach was based on the description of
the word reduction module in
German Stop Word Removal - A stopword list, generated from the CELEX German database, was used to remove words in the query that carried little meaning and would otherwise introduce noise to the translation. The stop-word lists contain all of the German words marked as articles, pronouns, prepositions, conjunctions or interjections in the CELEX database.
Translation - The translation process used tables derived from EuroWordNet to translate between two languages. The lemma to synset-id table for the first language and the synset to lemma table for the second language were used to map words in the first language to words in the second. All the possible translations through the intermediate synset-ids were returned. Three different translations were created for each query: a direct German to English translation, a transitive translation using Spanish as the intermediate language, and a transitive translation using Dutch as the intermediate language.
Merging - The results of the two transitive translation routes were merged to produce a fourth
translation, the triangulated translation. The merge process was conducted on an “original German
Lemma” by “original German Lemma” basis. The translations from each route for each lemma were
compared and only translations common to both routes were used to translate the lemma.
Retrieval – the translation and merging process produced four different versions of the queries
translated into English, these were submitted to the GLASS IR system which had been used to index
the English corpus. The GLASS system normalised both documents and queries to lower case, and
removed any English stopwords (using a standard English stop word list). Porter stemming (
5 The wordform to lemma table is a many-to-many mapping as a wordform may be a valid inflection of more than one lemma. 3.1 • • • •
We submitted 4 official runs to the CLEF evaluation process.
A “bilingual” run (shefbi), generated from the direct translation from German to English A “Spanish transitive” run (shefes), generated from the transitive translation using Spanish as the intermediate.
A “Dutch transitive” run (shefnl), generated from the transitive translation using Dutch as the intermediate.
And a “triangulated” run (sheftri), generated from the result of merging of the two transitive translations.
Only the triangulated run (sheftri) was judged and contributed to the relevance judgement pool.
In order to provide a baseline for comparison we conducted an additional English monolingual run using the same parsing and retrieval processes. This unofficial run is presented below to enable comparisons to be made.
In summary then, the experimental conditions were as follows:
Queries Corpus Relevance Judgements Corpus and Query Stemming Lemmatiser German Stop-words removed pre-translation Translation
Yes, Porter based Yes, including German Compound Splitting Yes, all articles, pronouns, prepositions, conjunctions or interjections from the CELEX German database.
Simulated Dictionary based, using lookup-tables derived from EuroWordNet eq_synonym and eq_near_synonym relations.
Only translations common to both transitive routes, 3.2
The table below shows the average precision for the 5 runs that made up the CLEF experiment. Only the cross language runs were submitted to the CLEF, and of those, only the triangulated run contributed to the pooled results.
Bilingual (shefbi) Triangulated (sheftri) Spanish Transitive (shefes) Dutch Transitive (shefnl)
The standard 11-point recall and precision curves for the 5 runs are shown below, the second graph shows only the 4 cross language runs.
Comparing the average precision of the monolingual run with the bilingual run we see that the
bilingual run is some 76%6 below the monolingual. This compares to the 60% below worst case reported
by
Taking next the two transitive runs, we observe a differential of -88% in the case of the Spanish
transitive run and -92% in the case of the Dutch transitive run relative to the bilingual run. Both of these
results are statistically significant at the 0.01 level under both the sign and Wilcoxon tests. These figures
are in line with the -92% differentials reported by
Comparing the triangulated run with the two transitive runs reveals the expected improvement in performance. The differentials for the two transitive runs relative to the triangulated run are -79% for the Spanish transitive run and -85% for the Dutch transitive. Both of these figures are statistically significant at the 0.01 level under both the sign and Wilcoxon tests.
There is also a statistically significant differential of -47% between the triangulated run and the bilingual in favour of the bilingual. This significance is at the 0.01level under both the sign and Wilcoxon tests. 4
In summary, these results support the results of
This work made use of relatively rich resources in the form of EuroWordNet. However, it remains to be seen if these results could be repeated using the poorer quality resources that are likely to be available for translating between less common pairs of languages.
As Samuel Johnson said “Dictionaries are like watches; the worst is better than none, and the
best cannot be expected to be quite true.” (
6 Statistically significant at the 0.01 level under both the sign and Wilcoxon tests.
Ballesteros, L. & Croft, W. B. (1997). "Phrasal translation and query expansion techniques for crosslanguage information retrieval". In: Proceedings of the 20th annual international ACM SIGIR conference on Research and development in information retrieval, pp. 84 - 91. Association for Computing Machinery. [Online]. Available: http://www.acm.org/pubs/articles/proceedings/ir/258525/p84-ballesteros/p84-ballesteros.pdf [29/02/2000].