The UNED phrase-based cross-language summaries were first introduced at iCLEF 2001 as a translation strategy which permitted faster document selection with roughly the same accuracy than full Machine Translation. For our iCLEF 2003 participation, we test the validity of our summaries as cross-language indexes for the retrieval stage of the interactive search process. We compare a reference system that performs query translation (and then retrieves target-language documents) with a proposed system that directly retrieves cross-language summaries with the source-language query. The performance of both systems is very similar, confirming that UNED summaries are viable for cross-language indexing. This approach is trivially scalable to more than one target language, opening an interesting path for truly multilingual search assistance.
Therefore, in interactive CLIR the question is no longer whether we should translate documents or not, but how to translate documents in a way that a) facilitates searching tasks to the user (document selection and query refinement); b) minimizes the computational cost of translation, and c) in the case of a document translation approach, that provides optimal material for the automatic retrieval stage in the user’s language.
In previous iCLEF editions, we have proposed a method of translation that reasonably satisfies requisites a and b. The method generates summarized translations of target-language documents which rely essentially on noun phrase extraction and translation. While rather crude from a Computational Linguistics point of view, such summaries have excellent features for Multilingual Information Access: • In iCLEF 2001, we obtained quantitative evidence that such cross-language summaries could be better for document translation purposes than full MT (users judgements were equally precise with both approaches, but summaries permitted faster judgments). • In iCLEF 2002, our noun-phrase based summaries proved to be useful as a basis for query formulation and refinement. • Phrase-based summaries contain only 30% as many words as the original documents, and can be generated more than one order of magnitude faster than full machine translations.
It seems, then, reasonable to think of phrase-based summaries as good candidates for a document translation approach to Multilingual Information Access. This is the hypothesis that we seek to test in our iCLEF 2003 experiment. The challenge is performing retrieval with phrase-based summaries as the only source of query-language indexes for target-language documents, because the size of the index set will be just one third of its monolingual or MT counterparts.
In Section 2, we review the main features of phrase-based summaries. In Section 3 we discuss the experiment design; in Section 4 we present the results of our experiment, and finally we draw some conclusions. 2
UNED approach to Cross-Language Search assistance (defined as the problem of assisting a user to search and detect relevant documents in a foreign-language text collection) is based on noun phrases as fundamental units for translation (either query or document translation) and formulation of user needs.
Cross-language pseudo summaries are an essential part of the approach. They simply consist
in the list of noun phrases present in the document, listed in order of appearance, and translated
according to a simple greedy algorithm that makes use of a database of bilingual alignments
between two and three-lemma phrases in the source and target languages [
spanish acuerdo de libre comercio acuerdos de libre comercio acuerdo libre comercio acuerdo de libre cambio acuerdos de libre cambio convenio de libre comercio convenios de libre comercio compromiso de libre comercio ...
english free trade agreement free trade accord free trade pact free trade beyond the pact free trade pacts free trade agreements free trade arrangements
Translation of non-aligned noun phrases (including phrases with more than three lemmas) is
done with a greedy algorithm that translates, at each step, the two or three-lemma sub-phrase
which has a better translation under the alignment resource, and uses overlapping phrases to
translate the remaining words taking the context into account. The algorithm is described in
detail in [
A comparative example of translations provided by Systran and by our system can be seen in the Appendix.
Spanish
1,700,183 1,953,849
288,872 347,920 The goal of the experiment is testing whether searching cross-language summaries with the original query can match searching the original documents with a translated version of the query, in an interactive CLIR setting. As reference system, we have chosen the best of the two approaches tested in our iCLEF 2002 experiment. In this approach, users interact with the system to formulate an optimal query as a set of noun phrases. Then, query translation is performed automatically (via the database of aligned phrases) and the retrieved set of documents can be examined via cross-language summaries. As the user do not have to deal with foreign-language expressions at any time, the translation and retrieval steps can be substituted for a direct retrieval on document translations without altering the interface with the user. This is very convenient for our experiment, because it permits a direct comparison of query translation versus document translation strategies without any additional interference.
A detailed description of both systems to be compared follows: 1. Initial query formulation: the user reads the topic description and formulates an initial query freely. The time for reading and typing this initial query is not computed as searching time. 2. Query formulation by phrases: the system suggests a maximum of 10 phrases related to the initial user query. The user can either a) select a number of them and perform the initial search, or b) type in some additional words and ask the system to recompute the phrase suggestions. 3. Document retrieval: • In the reference system (Query Translation) , phrases are translated into English via the phrase alignment dictionary, and then a search is performed against the LA Times collection. • In the contrastive system (Document translation), the original Spanish phrases are used to search the collection of Spanish summaries of LA Times documents. 4. Document ranking: The result of the search is a ranked list of LA Times documents, with a colour code to indicate whether each document has been judged as relevant, not relevant, unsure, or has not been judged yet. Each document is displayed in the ranked list as a Systran translation of its title. 5. Document selection: When the user clicks on a document title, its Spanish summary is shown to the user. The document can then be judged as relevant, not relevant or unsure. 6. Query refinement: There are two ways of refining the query: • Phrase feedback : If the user clicks on a phrase inside a document summary, the phrase is added to the query and the document ranking is updated with the enhanced query. • Direct reformulation: at any point during the search, the user can select/deselect additional phrases to the query, and can introduce new words to the phrase suggestion window.
Although the difference between both systems is transparent to the user, the architecture and implications of each of them are quite different. Figures 3 and 4 compare both approaches visually. The document translation approach can be trivially extended to more than one document language, as can be seen in Figure 5. Using phrase-based summaries, the document translation approach can be applied to a multi-language collection increasing its size in only 30% with respect to the original size per user language considered. For instance, a collection with four document/user languages would only double its size under this document translation approach. This is a challenge for the retrieval phase, because the set of indexes is much smaller than the original. Our hypothesis is that any possible difference in the quality of the rankings will not have an appreciable impact on the interactive searching task.
(”*” indicates interaction with the user)
We have used eight native Spanish searchers for our experiment, the LA Times 1994 collection as the document set, and the eight official iCLEF topics (in Spanish) extracted from the CLEF 2002 set. This year, our searchers do not interact with English at any moment, hence we were not specially cautious with English proficiency when recruiting volunteers. We could then focus on recruiting searchers with long experience using search engines (something that previous years was not granted). In the end, all eight users have medium English skills (which are not put into practice in the experiment) and are highly experienced in web searching.
As specified in the track guidelines, each search session consisted on a unique user/system/topic
combination according to a latin-square matrix design [
Official results The official Van Rijsbergen’s Fα = 0.8 measure (α = 0.8 favors precision rather than recall) can be seen in Figure 6. Both systems receive the same score, confirming our hypothesis that phrasebased summaries can be used for a cross-language indexing of the collection. Precision and recall are also almost identical for both approaches.
Query translation Document translation
.51 .53
.14 .14
Fα .29 .29
Figure 7 separates results by searcher and by topic. Topics vary in difficulty as expected, being topics 3 and 4 harder than the others. There is also variability between users, but part of this variability can be explained by the distribution of topics to systems assigned to each searcher. For instance, users 1 and 3 have opposite results (query translation is much worse for user 1 and much better for user 3), but they have an inverse assignment of topics and systems, and the two difficult topics are done with different systems from user 1 to user 3. Overall, the detailed results are a good sample of how the latin square design filters out possible topic/system/user combination bias. Document translation can be a viable approach to Multilingual Information Access once we find simplified, task-oriented ways of translating documents without the computational cost associated to commercial Machine Translation systems. In this experiment, we have proved that phrasebased summaries, although rather crude from a pure MT point of view, can be successfully used for Cross-Language searching.
Phrase-based summaries can be generated much faster than full machine translations, and occupy only 30% of disk space. In these conditions, phrase-based summaries can be used to benefit from the main advantages of document-translation approaches to interactive CLIR: • At document selection time, translations do not have to be generated on the fly, because all documents have been previously translated for indexing. Hence the interactive search process is not retarded by on-line translation. • In a truly multilingual setting (with more than one target language), the complexity of translating the query to several languages (a big impediment if query translation is done interactively) and the problem of merging ranked results from different languages dissappear.
In our present experiment, query formulation is somewhat restrained by the fact that the user has to formulate his/her query as a set of noun phrases that can in turn receive an appropriate automatic translation via the alignment resource. This makes perfect sense in the reference (query translation) system, as proved in our iCLEF 2002 experiment. But it might be an excessive constraint in the document selection approach. We plan to experiment whether a more classical, monolingual search interface (with the possibility of adding free terms at any moment of the search process) might even improve the results obtained by query translation approaches to foreignlanguage search assistance.
This is an example of the translation of a short LA Times document, both with Systran Professional 3.0 (as provided to iCLEF participants) and with phrase-based summaries.
WORLD CUP SOCCER ’94 / THE FIRST ROUND; SPOTLIGHT; NOT AGAINST BRAZIL Reuters news service sent a picture of Carlos Alberto Torres, the captain of Brazil’s 1970 World Cup championship team, talking with Lothar Matthaeus, captain of Germany’s team, at a recent practice at Southern Methodist University. The caption information included with the photo identified Torres as a German fan. ELLIOTT ALMOND
FU´TBOL ’94 / EL PRIMER REDONDO DE LA TAZA DEL MUNDO;
PROYECTOR; NO CONTRA EL BRASIL El servicio de noticias de Reuters envi´o un cuadro de Carlos Alberto Torres, el capit´an de Equipo 1970 del campeonato de la taza del mundo del Brasil, hablando con Lothar Matthaeus, capit´an del equipo de Alemania, en una pr´actica reciente en Methodist meridional Universidad. La informaci´on del subt´ıtulo incluida con la foto identific´o a Torres como ventilador alem´an. ALMENDRA DE ELLIOTT
Cross-lingual pseudo-summary copa del mundo de fu´tbol primera ronda
We are indebted to Anselmo Pen˜as for the provision of the WTB phrase extraction software. This research has been funded by the Spanish Government, project Hermes (TIC2000-0335-C03-01).
Searcher \ Topic 1 2 3 4 5 6 7 8 Avg.