This paper evaluates the effectiveness of a query translation and disambiguation as well as expansion techniques on CLEF Collections, using SMART Information Retrieval System. We focus on the query translation, disambiguation and methods to improve the effectiveness of an information retrieval. Dictionary-based method with a combination to statistics-based method is used, to avoid the problem of translation ambiguity. In addition, two expansion strategies are tested on their ability to improve the effectiveness of an information retrieval, an expansion via a relevance feedback before and after translation as well as an expansion via a domain feedback after translation.
Dictionary-based method, where each term or phrase in the query is replaced by a list of all its possible translations, represents a simple and an acceptable first pass for a query translation in Cross-Language
In our approach [
A simple stemming process of query terms was performed before the query translation, to replace each term with its inflectional root, to remove most plural word forms, to replace each verb with its infinitive form and to reduce headwords to their inflectional roots. The next step is a term-by-term translation using a bilingual machine-readable dictionary. An overview of the Query Translation and Disambiguation Module is shown in Fig 1.
Query Translation Module
Terms Stemming Term-by-term Translation
Two Terms Disambiguation Stop List
L e m m a s
Query terms n s l a t i o n s
C o o c c u r r e n c e One Language Stemmer Bilingual Dictionary
Monolingual Training Corpus
R In itreev rfaom la it o n Web Search
Engine
Fig 1. Query Translation and Disambiguation Module Phases 2.1
A Word is Polysemous, if it has senses that are different but closely related; as a noun, for example, right can
mean something that is morally approved, or something that is factually correct, or something that is due one. In
the proposed system, a disambiguation of the English translation candidates is performed, by selecting the best
English term, equivalent to each French query term, by applying a statistical method based on the co-occurrence
frequency. For the purpose of this study, we decided to use the mutual information [
Log 2
N
f (w1,w2) f (w1) f (w2)
Where N is the size of the corpus, f (w) is the number of times the word w occurs in the corpus and f (w1, w2) is the number of times both w1 and w2 occur together in a sentence bead. 3
Query expansion, which modifies queries using judgments of the relevance of a few highly ranked documents, has been an important method for increasing the performance of an information retrieval.
In this study, we have proposed two sorts of query expansion: a relevance feedback before and after the translation and disambiguation of query terms, and a domain feedback after the translation and disambiguation of query terms. 3.1
We apply an automatic relevance feedback, by fixing the number of retrieved documents and assuming the top–ranking documents obtained in an initial retrieval. This approach consists to add some term concepts, about 10 terms from a fixed number of the top retrieved documents (about 50 top documents), which occur frequently in conjunction with the query terms, on a presumption that those documents are relevant, to make a new query. One advantage of the use of a query expansion, such as an automatic relevance feedback is to create a stronger base for short queries in the disambiguation process, in the purpose of using co-occurrence frequency approach. 3.2
We introduced a domain feedback as a query reformulation strategy, which consists of extracting a domain field from a set of retrieved documents (top 50 documents), through a relevance feedback. Domain key terms will be used to expand the original query set.
The evaluation of the effectiveness of the French-English Information Retrieval System, was performed by using the following linguistic tools : Monolingual Corpus
The monolingual English part of the Canadian Hansard corpus (Parliament Debates) was used in the disambiguation process.
Bilingual Dictionaries
A bilingual French-English COLLINS Electronic Dictionary Data, version 1.0 was used for the translation of French queries to English. Missing words in the dictionary, which are essential for the correct interpretation of the query, such as Kim, Airbus, Chiapa was not compensated. We just kept the original source words as target translations, by assuming that missing words could be proper names, such as Lennon, Kim, etc… Stemmer and Stop Words
Retrieval System
SMART Information Retrieval System3 was used to retrieve English and French documents. SMART is a vector model, which has been used in many researches for Cross-Language Information Retrieval. 4.1
We submitted 4 runs for the bilingual (non-English) task with French as a topic language, and one run for the
ftp://ftp.cs.cornell.edu/pub/smart
RunindexORG RunindexTR
RunindexFEED
RunindexDOM
RunindexFR Our participation in CLEF 2001 showed two runs, which contributed to the relevance assessment pool: RunindexTR, the translation and disambiguation method, and RunindexFR, the monolingual French retrieval. The rest of bilingual runs were not judged, because of limited evaluation resources, the result files did not directly contribute to the relevance assessment pool. However, the runs were subject to all other standard processing, and are still scored as official runs. Table 1 shows the average precision for each run. 4.3
In our previous research [
However in this study, the submitted bilingual runs did not show any improvement in terms of average precision for a query expansion via a relevance feedback before and after translation or a domain feedback after translation. The proposed translation and disambiguation method RunindexTR, achieved 85.30% accuracy of the monolingual performance RunindexORG (English information retrieval) and 77.23% of the monolingual performance RunindexFR (French information retrieval). This accuracy is higher than that of relevance feedback or domain feedback, as shown in Table 1. The relevance feedback before and after translation RunindexFEED showed a second best result in term of average precision, 78.69% and 71.25% of the monolingual English and French retrieval, respectively. RunindexDOM, the domain feedback showed a less effective result in terms of average precision, with 76.92% and 69.64% of the monolingual English and French retrieval, respectively. Fig 2 shows the precision-recall curves for the submitted runs to CLEF 2001.
These results were less effective than we expected. This is because; first the bilingual dictionary does not
cover technical terms and proper nouns, which are the most useful in improving the total IR accuracy. In this
study, the French query set contains 11 untranslated English terms. Using Collins French-English Bilingual
dictionary as the only resource for term translations puts the burden of discovering the right translation of proper
nouns, which are used in Clef query set. When a word is not found in the dictionary, we just kept the original
source word as a target translation one. This method should be successful for some proper names, such as
Lennon, Kim, etc…but not for others, such as Chiapa. Missing words in the bilingual dictionary is one major
reason to introduce noise into our results. The second reason is due to the selection of terms to expand the
original queries, domain keywords for a domain feedback or terms that occur most often with the original query
terms. This made the expansion methods ineffective, comparing to our previous work [
Domain Feedback A combination of the proposed domain feedback method to a relevance feedback before or after translation could be a solution, to improve the effectiveness of an information retrieval.
0.4 0.3 ll cae 0.2 R 0.1
RunindexORG
RunindexTR RunindexFEED RunindexDOM
RunindexFR 0 0 0.2 0.4
0.6 Precision 0.8 1
Fig 2. A Precision-Recall curves for the submitted Bilingual and Monolingual runs
Our conclusion at this point can only be partial. We still need to perform more experiments to evaluate the proposed query expansion methods. The purpose of our investigation was to determine the efficacy of translating and expanding a query by different methods. The study compares the retrieval effectiveness using the original monolingual queries, translated and disambiguated queries and the alternative expanded user queries on a collection of 50 queries, via a relevance feedback or a domain feedback techniques. An average precision measure is used as the basis of the experiments evaluation.
However, the proposed translation and disambiguation method showed the best result in terms of average precision, comparing to the query expansion methods: via a relevance feedback before and after query translation and disambiguation and via a domain feedback after query translation and disambiguation.
What we presented in this paper is a rather simple study, which highlights some areas in Cross-Language Information retrieval. We hope to be able to improve our researches and find more solutions to fulfill the needs for Information retrieval cross languages.
This work is partially supported by the Ministry of Education, Culture, Sports, Science and Technology, Japan, under grants 11480088, 12680417 and 12208032, and by CREST of JST (Japan Science and Technology).