We present a report on our participation in the Indonesian-English bilingual task of the 2005 Cross-Language Evaluation Forum (CLEF). We chose to translate an Indonesian query set into English using a commercial machine translation tool called Transtool, instead of using freely available resources for Bahasa Indonesia on the Internet which are not as complete as those for English. We show that improvement in retrieval effectiveness can be obtained using a query expansion technique.
2.1 We participated in the bilingual task with English topics. The English document collection contains 190,604 documents from two English newspapers, the Glasgow Herald and the Los Angeles Times. We opted to use the query title and the query description provided with the query topics. 1 See http://www.geocities.com/cdpenerjemah/.
The query translation process was performed fully automatic using Transtool. Using the query titles, the average length of the Indonesian queries was 3.1 words; the average length of the original English queries was 2.6 words; and the average length of the translated English queries was 2.7 words. Using the query descriptions, the average length of the Indonesian queries was 12.1 words; the average length of the original English queries was 9.5 words; and the average length of the translated English queries was 11.3 words. The number of Indonesian words that cannot be translated into English was 10 for the query titles and 26 for the query descriptions. We then applied a pseudo relevance-feedback query-expansion technique to the queries that were translated using the machine translation tool. We used the top 20 documents from the collection to extract the expansion terms. The terms that were used to expand the query were noun only terms. We used the Monty Tagger2 to identify noun terms in those top 20 documents.
In these experiments, we used Lucene3 information retrieval system which is based on the vector space model [4] to index and retrieve the documents. 4
Our work focused on the bilingual task using Indonesian queries to retrieve documents in the English collections. Table 1 shows the result of our experiments.
0.2810 0.2364 0.3508
CLIR (translation) 0.1582 0.1731 0.1830 % Change - 43.70% - 26.77% - 47.83%
The retrieval performance of the title-based translation queries dropped 43.70% below that of the equivalent monolingual retrieval (see Table 1). The retrieval performance of the description-based translation queries dropped 26.77% below that of the equivalent monolingual queries. The retrieval performance of using a combination of query title and description dropped 47.83% below that of the equivalent monolingual queries.
Query translation using machine translation (title) 0.1582 (0%) 2 See http://web.media.mit.edu/~hugo/montytagger/. 3 See http://lucene.apache.org/.
Query translation using machine translation (description) 0.1731 (0%) 0.0936 (-45.92%)
Next, the translated description queries were then expanded using noun terms from the top 20 documents using the pseudo relevance feedback technique. Adding 10 noun terms reduced the retrieval performance by 45.92% and adding 20 noun terms reduced the retrieval performance further by 47.60% (see Table 3).
Query translation using machine translation (description + title) 0.1830 (0%) 10 terms added
Our results demonstrate that the retrieval performance of queries that were translated using machine translation for Bahasa Indonesia was about 53%-74% of that of the equivalent monolingual queries. The pseudo relevance feedback that is commonly used to improve the retrieval performance did not improve the retrieval performance. In fact, the longer the query is the worse the effect of using the query expansion technique. In our experiments, adding noun terms to the translated queries dropped the retrieval performance to 21%-47% of that of the equivalent monolingual queries. With such a short time available, we were not able to try different approaches to this task. We hope that we will obtain better results in our next participation in CLEF. 5