We present a report on our participation in the Indonesian-English ad hoc bilingual task of the 2006 Cross-Language Evaluation Forum (CLEF). This year we compare the use of several language resources to translate Indonesian queries into English. We used several readable machine dictionaries to perform the translation. We also used two machine translation techniques to translate the Indonesian queries. In addition to translating an Indonesian query set into English, we also translated English documents into Indonesian using the machine readable dictionaries and a commercial machine translation tool. The results show performing the task by translating the queries is better than translating the documents. Combining several dictionaries produced better result than only using one dictionary. However, the query expansion that we applied to the translated queries using the dictionaries reduced the retrieval effectiveness of the queries.
2.1
Adding translated queries with relevant terms, known as query expansion, has been shown to improve CLIR effectiveness [1, 3, 12]. One of the query expansion techniques is called the pseudo relevance feedback [4, 5]. This technique is based on an assumption that the top few documents initially retrieved are indeed relevant to the query, and so they must contain other terms that are also relevant to the query. The query expansion technique adds such terms into the previous query. We apply this technique to the queries in this work. To choose the relevant terms from the top ranked documents we employ the tf*idf term weighting formula [10]. We added a certain number of terms that have the highest weight scores. 3 4
In the experiments, we used Lemur2 information retrieval system which is based on the language model to index and retrieve the documents.
We then applied a pseudo relevance-feedback query-expansion technique to the queries that were translated using the machine translation tool. We used the top 10 relevant documents retrieved for a query from the collection to extract the expansion terms. The terms were then added to the original query.
The results of our CLIR experiments were obtained by applying the three methods of translation, i.e., using the machine translations, using dictionaries, and using parallel corpus. Table 1 shows the result of the first technique, which shows that translating the English queries into Indonesian using Toggletext machine translation tool is better than using Transtool. Lastly, we also attempted to improve our CLIR results by expanding the queries translated using the dictionaries. We were unable to do the expansion process to all the translated queries because of time limitation. The results is as shown in Table 4, which indicates that adding the queries with 5 terms from the top-10 documents obtained from a pseudo relevance feedback technique hurt the retrieval performance of the translated queries. Our experiments demonstrate that translating queries using machine translation tools is better than translating documents. The retrieval performance of queries that were translated using machine translation tools for Bahasa Indonesia was about 14.28%-18.83% of that of retrieving the documents translated using machine translation. There was no significant difference in retrieval performance between the two machine translation tools that we used.
Taking the first definition in the dictionary when translating an English query into Indonesian appeared to be effective. The result of combining several dictionaries is much better than only using one dictionary. In order to improve the retrieval performance of the translated queries, we expanded the queries with the terms extracted from the top-10 documents. However, the pseudo relevance feedback technique that is known to improve the retrieval performance did not improve the retrieval performance of our queries. 4. Ballesteros, L, and Croft, W. Bruce. (1998). Resolving Ambiguity for Cross-language Retrieval. In Proceedings of the 21st International ACM SIGIR Conference on Research and Development in Information Retrieval (pp.64-71). 5. Attar, R. and Fraenkel, A. S. Local Feedback in Full-Text Retrieval Systems. Journal of the Association for
Computing Machinery, 24: 397-417, 1977. 6. Davis, Mark and Dunning, Ted. A TREC Evaluation of Query Translation Methods for Multi-Lingual Text
Retrieval. Ed. D. K. Harman. The Fourth Text Retrieval Conference (TREC-4), November. NIST, 1995. 7. Jones, Gareth and Lam-Adesina, Adenike M. Exeter at CLEF 2001: Experiments with Machine Translation for Bilingual Retrieval. In Proceedings of the Workshop of the Cross-Language Evaluation Forum (CLEF 2001), p. 105-114. Darmstadt, Germany, September 2001. 8. McCarley, J. Scott. Should We Translate the Documents or the Queries in Cross-Language Information
Retrieval?. In Proceedings of the Association for Computational Linguistics (ACL’99), p. 208-214, 1999. 9. Oard, D. W. and Hackett, P. G. Document Translation for Cross-Language Text Retrieval at the University of Maryland. In Proceedings of the Sixth Text Retrieval Conference (TREC-6), Gaithersburg MD, November 1997. 10. Salton, Gerard, and McGill, Michael J. Introduction to Modern Information Retrieval, New York: McGraw
Hill, 1983. 11. Sheridan, P. and Ballerini, J. P. Experiments in Multilingual Information Retrieval using the SPIDER System. Proceedings of the 19th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, Zürich, Switzerland, August 1996. 12. Xu, Jinxi and Croft, W. Bruce. Query expansion using local and global document analysis. In Proceedings in the 19th Annual International ACM SIGIR Conference on Research and development in Information Retrieval. 1996.