We describe Amharic-English cross lingual information retrieval experiments in the adhoc bilingual tracs of the CLEF 2006. The query analysis is supported by morphological analysis and part of speech tagging while we used different machine readable dictionaries for term lookup in the translation process. Out of dictionary terms were handled using fuzzy matching and Lucene[4] was used for indexing and searching. Four experiments that differed in terms of utilized fields in the topic set, fuzzy matching, and term weighting, were conducted. The results obtained are reported and discussed.
representation using SERA1. The transliteration was done using a file conversion utility called g2 2 which is available in the LibEth3 package.
We designed four experiments in our task. The experiments differ from one another in terms
of query expansion, fuzzy matching, and usage of the title and description efilds in the topic sets.
Details of these is given in the Experiments section. Lucene [
The paper is organized as follows, section 1 gives an introduction of the language under consideration and the overall experimental setup. Section 2 deals with the query analysis which consists of morphological analysis, part of speech tagging, filtering as well as dictionary lookup. Section 3 reports how out of dictionary terms were handeled. It is followed by the setup of the four retrieval experiments in section 4. Section 5 presents the results and section 6 discusses the obtained results and gives concluding remarks. 2
The dictionary lookup requires that the (transliterated) Amharic terms are first morphologically analyzed and represented by their lemmatized citation form. Amharic, just like other Semitic languages, has a very rich morphology. A verb could for example have well over 150 different forms. This means that successful translation of the query terms using a machine readable dictionary will be crucially dependent on a correct morphological analysis of the Amharic terms.
For our experiments, we developed a morphological analyzer and Part-of-speech tagger for Amharic, and were used as the rfist pre-processing step in the retrieval process. We used the morphological analyzer to lemmatize the Amharic terms and the POS-tagger to filter out less content bearing words. The 50 queries in the Amharic topic set were analyzed and the morphological analyser had an accuracy of 86.66% and the POS tagger 97.45%. After the terms in the queries were POS tagged, the filtering was done by keeping Nouns and Noun phrases in the keyword list being constructed while discarding all words with other POS tags.
Starting with tri-grams, bi-grams and nfially at the word level, each remaining term was then
looked up in the an Amharic - English dictionary [
For the terms that were found in the dictionaries, we used all senses and all synonyms that were found. This means that one single Amharic term could in our case give rise to as many as up to eight alternative or complementary English terms. At the query level, this means that each query was initially maximally expanded. 3
Those terms that where pos-tagged as nouns and not found in any of the dictionaries were
selected as candidates for possible fuzzy matching using edit distance. The assumption here is
that these words are most likely cognates, named entities, or borrowed words. The candidates
were rfist filtered by counting the number of times they occurred in a large (3.5 million words)
Amharic news corpus. If they occur in the new corpus (in either their lemmatized or original
form) more frequently than a predenfied threshold value of 10 4, they would be considered likely
1SERA stands for System for Ethiopic Representation in ASCII,
http://www.abyssiniacybergateway.net/fidel/serafaq.html
2g2 was made available to us through Daniel Yacob of the Ge’ez Frontier Foundation (http://www.ethiopic.org/
3LibEth is a library for Ethiopic text processing written in ANSI C http://libeth.sourceforge.net/
4It should be noted that this number is an empirically set number and is dependent on the type and size of the
corpus under consideration
to be non-cognates, and removed from the fuzzy matching unless they were labeled as cognates
by an algorithm specifically designed to find (English) cognates in Amharic text [
The set of possible fuzzy matching terms was further reduced by removing those terms that
occurred in 9 or more of the original 50 queries assuming that they would be remains of non
informative sentence fragments of the type ”Find documents that describe...”). When the list of fuzzy
matching candidates had been nfially decided, some of the terms in the list were slightly modiefid
in order to allow for a more ”English like” spelling than the one provided by the transliteration
system [
The retrieval was done using the Apache Lucene, an open source high-performance, full-featured
text search engine library written in Java [
Four experiments were designed and run using Lucene. 4.1
The translated and maximally expanded query terms from the title and description fields of the Amharic topic set were used in this experiment. In order to cater for the varying number of synonyms that are given as possible translations for the terms in the queries, the corresponding synonym sets for each Amharic term were down weighted. This is done by dividing 1 by the number of synonyms in each set and giving those equal fractional weights that adds up to 1. An edit distance based fuzzy matching was used in this experiment to handle cognates, named entities and borrowed words. 4.2
The above experiment is repeated in this one except the usage of only the title field in the topic set. This is an attempt to investigate how much the performance of the retrieval is affected with and without the presence of the description field in the topic set. 4.3
Up
In this experiment, both the title and description fields were used and is similar to the rfist experiment except that fuzzy matching terms were given much higher importance in the query set by boosting their weight by 10. 4.4
This experiment is designed to be used as a comparative measure of how much the fuzzy matching affects the performance of the retrieval system. The setup in the first experiment is adopted here, except the use of fuzzy matching. Cognates, named entities and borrowed words, which so far have been handled by fuzzy matching, were treated manually. They were picked out and looked up separately and all translations for such entries are manual. 5
We have been able to get better retrieval performance for Amharic compared to runs in the previous two years. Linguistically motivated approaches were added in the query analysis. The topic set has been morphologically analyzed and POS tagged. Both the analyzer and POS tagger were trained with a large news corpus for Amharic, and performed very well when used to analyze the Amharic topic set. It should be noted that these tools have not been tested for other domains. The POS tags were used to remove non-content bearing words while we used the morphological analyzer to derive the citation forms of words.
The morphological analysis ensured that various forms of a word would be properly reduced to the citation form and be looked up in the dictionary rather than being missed out and labeled as an out-of-dictionary entry. Although that is the case, in the few times the analyzer segments a word wrongly, the results are very bad since that entails that the translation of a completely unrelated word would be in the keywords list. Especially for shorter queries, this could have a great effect. For example in query C346, the phrase ’grand slam’, the named entity ’slam’ was analyzed as ’s-lam’, and during the dictionary look up ’cow’ was put in the keywords list since that is the translation given for the Amharic word ’lam’. We had a below median performance on such queries.
On the other hand, stop word removal based on POS tags by keeping the nouns and noun phrases only worked well. Manual investigation showed that the words removed are mainly noncontent bearing words.
The experiment with no fuzzy matching since all cognates, names, and borrowed words were added manually, gave the highest result. From the experiments that were done automatically, the best results obtained is for the experiment with the fully expanded queries with down weighting and using both the title and description efilds, while the worst one is for the experiment in which only the title fields were used. The experiment where fuzzy matching words were boosted 10 times gave slightly worse results than the non-boosted experiment. The assumption here was that such words that are mostly names and borrowed words tend to contain much more information than the rest of the words in the query. Although this may be intuitively appealing, there is room for boosting the wrong words. In such huge data collections, it is likely that there would be unrelated words matching fuzzily with those named entities. The decrease in performance in this experiment when compared to the one without fuzzy match boosting could be due to up weighting such words.
Further experiments with different weighting schemes, as well as different levels of natural language processing will be conducted in order to investigate the effects such factors has on the retrieval performance. representation ascii