This article describes an experimental investigation on the use of resources from the web on a common Natural Language Problem (NLP) problem that of Word Sense Disambiguation (WSD). In particular we use our disambiguation experiments with statistical query translation on a Greek-English cross language retrieval system using Google's n-grams. Results from our participation on the Ad-Hoc TEL track of CLEF 2009 are reported. Medas1 is a Greek-English cross language retrieval system that aims to support Greek users in the medical domain, to overcome the language barrier. It contains two subsystems: a multilingual subsystem, for retrieving bilingual documents (a collection of scientific articles in medicine available in the Greek web) and a cross language subsystem, which provides only the interface to the MEDLINE database using the PubMed search engine. Medas contains a dictionary based translation module and uses the MeSH thesaurus for on-line reformulation of the queries. Preprocessing the queries includes language identification, tokenization, capital-tolower letter conversion, stopword removal and stemming. Our participation in the Ad-Hoc multilingual retrieval track of CLEF this year is a first attempt to evaluate our system on a general-purpose (non-medical) set of queries. A Cross Language Information Retrieval (CLIR) system needs an online and shallow translation system. This translation step tends to cause a reduction in cross language retrieval performance as compared to monolingual retrieval. Such an approach phases the barrier of the lexicon and has to overpass the polysemy problem in languages. Thus the translation is achieved in two phases: a translation and a disambiguation phase. For the translation of the initial query we shall use a bilingual term list and the disambiguation phase is based on the target language model, which is based on the statistical properties of n-grams. For query translation a Greek-English bilingual term list is utilized as the main source of knowledge and Google's n-grams collection [1] is used for WSD to acquire the most appropriate translation. The remainder of this paper is organized as follows. In the next section we briefly review the work on GreekEnglish CLIR. We continue in section 3 with the WSD using the web as a corpus. In section 4 we present our experimental framework and our experimental results. Conclusions are summarized in the final section.
Introduction
translation using dictionary-based method [
Preprocessing Query includes language identification, tokenization, capital-to-lower letter conversion, stopword removal and stemming. This forces us to keep a stemmed copy of the dictionary. The dictionary contains single words and phrases –idioms, that is two or more words which translate to a single term and vice versa. One of the main factors that limit the performance of retrieval is the limited coverage of the dictionary used in query translation. Ambiguity in translation of queries is one of the major causes for large drops in effectiveness below monolingual performance, for the dictionary-based method in CLIR.
Figure 1 presents the methodology that we used to perform the translation of the queries. Two sets of
experiments were performed. In the first set we used as possible translations the translations that were derived
from the dictionary whereas in the second we embodied in the candidate translations the synsets taken from
WordNet [
In the following paragraphs of the current section we briefly describe the algorithms used for the word sense disambiguation, that is the language modelling and the maximum entropy model.
The quality of n-gram language models depends directly on the size of the training texts. Thus the web has become a valuable resource in order to build better statistical language models.
By the term n-grams we define a sequence of n single words. In language models each word depends only on the context of k words. By the definition of the n-gram language models they have their own limitations as they can support dependency ranges up to (n − 1) tokens.
When estimating the probability of a translation candidate, it is impossible to use maximum likelihood estimates on the entire string, by using the chain rule p(e) = p(e1) p(e2 | e1) p(e3 | e1, e2 )...p(en | e1,..., en−1) where e is a string in target language of length n, as one would run into serious data sparseness issues. Instead most language modeling approaches use a limited horizon of two words, called tri-gram models: Despite the limited context that is taken into account they still form a good compromise between prediction quality, robustness and computational tractability.
The second WSD method used is based on the calculation of the entropy of the translations with respect to the
language model in the web [
n H ( X ) = −K ∑ p(i) log p(i)
i=0
where K is an arbitrary constant which depends on the problem, i is a fragment of message X of length n and,
p(i) is the probability of the i-th fragment. In the case of machine translation, the message is represented by the
translation, and each fragment i corresponds to the i-th trigram of translation ti. The probability of each trigram
is calculated by means of web counts. Let’s define the i-th trigram ti = (wi, wi+1, wi+2) and its root bigram bi =
(wi, wi+1). Let’s name as c(x) the function that returns the number of Google 5-grams that contain the text
fragment x. According to [
m∈M
Google released a collection of English n-gram data in August 2006. This data has been collected from the web pages and contains billions of n-gram types up to a maximum order of 5. The data are available from the Linguistics Data Consortium as a set of 6 DVDs. All together, the compressed n-gram data is about 24 GB in size, the uncompressed version takes more than 90 GB of space.
In all our runs we used only the titles of the topics. Thus we had very short queries of 2.5 words in average each, which made the disambiguation step difficult in certain cases. The average length of the translated queries using only our Greek-English lexicon, before disambiguation, was 7,6 words. We submitted 5 runs: two of these used a 2-gram language model for disambiguation, two used a 3-gram model and the final was used a maximum entropy model. Our results are summarized in figure 2 (mean values of precision at the 11 recall values). As a conclusion we observed that the 2-gram model was best performed although in general our results were very poor compared to other participants.