Natural languages contain many ambiguous words. Detecting the correct sense of words within documents and queries could potentially improve the performance of an information retrieval system. This is the major motivation for the Robust WSD tasks of the Ad-Hoc Track of the CLEF 2009 campaign. For these tasks we have build a customizable and exible retrieval system. The best performing con guration of this system is based on research in the area of axiomatic information retrieval approaches. Further, our experiments show that con gurations that incorporate WSD information into the retrieval process did outperform those without. For the monolingual task the performance di erence is more pronounced than for the bilingual task. Finally we are able to show that our query translation approach does work e ectively, even if applied in the monolingual task.
For the CLEF2009 challenge we customized our retrieval system which has been developed for
the CLEF2008 tasks, see [
Results show, that the best performing runs are based on axiomatic retrieval approaches. Further, runs incorporating WSD information did outperform those without, whereas for the monolingual task the performance di erence is more pronounced than for the bilingual task. Finally, we are able to show that our query translation approach does work e ectively, even if applied in the monolingual task.
The paper is structuered as follows: The next section provides a detailed description of our system. In section 3 their results of the various evaluation runs are presented and the main observations are discussed. Finally section 4 concludes our ndings. 2
Our information retrieval system consists of multiple separate components, that can be split into two groups. The rst group of component processes and parses the input sources - articles and additional resources - and build the retrieval indices. The second group of components takes these indices together with the queries as input to retrieve and rank relevant documents. 2.1
The document index is build using the collection of articles from the Los Angeles Times (1994) and
the Glasgow Herald (1995) supplied by the organizers of the Robust WSD Task. These articles
have already been tokenized, lemmatized, contain POS tags and are annotated with senses using
WordNet synsets. These senses are computed using two di erent word sense disambiguation
systems - labeled UBC [
For indexing we used Lucene, which is an open-source search engine library implemented in Java. A single Lucene index can consist of multiple elds, that can be seen as separate indices, each with its own dictionary and statistics. We exploited this feature and for each article we created a single document that contains multiple elds. From the articles we only took the article body. The headline of the articles were not processed as they did not appear to contribute to the relevance of the articles judging by results of the experiments made with our CLEF2008 system. No stop word removal was applied in the indexing stage. 2.1.1
Using WordNet and the annotated sense of ambiguous terms it is possible to determine the
synonyms for a speci c sense. The relation between synonymous word are one of many semantic
relatedness relationship types between words. Statistical methods provide unsupervised means to
detect word pairs with a high semantic relatedness, without restriction to a speci c relationship
type. One of these methods is based on the co-occurrence statistics of words within a corpus.
Many algorithms have been proposed to accomplish this task, using di erent weighting functions
to measure the relationship between words. The Pointwise Mutual Information (PMI) has been
been found to provide good performance in this regard [
The calculated similarity between words - estimated through their distribution in a corpus
- can be used to enrich the retrieval approach. In our system, we utilized a query expansion
technique based on the ndings in [
Field Name Word-Form Lemma Stems Synonyms (NUS) Synonyms (UBC) Synset IDs (NUS) Synset IDs (UBC) Cooccurrence Terms co-occurrence statistics uses a modi ed PMI measure for the similarity between two word based on the occurrence probability Pwi of word wi:
SCondP MI (wi; wj ) = log2 P P(w(wijwj)j) log2( P (1wj) ) 2.1.2
Each article is represented in the retrieval system using the following di erent elds. Word-Form From the articles the word form for each token was taken as indexing term. The tokens were only marginally processed - non-letter characters were ignored and diacritical signs were removed from the letters.
Lemma The lemmas for each token were taken to build this eld. No further processing has been applied to these terms.
Stems The word form tokens were stemmed using the snowball stemmer2 and indexed in its own eld.
Synonyms For each token of the articles the synset with the highest score was selected. For this synset all synonyms were listed using the MIT Java Wordnet Interface3. These synonyms were added to a dedicated eld. Thus, this eld contains all synonyms of the most probable sense according to the WSD annotations.
Synset IDs This eld was lled similar to the Synonyms eld, but using only the Synset ID of the highest ranked sense of each token. This can be seen as representation of the article in the WordNet Synset ID feature space. 2http://snowball.tartarus.org/ 3http://projects.csail.mit.edu/jwi/
Entries 2896802 1304243 language. The multilingual index can be created using various multilingual resources. We used two resources in our system, the Wikipedia4 and the Europarl corpus5. Both di er largely in their characteristics, such as domain and number of distinct terms. 2.2.1
The free encyclopedia Wikipedia is an e ort of many voluntary contributors and is continuously growing. There exist various editions in di erent languages that also contain links between corresponding articles. We exploited this link infrastructure to automatically build a multilingual index for all query languages, namely English and Spanish. The articles contained in the XML dumps6 provided by Wikimedia organization were parsed the Wikipedia Java API 7. The Wikipedia multilingual index thus nally contains aligned articles that are available in the two target languages. 2.2.2
Additionally to the Wikipedia another multilingual resource was used. The Europarl corpus[
Table 2 gives an overview of the two multilingual indices. The Wikipedia index consists of whole articles whereas the Europarl index is build out of sentences. One can observe that there is huge gap in the number of terms between the two resources. 2.2.3
The goal of the multilingual index is to nd the best matching terms in a language that is di erent
to the original language of a input term. This is achieved using information retrieval techniques.
For each term to be translated, which can either be a singe word or a phrase, a query is build. This
query is then used to search for relevant documents in the source language. From this result set
the unique identi ers and the score of the top hits are collected. Using the identi er information
the version of the hits in the target language are retrieved. From these documents the translated
terms are extracted. The intuition behind this procedure is similar to selecting terms for query
expansion using the top ranked documents in pseudo relevance feedback methods[
We implemented two scoring algorithms for selecting the best translation for the input term. The rst is a simple heuristic based on the well known TF IDF weighting scheme. For each term the weight wi is calculated using the score of the most relevant documents D: wT F IDF = log( i
N docF reqi + 1 + 1)
D X scorej j
The intuition behind the second scoring algorithm is to maximize the likelihood that a term has caused the document to be relevant. To accomplish this the same formula that is used to calculate 4http://en.wikipedia.org/wiki/Main Page 5http://www.statmt.org/europarl/ 6http://download.wikimedia.org/backup-index.html 7http://matheclipse.org/en/Java Wikipedia API the score of a document in the source language is applied on all target language terms found in the most relevant hits. The aggregated di erence between the actual score and the reconstructed score serves as base for the weight of a single term: wreconstruction = i
1 PjD jtfi;j log( docF reqi+1 + 1)
N scorej j + 1 2.3
The rst step of the query processing is the selection which parts of the topics are used for the queries. In all our experiments we used the title and description part. The narrative section of the topics was not included in the query generation process. 2.3.1
Both the title and the description part of the topics do not only contain the word form of the tokens, but also o er a lemmatized version and annotations for the sense of the terms. As with the articles the sense information is also available from two di erent algorithms for the English topics. For the Spanish topics a rst sense heuristic was applied by the organizers. Using the available features of the topics our system can be con gured to generate di erent types of queries. Each of these query types are generated to search in the according elds of the CLEF article index. For example the synonyms for the query terms are searched in the synonyms eld of the articles. Word-Form The word form of the tokens in the topics and description elements of the topics are used to create the query Lemma The lemmatized version of the tokens are taken to build the query terms Stems The word form of the tokens were processed using the same stemming algorithm using for the articles (the Spanish version of the Snowball stemmer was used for the Spanish topics) Synonyms From the top scored synset of each token in the topics the synonyms according to the
English WordNet were selected.
Synset IDs The identi er of the synset with the highest score was used as query term Cooccurrence Terms For all stems in the query the terms with the highest co-occurrence weight are selected for query expansion 2.3.2
If the language of the topic di ers from the languages of the articles, the query terms are individually translated. This is done using the Wikipedia and Europarl multilingual indices. For each of the two indices a weighted list of translated terms was generated and then normalized between 0 and 1 using the highest score as denominator. The sum of the two normalized scores for each term was then used as nal weight for the translation candidates. The top n candidates were then added to the query as translation for a single query term. Using the training topics and relevance judgments we found that using only the two highest scoring translation terms to o er the best overall performance. 2.4
The result of the query generation is an unordered list of terms extracted out of a topic de nition.
In the next step relevant documents are retrieved and ranked. The TFIDF [
SBM25(Q; D) =
X t2Q\D k1((1 b) + b tft;D docLengthD averageDocLength ) + tft;D log
N
docF reqt + 0:5 docF reqt + 0:5
For our main experiments we have chosen to apply ndings in the area of axiomatic approaches
to information retrieval. Fang and Zhai present in [
X ( t2Q\D
N docF reqt )
tft;D tft;D + 0:5 +
docLengthD averageDocLength
Using the training topics we found the setting of 0:25 for performance. 2.5
and 0:75 for
to provide a satisfying
Due to the fact that the Robust WSD Task is not only an information retrieval task but also
a question answering (QA) task we experimented also with methods from that eld [
The rationale to use the POS graph is that we experienced a stylistic similarity between
the answersing documents and the questions. The POS graph was thereby constructed for each
document by a xed number of nodes (17 POS-Tags). For each co-occurring POS tag within a
sentence an edge was introduced or the appropriate edge weight was increased by one. The same
procedure was also applied to each query. Based on the trainingset and the spectral di erence as
de ned in [
The main motivation for the Robust WSD task is to measure the performance impact of using word sense disambiguation as part of a information retrieval system. A rst step to determine the in uence of WSD information is the creation of a state-of-the-art retrieval system that does not incorporate a disambiguation process. We tried to build such a system and then use the WSD information as an optional processing step using query expansion. The results of these two system con gurations should provide insights into the in uence of word sense disambiguation. To further increase the validity of the observed behavior we also report the performance of our system using query expansion based on co-occurrence term statistics. All reported performance gures were calculated using 160 test topics and relevance assessments.
8http://nlp.uned.es/ jperezi/Lucene-BM25/ Table 3 gives an overview of the results of the baseline system using the di erent token features of the topics. The best performance is achieved using the stemmed version of the word forms. Therefore in all following evaluation runs we report only the performance of the con guration that is based on the stemmed tokens of the topics.
In table 4 di erent retrieval functions are compared using the CLEF2009 test collection, using the stemmed tokens of the title and the description of the topics without query expansion. Although this comparison gives no insights into the question whether WSD information could improve the performance, it demonstrates that the results of the axiomatic approach is indeed a valuable contribution to the arsenal of information retrieval techniques. The according GMAP metric is improved over the BM25 run, which indicates that especially low performing topics did improve using the axiomatic approach.
For the comparison with the con gurations that utilize the WSD information we only report the performance gures achieved using the axiomatic retrieval function. Table 5 lists the performance metrics of the various query expansion con gurations. The best performing con guration combines the synonym, synset and term co-occurrence information. The performance gures do show that integrating the words sense disambiguation data into the retrieval process of our system does improve performance. Not only does the baseline con guration bene t from the sense annotations, but also the con guration that already uses a (successful) query expansion technique is improved further. The di erence between the two WSD data sets (NUS and UBC) and between the Synonym and the Synset features are too small to allow any conclusions. The p-values are calculated via a Wilcoxon signed rank test using R9 and re ect whether the improvement over the baseline (or the query expansion using co-occurrence statistics for the last two runs) is statistically signi cant. 3.2
For the Spanish topics of the Robust WSD task we added the translation step into the query processing as described in section 2.2.3. This processing step is executed prior to the query expansion step. The baseline performance of our system using this con guration is listed in table 6. Using another language for the queries than the languages used for the documents has clearly a negative e ect on the performance of our system10. Using the stemmed version yields the best performance.
As in the monolingual task the axiomatic retrieval function outperforms the other retrieval 9http://www.r-project.org/ 10We had considerable trouble processing the Spanish topics as their did contain numerous encoding errors, leading to a worse performance
Query Expansion MAP Synonyms (NUS) 0.4061 Synonyms (UBC) 0.4036 Synset IDs (NUS) 0.4047 Synset IDs (UBC) 0.4070 Cooccurrence Terms 0.4170 Cooccurrence Terms + Synonyms + Synset IDs (NUS) 0.4222 Cooccurrence Terms + Synonyms + Synset IDs (UBC) 0.4212 functions. The results for the Spanish topics are listed in table 7 using the same parameters as for the monolingual runs.
For the next evaluation runs we added the WSD information to our retrieval system, which again resulted in a performance improvement, see table 8. The gap between the best con guration and the baseline is just about 1%. The di erence between the con guration that incorporate the WSD information are not statistically signi cant better than their respective baseline. 3.3
For our nal evaluation runs we investigated the impact of the query translation step. The
motivation for this are the ndings in [
In order to investigate the in uence of words sense disambiguation in the area of cross language retrieval we built a system that can be operated in a number of con gurations. This system was designed in a way to also study the performance of di erent retrieval functions. Additionally to the well known TFIDF weighting scheme and the BM25 ranking function we adapted a retrieval function that has been developed using an axiomatic approach to information retrieval. This Retrieval Function TFIDF1 TFIDF2 BM25 Axiomatic
MAP 0.1992 0.2445 0.2713 0.2885
GMAP 0.0472 0.0665 0.0658 0.0746 method did provide the best performance, not only for the monolingual task, but also for topics that are formulated in a language other than the language of the documents. For the bilingual retrieval task we developed a translation mechanism based on the freely available Wikipedia and the Europarl corpus.
In our evaluation runs we have found that incorporating the word sense disambiguation information does indeed improve the performance of our system by a small margin. This was the case for the monolingual and the bilingual task, although for the bilingual task the improvements are not statistically signi cant. Also when using the WSD information additionally to an existing query expansion technique the performance was further improved. In none of our tests we observed that the performance did decrease when applying the word sense disambiguation information. Just on a few queries there has been a negative impact. The reason for this and possible means to detect and to avoid poor performing queries are still open questions and require further research.
The Know-Center is funded within the Austrian COMET Program - Competence Centers for Excellent Technologies - under the auspices of the Austrian Federal Ministry of Transport, Innovation and Technology, the Austrian Federal Ministry of Economy, Family and Youth and by the State of Styria. COMET is managed by the Austrian Research Promotion Agency FFG.