In this paper we explore the use of semantic classes in an Information Retrieval system in order to improve the results in the RobustWSD task at CLEF 2009. We use two ontologies of semantic classes (WordNet domain and Basic Level Concepts) to re-rank the retrieved documents and obtain better recall and precision. Finally, we implement an innovative method to weight the expanded terms taking into account the ones of the original query terms and their relations in WordNet with respect to the new ones which have demonstrated to improve the results.
The two main goals of the Robust-WSD last year’s edition task were to measure the robustness of the retrieval systems (good stable performance over all queries), and test the benefits of the use of Word Sense Disambiguation (WSD) on this kind of systems.
For our participation in the second edition, we decided to employ a system
already implemented and evaluated in the last year edition as starting point,
the one of Universidad Complutense de Madrid [
Our main strategy consists of experimenting the benefits semantic classes in Information Retrieval (IR) systems. Moreover, we propose an innovative and flexible way of weighting terms for the query expansion based on WordNet relations.
WSD, can be defined as the task of assigning the correct sense to words
depending on the context in which they appear. It is a challenging task, difficult to
be addressed and, despite the long time it is been studied, the results of
state-ofart WSD systems are still a long way to be useful in other Natural Language
Procesing applications, as shown in last international evaluations [
Answering. On the other hand, it seems that many word–sense distinctions are too subtle to be captured by automatic systems with the current small volumes of word–sense annotated examples. This could be a reason of the poor results of current WSD systems.
A possible solution that has been explored is the use of semantic classes instead of word senses. The task of WSD consists of assigning the proper semantic class to each ambiguous word, instead of its word sense. The use of semantic classes has several advantages. Firstly, they provide richer and more useful information than word senses. For example, for IR could be more informative that the word church belongs to the semantic class building, instead of knowing that the correct sense to that word is the 1. Secondly, the average polysemy of texts is decreased with the use of semantic classes; in fact they can group in the same class several senses of a concrete word. Therefore, the classification task is simplified, and finally, the amount of training data for each classifier is increased because semantic classes can group together senses of different words, senses of the same word, and also senses of word of different morphological categories. As a consequence, the number of examples to train each classifier is increased in semantic class approaches, and the problem of the lack of data in alleviated.
In [
We are convinced that IR could take advantage of the use of word sense disambiguation, from a semantic class point of view instead from the traditional word sense point of view. Due to the data of the robust adhoc IR task has been processed automatically by two WSD systems, and the information of word senses is available, we do not run any class–based WSD system over the data.
This paper is organized as follows: the next section describes the architecture of our system. In section 3 we discuss the results of this system at CLEF 2009 Robust-WSD Task. Finally, in section 4 we draw the conclusions and future works. 2
The system architecture is shown in Figure 1.
As we can see in Figure 1, the user query is pre-parsed to obtain a set of terms without stopwords and any special symbol. Next, a ranked list of relevant documents are retrieved using Lucene search engine1. With the retrieved documents, the initial query, the relations of the external resource WordNet and state-of-art query expansions methods an expanded query is obtained. The terms of this new query are weighted taking into account the weights of the original query words, their relations in WordNet with respect to the new ones, the weight assigned by the WSD system to each sense and the weight returned by the expansion method. Once we obtain a new list of weighted terms, we do, again, a search but this time using the expanded query instead of the original one in order to retrieved a new ranked list of documents. Finally, we employ the semantic class information from two semantic resources (WordNet Domains and Base Level Concepts) in order to obtain a re-ranked document list as result.
The following sections present each of this processes in more detail. 2.1
As we mentioned previously, the search engine we employ is the one provided
by the Universidad Complutense de Madrid . Its implementation is a modified
version of Lucene which uses the BM25 probabilistic model [
Bo1 model since this approach obtains the best results. We also decided to use the same constant values than they used in the last CLEF Robust-WSD edition in order to compare the effectiveness of our methods of semantic classes.
As we can see in the Figure 1, we make two search processes. For the first retrieval process, query terms are lemmatized and stemmed in order to increase the system recall. The first Search module gets these terms as input and returns a list of relevant documents using the BM25 probabilistic model. Furthermore, in the Query expansion module, we expand the original query obtaining new terms by means of the Bo1 model.
Even if [
We defined synseti,1 as a WordNet synset given and synseti,d as another WordNet synset which is related to the synseti,1 of a distance of d jumps (taken into acount only hyperonym and hyponym relations). Thus, weight(synseti,d) is the weight of the synset i, d and weight(synseti,1) is the score given by the WSD system to the synset i, d. α is a constant whose value is between 0 and 1 and d the distance of synseti,d to the synseti,1.
Once we calculated the previous synset weight, we combine this weight with the weight assigned by the expanded method bo1 in order to calculate the final term weight using the following equation: weight(termt) = weight(synseti,d) + weight0(termt) 2 (2)
Where weight(termt) is the weight of the expanded term t which is grouped in the WordNet synset i, d, and weight0(termt) is the weight assigned by bo1 to the term t.
Using these equations, we give importance to those expanded terms closely related with the original query terms and, in addition, we include the score given by the WSD system for each query term in the final term weight. Thus, we include all senses of a term in the search giving more importance those terms which are relationed with more likely senses and closer to the original query terms. 2.2
Our approach consists on mapping the assigned word senses to semantic classes, specifically to WordNet Domains labels and Basic Level Concepts.
WordNet Domains [
Basic Level Concepts [
As a result, Basic Level Concepts typically occurs in the middle of hierarchies and less than the maximum number of relations.
The authors developed a method for the automatic selection of BLC from WordNet. They use a very simple method for deriving a small set of appropriate meanings using basic structural properties of WordNet. The approach considers: – The total number of relations of every synset or just the hyponymy relations – Discard those BLCs that do not represent at least a number of synsets. – Optionally, the frequency of the synsets (summing up the frequency of the senses provided by WordNet).
The process of automatic selection of BLC follows a bottom-up approach using the chain of hypernym relations. For each synset in WN, the process selects as its Base Level Concept the first local maximum according to the relative number of relations. For synsets having multiple hypernyms, the path having the local maximum with higher number of relations is selected. Usually, this process ends with a number of fake Base Level Concepts. That is, synsets having no descendants (or with a small number) but being the first local maximum according to the number of relations considered. Thus, the process concludes checking if the number of concepts subsumed by the preliminary list of BLC is higher than a certain threshold. For those BLC not representing enough concepts according to a certain threshold, the process selects the next local maximum following the hypernym hierarchy.
Thus, depending on the type of relations considered to be counted and the threshold established, different sets of BLC can be easily obtained for each WN version. For our work, we selected the set of BLC built using all kind of relations and a threshold of 20 as the minimum number of synsets that each BLC must subsume.
We explain now the representation of documents or queries with semantic classes of the words contained on them. In the task data, each ambiguous word is annotated with their possible senses, each one with a certain probability. Starting from this information, we create a domain vector, for a query or for a document, containing all the semantic classes information of the query or document. The domain vector consists in a vector which, each element, represents a WordNet Domain or a Basic Level Concept and its associated weight. Note that there are 165 Domain Labels in WordNet Domains and 558 Basic Level Concepts for nouns. The way to build this vector is: each word has annotated several senses, with the associated probability; each word sense is mapped to its proper semantic class, and the element of the vector corresponding to this domain is increased with the probability associated to the word sense. After processing all terms, we obtain a domain vector representing the semantic information of the document or query. Finally to compare two documents, or a document and a query, and obtain their similarity in terms of their semantic content, we use the value of the cosine defined by the two domain vectors. 2.3
Once the final list of documents from the expanded query is retrieved, the Semantic re-ranking module re-arrange this list taking into account both the similarity returned by the BM25 probabilistic model and the similarity calculated by semantic class system. In order to do this, we studied the following 5 different equations: semsim1(i, j) = simij ∗ semij semsim2(i, j) = semsim3(i, j) = semsim4(i, j) = 8 simmaxi + simij < : simij 8 (simmaxi + simij ) ∗ semij < : simij 8 simmaxi + simij ∗ semij < : simij if semij > h otherwise if semij > h otherwise if semij > h otherwise semsim5(i, j) = simmaxi ∗ semij + simij (3) (4) (5) (6) (7)
Where semsimx(i, j) is the final similarity between the query i and the document j using the method x, simij is the similarity of the query i with respect to the document j returned by the search engine, semij is the same similarity but returned by the semantic class system, simmaxi is the greatest value of similarity returned by the search engine for the query i and h is a constant which determines a semantic similarity threshold defined empirically.
As both similarity values (simij and semij) are normalized ones, our first approximation was the equation 3. In this equation we simply multiply both values (the similarity returned by the search engine and the similarity obtained by the semantic class system) in order to obtain a new similarity value between the query i and the document j. With this equation both values have the same importance.
The equation 4 was thought in order to put those documents with a certain level of semantic relation with the query above other ones. Therefore, if the similarity obtained by the search engine will be summed to the greatest similarity if, and only if, the semantic class similarity between the document and the query is greater than a threshold h, otherwise only the BM25 similarity will be taken into account.
The next equation (5) is based on the previous one, however the semantic similarity multiplies the sum of the BM25 similarity for the document j and the greatest similarity. The main idea of this equation is to overlap some of the less semantic related documents which exceed the threshold with those which do not exceed.
In order to give more relevance those documents with high semantic similarity but taking into account the semantic class score in the final similarity value, the equation 6 was used.
Finally, the equation 7 multiplies the greatest similarity value with the document semantic similarity and, next, sums this result to the search engine document similarity. This equation tries to improve the search engine similarty value using the semantic similarity as reduction factor of maximum similarity value. 3
In this section we report the results of each one of our proposals.
For the evaluation of the Expanded query terms weighter, we set the value for two variables: α and d (distance). In order to get the best values for these variables, we experimented with several values for them. Table 1 presents two of the best results of those experiments. With α = 0.8 and d = 1 we improve the baseline GMAP in a 9.97%. With α = 0.92 and d = 6 we improve both the baseline MAP in a 0.02% and the baseline GMAP in a 8.19%. The results of BM25+Bo1+WD (α = 0.92, d = 6) correspond to our CLEF experiment named ali02wsd.
The results of our second proposal, the Semantic re-ranking module, depend on the function used for the integration of the documents’ weights given by the semantic classes and the search engine. In addition, for some of the integration functions, the variable h (threshold) has to be set too. We experimented with those functions and different values for the threshold. For each integration function we obtained its best threshold value (when needed), as can be seen in Tables 2 and 3. As we can see in Table 2, some of the results of different integration functions are the same. In the case of RR2, RR3 and RR5, this occurs because the best results they can reach are the same as the baseline results. This integration functions do not improve the system. In the case of RR1 and RR4, they mathematical function are the same, except the second one, that can be affected by the threshold. Both of them obtain the best results for the WND model for all measures. Table 3 presents the results of the integration with the BLC20 model. The best results are obtained by the RR2, RR3 and RR4 integration functions. These are the functions that use a threshold. Note the threshold that gives the best results is the same for all them. Thus, we can deduce that the threshold h = 0.8 is the ideal for determining if a semantic class is relevant or not.
For our final comparison, we chose the best integration function for each model (WND, BLC20), as shown on Table 4 and Figure 2. The results of BM25+Bo1+WND correspond to our CLEF experiment named ali01wnd.
The integration of the semantic classes to the search engine improves the baseline results. With WND we improve both the baseline MAP in a 0.4% and the baseline GMAP in a 0.31%. With BLC20 we improve both the baseline MAP in a 0.64% and the baseline GMAP in a 1.77%. 4
The results of the experiments with our two proposals demonstrate improvements to the initial IR system. Regarding the Expanded query terms weighter module, we experimented with the weights of the terms in a probabilistic IR system. We have applied a smoothing function based on the WordNet distance to the weights given by the IR system. The experiments show GMAP improvements of nearly 10% but not significant MAP improvements.
As future work we propose to continue with the experiments on this module. For the propagation function 2, the search of the best values for α and d can be more exhaustive, finding better values for this variables. Moreover, new relations can be explored in WordNet (not only hyponyms and hyperonyms), in order to improve recall. Even new weight propagation functions can be proposed to better exploit the concept of distance in WordNet.
Regarding our second proposal, the Semantic re-ranking module, we have integrated the semantic classes to a IR system. We carried out this integration recalculating the weight of the documents retrieved depending on the similarity between the semantic class of each document and the semantic class of the query. The results of the experiments made reveal that the semantic classes resources can be effectively be integrated to the IR systems.
This module can also be led to new levels. We only used five simple integration functions for the search engine and the semantic classes weights. More functions can be studied with the purpose of finding the best way to integrate the available resources of semantic classes.
This paper has been partially supported by the next projects: “Question Answering Learning technologies in a multiLingual and Multimodal Environment (QALL-ME)” (FP6 IST-033860), “Intelligent, Interactive and Multilingual Text Mining based on Human Language Technologies (TEXT-MESS)” (TIN200615265-C06-01) and “Virtual Observatory for Technology Transfer”. We also want to thank the support given by “El Taller Digital”.