This paper describes our participation to the English Girt Task of CLEF 2005 Campaign. A method for conceptual indexing based on WordNet is used. Both documents and queries are mapped onto WordNet. Identified concepts belonging to WordNet synsets are extracted from documents and queries and those having a single sense are expanded. All runs are carried out using a conceptual indexing approach. Results prove a primacy of using queries from the title field of the topics and a slight gain of using stemming compared to the non stemming cases.
The objective of our participation to the English GIRT task in 2005, was to evaluate the use of a conceptual
indexing method based on the WordNet [
This paper is organized as follows. In section 2, we describe the synoptic scheme of our system which includes the Mercure search engine . In section 3, the tests required for conceptual indexing are formally described: the concept detection and weighting methods in 3.1, and the disambiguation-expansion method in 3.2. Section 4 reports the official evaluation results compared with the median average obtained by all participating systems. Finally, section 5 gives some conclusions and prospects.
In this section, we describe the conceptual indexing method based on WordNet. The principle involves, being
given a document (resp. a query), mapping it onto WordNet and then to extract the concepts (mono and multi
terms) that belong to WordNet and appear in the text of the document (resp. the query) [
WordNet 1 In the following, the word “document” will refer to both queries and documents in the collection. 2 WordNet is organised around the notion of Synset (Synonym set). Each Synset contains terms that are synonyms in a given context. Synsets are interrelated by different relations like Hypernymy (Is-a).
A total of all, five runs were carried out. They are described in Table2 of section 3.
In the next section we will explain the main steps of our system: the concept detection and weighting methods used to carry out our experiments.
I. Detail of the approach
D= {w1, w2, …, wn}
Dc= {c1, c2, …, cm, w’1, w’2,…,w’m’} Concept detection consists of extracting mono and multiword concepts from documents and queries that correspond to nodes (synsets) in WordNet. Formally, let consider: the initial document composed of n single words. The result of the concept detection process will be a document Dc. It corresponds to: (1) (2) where c1, c2, , cm are concepts recognized as WordNet entries. These concepts could be mono or multiword. It may also happen that single words w’1, w’2,…,w’m’ of the initial document (query) do not belong to the WordNet vocabulary. They will not be used for expanding the document (the query). However, they will be added to the group_president_and_chief_operating_officer_mike_cramer_called… group_president_and_chief_operating_officer_mike_cramer_called group_president_and_chief_operating_officer_mike_cramer group_president_and_chief_operating_officer_mike group_president_and_chief_operating_officer group_president_and_chief_operating group_president_and_chief group_president_and group_president …. chief_operating_officer_mike_cramer_called chief_operating_officer_mike_cramer_called chief_operating_officer_mike_cramer chief_operating_officer_mike chief_operating_officer mike_cramer_called mike_cramer_called …
Concept: "chief_operating_officer#n" detected final expanded document in order to be used at the search stage.
To detect concepts in the query, we use an ad hoc technique that relies solely on concatenation of adjacent words to identify compound (multiword) concepts in WordNet. In this technique, two alternative ways may be carried on. The first one would be projecting WordNet on the document : all WordNet multiword concepts are mapped onto the document and those occurring in it. This method has the advantage of creating a reusable resource (a document representation made out of WordNet concepts). Its drawback is the possibility to omit concepts which appear in the document and in WordNet under different forms. For example, if WordNet containts the multiword concept “solar battery”, a simple comparison with document would miss the same concept appearing in its plural form “solar batteries”. The second way, which we adopt in our experiments, follows an the opposite path, projecting the document onto WordNet: for each multiword candidate concept derived by combining adjacent words in the document, we first question WordNet using these words just as they are, and then we use their base forms if necessary.
Word are combined, as shown in Figure1, according to the longest succession of words for which a concept is detected. In the example of Figure1, the longest concept "chief_operating_officer#n" (#n is used for the POS name) is selected although "chief " and "officer" could also be identified as single word concepts. This concept is defined by WordNet as follow: chief executive officer, CEO, chief operating officer -- (the corporate executive responsible for the operations of the firm; reports to a board of directors; may appoint other managers (including a president)) Example of a document after its projection onto WordNet: In Figure 3 below, we can see a document example from the collection (named GIRT-EN19950120120), after its projection onto WordNet conceptual network. For example health_care_delivery#n is a concept that belongs to a WordNet synset identified in the document. Words that are not tagged (like "ddr" in this example) do not belong to WordNet terminology.
<DOC> <DOCNO> GIRT-EN19950120120 </DOCNO> <TITLE-EN> establishment#n and development#n of the health_care_delivery#n system#n in#n syria#n with regard_to#n morbidity#n especially#r infectious_disease#n </TITLE-EN> ddr syria#n asia#n health_care_delivery#n system#n arab#n country#n historical#a development#n near_east#n contagious_disease#n developing#n country#n epidemiology#n morbidity#n health#n policy#n descriptive#a study#n medical#n sociology#n health#n policy#n sociology#n of developing#n country#n developmental#a sociology#n </DOC>
The notations "#n", "#a", "#v", "#r" are used to indicate the part of speech (POS) of the terms belonging to WordNet. They refer respectively to names, adjectives, verbs and adverbs. For the moment, the POS is not used in the index. We need it only to expand the identified mono-sense WordNet terms.
As seen in the previous example, a large majority of the vocabulary used in the collection documents is covered by WordNet. Table1 summarizes the cover rate concerning both queries and documents. More than 92.87% of the vocabulary used in the documents is covered by WordNet and 99.39% (so almost totality!) of the vocabulary used in the queries is covered.
Concerning compound concepts (or multiterms), they represent about 9% for the document and only 7.83% (0.52 compound term in average) for the queries. Multiterms have often only one sense. It is important to use them in our case, as only mono sense terms from the documents and the queries are expanded in our approach.
The extracted concepts (single or multiwords) are then weighted as in the classical keywords case according to a
kind of TF.IDF which is also a variant of the OKAPI system [
Thus, a weight Weight(ti , d j ) of a term ti in a document d j is given by the following formula [
(3)
N tf ij * (h1 + h2 * log( ))
ni h3 + h4 * dl j + h5 *tf ij ∆d Where: tf ij : The frequency of the term ti in the document d j , h1 , h2, h3 , h4 , h5 : Constants, ni : The number of documents containing the term ti , N : The total number of documents, ∆ d : Average document length, dl j : Length of document d j .
The objective of this measure is to attenuate the impact of terms having too much high frequency values.
We submitted five official runs to the monolingual English GIRT task ("GIRT_EN"): CWN_T, C_T, CWN_TD, CWNSE_T and CWNSE_TD. The runs are carried out by using title and/or description fields, using or not the term stemming and by performing or not expansion. They are summarized in Table2.
The results obtained by the different runs are summarized in Table3. It should be noticed that an error slipped into the program in the name of query 132 (named by error 232). Consequently, the query 132 is not evaluated at all. The first column of Table 3 gives the median average precision (MAP) obtained by our five official runs on all the queries. We give in the second column the same runs when using the query relevance file obtained after submission and with the query 132 corrected. Concerning the official results, as it can be shown in Figure4, the best results are obtained when using only the title field of the topics and stemming the extracted terms (run C_T). Followed by the run CWN_T where WordNet terms are not stemmed, and then the run CWNSE_T where a short expansion (by synonyms and one hypernym) is applied to non polysemic terms. The two last runs (CWN_TD and CWNSE_TD) are obtained when both title and description fields are used to build the queries respectively with and without expansion.
Concerning the non official runs, the results follow the same logic while being better than the official ones. The fourth column of Table 3 gives the difference of the global results, for the five runs, between the submitted results and the results obtained after the error has been fixed. Roughly the official results could be enhanced by 10,36% in average for each run by using the query 132 and with changing nothing to the system. The reason is that the omitted query (132) brings very good results, which also increases the global result. The detailed results of query 132 are given in Table 4 for the five runs.
We have evaluated the performances of our conceptual indexing method which consists of matching documents
and queries with WordNet. In this method, documents and queries are represented by WordNet nodes. The first
remark, when comparing our submitted runs, is that using only title field (runs C_T and CWN_T) from the
topics seems to bring the better results than using the title and description fields together. The second remark
concerns the use of term stemming. Results showed that stemming indexing terms (run C_T) is slightly better
than not stemming them (run CWN_T) when we consider only the first retrieved documents. However, by using
a more global judgment (MAP), both cases are close. Another remark concerns the Expansion method used in
our experiments. Even though it is made so as to avoid the disambiguation problem (only mono sense terms are
expanded), expansion does not seem to bring the best results. The best run is obtained without expansion and by
using only the title field of the topics. However, the results obtained by the expansion method, when expanding
titles, are better than those obtained when the description fields are used in addition to titles in the queries and
without expansion. So we still believe that a more sophisticated expansion method could bring better results [
Another conclusion concerns the suitability of using WordNet for the domain specific collection. It appears that WordNet largely covers the vocabulary of the English GIRT collection (more than 90% for the documents and practically the entire vocabulary of the 25 used queries) and is suitable to be used for this particular collection.