This document present work we have done for the CLEF 2004 participation. We promote the use of surface-syntactic parsing to extract indexing terms. We also promote the Derivation From Randomness weighting. For the bilingual part, we have tested reinforcement query weighting using an association thesaurus. In our previous participation at CLEF in 2003 [3], we have tested the use of an association thesaurus to enhance query translation. We have only use the association thesaurus to add some new terms to the proposed translation terms. In our current participation, we have tried another use of such a thesaurus: we do not enlarge the query, but rather we use it to modify the weighing of a given translated query. Our basic idea is the selection of the best term translation using query context and association thesaurus from the corpus. Last year, we have neglected the study of the matching function and the influence of the weighting scheme. For this participation we will also focus on this aspect: we have test the Derivation From Randomness (DFR) against Okapi measure and some other classical IR weighting. We also promote the use of a surface-syntactic parser. All documents are first transform by the parser. The stemming is then proposed by the parser. Finnish is an agglutinative language, and using such an NL parsing enable to correctly split the glued words into separated correct indexing terms. The paper present first the training experiments performed on 2003 collection in part 2. In part 3 we discuss the monolingual results. Then, in part 4, we present the technique used for bilingual results and present hypothesis based on the results.
2.1
All experiments are grounded on the classic vector space model. Goal of experiment is to compare the probabilistic model of Okapi with Derivation From Randomness model, versus more classical weightings. This comparison will be done on two different languages.
Basically, the final matching process is achieved by a product between query vector and document matrix, which computes the Relevant Status Value (RSV) of all document against the query. For a query vector Q = (qi) with a dimension of t term i ∈ [1..t], and a index document matrix of n documents Dj = (dij ), j ∈ [1..n], the RSV is computed by:
RSV (Q, Dj) =
qi ∗ dij i∈[1..t] We keep this matching process for all tests, the changes are in the documents and query processing to select indexing terms, and in the weighting scheme. We recall here the scheme that is inspired by the SMART system. We suppose the previous processing steps have produced a matrix D = (di,j ). Usually, the value di,j is only the result of term ti counting in the document Dj, called term frequency tfij. Each weighting scheme can be decomposed in three steps: a local, a global and a normalization step. The local is related to only one vector. All these transformations are listed in table 1. For all measure we use the following symbols: fi total number of term i in the corpus di∗j is a normalization of dij λi is the fraction fTi T is the corpus size : T = i fi dfi the document frequency of i dij current value in the matrix c a constant for DFR L(Dj) the length of Dj equals to t dij awr(L(Dj )) n t qi mean document length equals to number of document in the corpus number of unique terms in the corpus weight of term i of query q j L(Dj)
n n b a l d wij = dij none, no change wij = 1 binary wij = 0m.5a+x0i.(5d∗idji)j local max wij = ln(dij + 1) natural log wij = ln(ln(dij + 1) + 1) double natural log
The global weighting is related to the matrix, and then it is a weighing which takes into account the relative importance of a term regarding the whole document collection. The most famous is the Inverse Document Frequency : Idf. The table 2 lists the global weighting we have tested. Okapi and DFR are not global weighting per se but rather complete weighting scheme themselves. In our X-IOTA system, they are computed at the same time than global weighting, and it is technically feasible to use them with a local and a final normalization. DFR is presented in the next part.
The Okapi measure described in [
A weighting scheme is composed by the combination of the local, global and final weighting. We represent a weighting scheme by 3 letters. For example, nnn is only the raw term frequency. The scheme bnn for both documents and queries leads to a sort of Boolean model where every term in the query is considered connected by a conjunction. In that case the RVS counts the terms intersection between documents and queries. The c normalization applied to both document and query vector leads to the computation of the cosine between these two vectors. This is the classical vector space model if we use the ltc scheme for document and queries. The scheme nOn for the documents, and npn with the queries, is the Okapi model, and the use of nRn for document and nnn for the queries is the DFR model. For these two models, constants have to be defined.
Notice that the c normalization of the queries, leads to divide the RSV for this query by q2. For each query this is a constant value which does not influence the relative order of i i answered document list. It follows that this normalization is useless for queries and should not be used. In the next section we briefly present the derivation from Randomness weighting that seems to give best results, and that we have used for all CLEF 2004 runs. 2.2
derivation from randomness (DFR)
This weighting scheme has been proposed by Gianni Amati in [
fi + 1 ) ∗ ni ∗ (fi∗j + 1) The value di∗j is a normalization by the length L(Dj) of the document Dj regarding the average size of all document in the corpus : awr(Dj ). A constant value c adjusts the effect of the document length in the weight. (1) (2) di∗,j = dij ∗ log2(1 + c ∗ awr(L(Dj )) )
L(Dj)
For this participation of CLEF, we have test this weighting scheme against another set of other computation. We present these results on Finnish and French collection. 2.3
In these experiments, we have first tested the influence of stop words (SW) and stemming. We have not tested the influence of the surface syntactic parser, because the parsing was not available the time we have made these tests. The test performed here is done on the Finnish collection with 2003 queries. As the best results is of course, obtained with stop word and stemming, we have then tested the influence of the c constant in order to find out when we reach the optimum. The treatment we apply to both documents and queries is given by: xmlFilterTag | xml2Latin1 | xmldeldia | xmlcase | xmlAntiDico -dico common_word.fi | xmlcase -noAcc | xmlStemFi The first step is filtering the relevant tags from documents or queries. Then we transform XML special characters to their ISO counterpart. We delete all diacritic characters, and change to lower case. At this stage we still have special Finnish characters and accents. We eliminate common words using a list provided by Savoy1 and then suppress all accents from characters. We apply a Finnish stemmer also proposed by Savoy and modifies to accept XML input/output to produce the final vector. For the queries, we have used the following fields: FI-title FI-desc FI-narr. For documents only the text field has been used.
Results of DFR test with nnn query weighting scheme is in the table 5. When c is zero, then the equation becomes (4), where term weight are all equal for all documents.
run raw
SW stem SW
When we examine DFR formula, one can see that when a term does not appear in document d, then only di,j is null. Then di∗,j is also equal to zero. If we strictly apply the formula in that case, the weight of the term is still not null and is equal to the formula (4). For practical reason, we have replaced this residual value by zero. This approximation reduces the size of the inverse file, because we do not store null values in file. In fact we have applied the following weighting: wi,j = wi,j if di,j = 0
0 if di,j = 0 wt,d = log2(1 + λt). ft + 1 nt
The conclusion of the use of this weighting is that a good constant c value seems to be 0.83. In the rest of the test, we will use the approximated value c = 0.8.
For the Okapi weighting, we have use the same value as in [
Testing query weighting We have tested all combination of the following weight: c 0.00 0.10 0.50 0.70 0.75 0.80 0.81 0.82 0.83 0.84 0.85 0.86 0.87 0.90 0.95 1.00 1.50 2.00 5.00 10.0 nnn: Only the term frequency is used. bnn: This is the binary model. Terms presents are associated to the value 1, and 0 otherwise. lnc: The cosine is the finale normalization. When both used in document and queries, it ensure true vector space model matching, ie. only angle between query et document vector is used.
This weighting suppose a log distribution of frequency. ntc: This is the classical tf*idf measure. We used with queries, the idf is taken from the document collection, not the query collection. ltc: The same classical measure using log on term frequency. ltn: The log tf*idf without the cosine normalization. atn: Normalization with the local maximum term frequency is used with idf. dtn: The double natural log is used in place of the simple one in ltn. npn: It is the idf variant use for the Okapi system. nRn: This is the name for Derivation from randomness. nOn: This is the name for the Okapi probabilistic weighting.
Results are sum up in the table 6.
We notice that the derivation from randomness model is very stable again the query weighting and that it has the best results in the majority of query weighting. We have the decided to use it for CLEF 2004 in all tests. For training, we have used the French corpus of CLEF 2003. We have used our own stemmer, and our own list for removal of common French terms. In this collection, there are 3 sets of documents. For each collection we have selected the following fields: lemonde94 TITLE TEXT, and TI KW LD TX ST for sda 94 and 95. For the queries, we have selected the fields FR-title FR-desc FR-narr. We have tested the same combination of weighting schemes as the one tested in the Finnish collection. The results are in the table 7 and 8.
Doc. nnn bnn atn ntc lnc ltc ltn nRn nOn nnn 7.72 16.01 31.02 33.53 36.20 35.39 35.65 46.98 42.25 In this part, we comment the results we have obtained at CLEF 2004. We have participated to the monolingual track on French, Finnish and Russian. As we promote the use of syntactic parsing, we have submitted mono lingual run all using surface syntactic parsing. Because of time constrains, we have not trained the system with the parsed collection. So we can only compare with CLEF 2003 without natural parsing.
After each parsing, we have transformed the output into a common XML simplified format. One of the main interests in using a natural language parser is the correct normalization of words, the correct detection of compound nouns and correct filtering using lexical categories. For all run, we have choose the derivation from randomness weighting with the constant value fixed to c = 0.8, according to the training experiments. No special treatments are done on queries.
These characteristics are important for language which has a morphological derivation like French and Finnish. For all language, we have filtered only nouns, proper noun, verb and adjective. For French, we have used the XIP system from XEROX. After this filtering, we still remove some terms using a stop list, and used also a French stemming. The French queries are weighted using ntn. Hence we only modify the weight according to the inverse document frequency. This computation is of course performed using document corpus. The average precision is 44%, which is not an absolute good result. This value is a little lower than our training.
When we examine more closely the results, we discover a big discrepancy between queries. Figure 1, shows the histogram repartition of the 29 queries (from 201 to 250 without 227). There are a lot of query that are either very low precision level (18 queries under 20%) of very high (13 upper 80%).
For the Finnish monolingual run, we obtain an average prevision of 53%, witch is better than the results obtained on CLEF 2003. The histogram in figure 2 shows that 10 query are above 90%, in fact exactly 5 queries reach 100% of precision.
We have use also a surface syntactic parser for the Russian collection, but we cannot compare yet with a more simple raw term indexing because we do not have a Russian stemmer and stop list. The average result of 35% is the lowest for all three languages. Query precision repartition in figure 3 shows that a lot of query (12) have very low precision (under 10%).
The conclusion that we draw is the good behavior of DFR weighting, and probably the benefit of using a surface syntactic parsing on Finnish. In this language, the parser is able to "unglue" terms and so could achieve better results. We cannot investigate more our results, because we should compare on the same collection the use of the syntactic surface parsing. 4
Bilingual results are obtained by translating the topics using general dictionaries built by compiling several bilingual dictionaries available online (see section 4.1). Then, we experimented 2 methods of translation (see sections 4.2). Both methods take the topic vectors as input and outputs a new translated topic vector. 4.1
We compiled 6 bilingual translation dictionaries (see figure 4) using several resources available in house or from Internet. Each resulting dictionary associates a word form to a set of translations and is stored as an XML file (see figure 5).
Dictionary
nb of entry fr - en fr - fi fr - ru en - fr en - fi en - ru • the French English dictionary available for the participants on the CLEF web site, • dictionary entries from the Logos website2, • the “engrus” English Russian dictionary available on many web sites3.
As for the French-Russian, French-Finnish and English-Finnish dictionaries, the only available online resource we used is the Logos web site. As it is the only online service we used (other data was available off-line), we chose to only extract entries that were present in the topics to be translated in order to avoid high loads on a public web site. This explains the very small size of these dictionaries.
As French and English were our topic languages of choices, we also reverted the merged FrenchEnglish dictionaries. 4.2
For each bilingual task we participated in, we propose 2 methods of translation. Both methods take the topic vectors as input and outputs a new translated topic vector.
2http://www.logos.it/ 3see list of mirrors at http://sinyagin.pp.ru/engrus-mirrors.html English to Russian <entry id="issue"> <equiv>выпускать</equiv> <equiv>выпуск</equiv> <equiv>излияние</equiv> <equiv>результат</equiv> <equiv>исход</equiv> <equiv>спорный_вопрос</equiv> <equiv>исходить</equiv> <equiv>вытекать</equiv> <equiv>вытекание</equiv> <equiv>издать</equiv> <equiv>издание</equiv> <equiv>происходить</equiv> <equiv>vypuskat’</equiv> <equiv>выход</equiv> <equiv>потомство</equiv> </entry> <entry id="harp"> <equiv>арфа</equiv> <equiv>играть_на_арфе</equiv> </entry>
English to French <entry id="issue"> <equiv>nume´ro</equiv> <equiv>e´mettre</equiv> <equiv>e´mission</equiv> <equiv>matie`re</equiv> <equiv>de´livrer</equiv> <equiv>cre´er</equiv> <equiv>affliger</equiv> <equiv>impression</equiv> <equiv>question</equiv> <equiv>de´livrance</equiv> <equiv>ligne´e</equiv> </entry> <entry id="harp">
<equiv>harpe</equiv> </entry> The first method substitutes each term by all of its available translations. The weight associated to each translation is equal to the weight of the original term divided by the number of available translation (see figure 6). 4.2.2
Filtering by way of an association thesaurus As one may see in figure 5, many different translations may be found for a single term. Hence, we tried to develop a strategy to give more importance to the “correct” translation(s). For this, we tried to take some context into account, without changing anything to the available lexical resource.
For this, we needed contextual information in each language. Hence, we automatically built
an association thesaurus (as exposed in [
Each association thesaurus is as a graph linking terms. Each arc in the graph links 2 terms that “regularly”4 appear in the same context. For this experiment, 2 terms are said to be in the same context when they appear in the same document.
For our experiment, we assume that terms that are close to each others share some common semantic. We also assume that their “correct” translations should also share the same semantic. Hence, we used these association thesaurus to know if terms and translations share some semantics. Hence, we chose to associate each translations ti,j of a term cj with a weight wti,j depending on its distance (dti,j ) with the translated context. The distance of a translation to the translated context is given by formula 5.
4In this experiment, we filtered out arcs that had a confidence score lower than 20% or higher than 90%. CLIPS results on the bilingual tasks are rather disappointing, with interpolated recall-precision averages at 0.00 dropping from 57.68% (Monolingual Russian) to 17.1% with simple topic translation (Bilingual English-Russian: CLIPSENRU1) and even to 8.59% with filtered topic translation (CLIPSENRU2).
The main reason for this drop is certainly due to the lack of wide coverage bilingual lexical resources. The dictionaries we used were very small and did not provide translations for many terms of the topics. This is especially true for French to Russian and Finnish lexical resources where 60% to 70% of the source terms are not translated. However, English to Russian lexicon was a little better, and about 18% of the terms remain without translation.
However, this does not explain the drop in interpolated recall-precision averages when filtering the translations through the association thesaurii, as it does not change the set of translations, but only the weight of those translations. Moreover, when manually evaluating the weighted translation, one usually agree with the translation that are chosen.
We think that 2 factors explains theses drops: • First, in the simple topic translation method, the weight of each translation is divided by the number of translations for the source term. This lowers the relative importance of terms that bear many translations, (which is usually the case of general nouns or support verbs). • Second, when raising the weight of “correct” translations by way of the association thesaurii, we also raise the weight of such general terms. Hence, we give more importance to terms that do not bear any thematic closeness with the requested documents (and this is especially the case with CLEF topics that are instructions usually containing “find documents reporting on. . . ” or “find information on. . . ”). 5
All run are performed on the collection parsed using a syntactic surface parsing. Best monolingual results are obtained for the Finnish collection, probably because of the correct word splitting. We have to redo the tests with no analyzer to have a strong conclusion on its use in an IR context/
Bilingual results are disappointing but they are partly explained by the difficulty in finding wide coverage lexical resources for languages in which we previously had no experience whatsoever.
The filtering of translations through association thesaurii is rather interesting, even if we did not have enough time to use it appropriately. This technique may also be interesting in translation selection tasks or, with adaptation, on lexical disambiguation tasks. It’s main interest in such tasks comes from the fact that it does not require any special training data (like parallel documents or manually disambiguated corpora) as association thesaurii may be computed automatically from the corpus. Hence such technique may easily bring some result in those tasks in any language, provided that monolingual data is available as well as an automatic process to lemmatize such corpora. <vector id="C250" size="57"> ... <!-- Translation of id="reader" w="1" --> <c id="reader" w="1" untranslated="true"/> <!-- Translation of id="human" w="5" --> <c id="ЧЕЛОВЕЧЕСКИЙ" w="2.5"/> <c id="ЧЕЛОВЕК" w="5"/> <!-- Translation of id="document" w="3" --> <c id="ПОДТВЕРЖДАТЬ_ДОКУМЕНТАМИ" w="1"/> <c id="ДОКУМЕНТ" w="1"/> <c id="СВИДЕТЕЛЬСТВО" w="1"/> <!-- Translation of id="Rabie" w="1" --> <c id="Rabie" w="1" untranslated="true"/> <!-- Translation of id="prevention" w="2" --> <c id="prevention" w="2" untranslated="true"/> <!-- Translation of id="interest" w="1" --> <c id="ИНТЕРЕС" w="0.5"/> <c id="ВЫГОДА" w="0.25"/> <c id="ИНТЕРЕСОВАТЬ" w="0.25"/> <c id="ЗАИНТЕРЕСОВЫВАТЬ" w="0.25"/> <!-- Translation of id="rabies" w="4" --> <c id="БЕШЕНСТВО" w="4"/> ...
</vector>