We describe evaluation experiments conducted by submitting retrieval runs for the monolingual German, French, English and Persian (Farsi) information retrieval tasks of the Ad Hoc Track of the Cross-Language Evaluation Forum (CLEF) 2009. In the ad hoc retrieval tasks, the system was given 50 natural language queries, and the goal was to find all of the relevant records or documents (with high precision) in a particular document set. We conducted diagnostic experiments with different techniques for matching word variations, comparing the performance on the robust Generalized Success@10 measure and the non-robust Mean Average Precision measure. The measures generally agreed on the mean benefits of morphological techniques such as decompounding and stemming, but generally disagreed on the blind feedback technique. Also, for each language, we submitted a sample of the first 10000 retrieved items to investigate the frequency of relevant items at deeper ranks than the official judging depth of 60 for German, French and English and 80 for Persian. The results suggest that, on average, the percentage of relevant items assessed was less than 62% for German, 27% for French, 35% for English and 22% for Persian.
http://www.opentext.com/ Open Text eDOCS SearchServerTM is a toolkit for developing enterprise search and retrieval applications. The eDOCS SearchServer kernel is also embedded in various components of the Open Text eDOCS Suite1.
1Open Text eDOCS SearchServer and Open Text eDOCS Suite are trademarks or registered trademarks of Open Text Corporation in the United States of America, Canada, the European Union and/or other countries. This list of trademarks is not exhaustive. Other trademarks, registered trademarks, product names, company names, brands and service names mentioned herein are property of Open Text Corporation or other respective owners.
Code DE EN FA FR
The eDOCS SearchServer kernel works in Unicode internally [
This paper describes experimental work with SearchServer (experimental post-6.0 versions) for the task of finding relevant documents for natural language queries in various European languages using the CLEF 2009 Ad Hoc Track test collections. 2 2.1
The CLEF 2009 Ad Hoc Track document sets were the same as used in 2008. They consisted of XML-tagged records or documents in 4 different languages: German, French, English and Persian (also known as Farsi). For German, French and English, the records were library catalog cards (bibliographic records describing publications archived by The European Library (TEL)). For Persian, the documents were newspaper articles (Hamshahri corpus of 1996-2002). Table 1 gives the collection sizes.
The CLEF organizers created 50 natural language “topics” (numbered 701-750 for German, French and English and 601-650 for Persian) and translated them into many languages. Sometimes topics are discarded for some languages because of a lack of relevant documents (though that did not happen this year). Table 1 gives the final number of topics for each language and their average number of relevant documents (along with the lowest and highest number of relevant documents of any topic). For more information on the CLEF test collections, please see the track overview paper. 2.2
Our indexing approach was the same as last year [
For some experiments, some stop words were excluded from indexing (e.g. words like “the”,
“by” and “of” in English). For our Persian experiments, our stop word list was based on Savoy’s
list [
By default, the SearchServer index supports both exact matching (after some Unicode-based normalizations, such as decompositions and conversion to upper-case) and morphological matching (e.g. inflections, derivations and compounds, depending on the linguistic component used). 2.3
We experimented with the SearchServer CONTAINS predicate. Our test application specified SearchSQL to perform a Boolean-OR of the query words. For example, for German topic 701 whose Title was “Tiere in der Arktis” (Arctic Animals) a corresponding SearchSQL query would be: SELECT RELEVANCE(’2:3’) AS REL, DOCNO FROM CLEF09DE WHERE FT_TEXT CONTAINS ’Tiere’|’in’|’der’|’Arktis’ ORDER BY REL DESC;
Most aspects of the SearchServer relevance value calculation are the same as described last year
[
For the diagnostic runs listed in Table 2, the run names consist of a language code (“DE” for
German, “EN” for English, “FA” for Persian, and “FR” for French) followed by one of the following
labels:
² “none”: No linguistic variations from stemming were matched. Just the surface forms were
searched on (after case-normalization).
² “lexstem” (German, French and English only): Same as “none” except that linguistic
variations from stemming were matched. The lexicon-based inflectional stemmer in SearchServer
was used. For German, this stemmer includes decompounding.
² “algstem”: Same as “lexstem” except that an algorithmic stemmer was used. For Persian, our
stemmer was ported from Savoy’s [
Note that all diagnostic runs just used the Title field of the topic. 2.5
Traditionally, different retrieval measures have been used for “ad hoc” tasks, which seek relevant items for a topic, than for “known-item” tasks, which seek a particular known document. However, we argue that the known-item measures are not only applicable to ad hoc tasks, but that they are often preferable. For many ad hoc tasks, e.g. finding answer documents for questions, just one relevant item is needed. Also, the traditional ad hoc measures encourage retrieval of duplicate relevant documents, which does not correspond to user benefit.
The traditional known-item measures are very coarse, e.g. Success@10 is 1 or 0 for each topic,
while reciprocal rank cannot produce a value between 1.0 and 0.5. In 2005, we began investigating a
new measure, Generalized Success@10 (GenS@10 or GS10) (introduced as “First Relevant Score”
(FRS) in [
Primary Recall Measures “Primary recall” is retrieval of the first relevant item for a topic. Primary recall measures include the following: ² Generalized Success@30 (GenS@30 or GS30): For a topic, GS30 is 1:0241¡r where r is the rank of the first row for which a desired page is found, or zero if a desired page was not found. ² Generalized Success@10 (GenS@10 or GS10): For a topic, GS10 is 1:081¡r where r is the rank of the first row for which a desired page is found, or zero if a desired page was not found. ² Success@n (S@n): For a topic, Success@n is 1 if a desired page is found in the first n rows, 0 otherwise. This paper lists Success@1 (S1) and Success@10 (S10) for all runs. ² Reciprocal Rank (RR): For a topic, RR is 1r where r is the rank of the first row for which a desired page is found, or zero if a desired page was not found. “Mean Reciprocal Rank” (MRR) is the mean of the reciprocal ranks over all the topics.
Interpretation of Generalized Success@n: GS30 and GS10 are estimates of the percentage of potential result list reading the system saved the user to get to the first relevant item, assuming that users are less and less likely to continue reading as they get deeper into the result list. Comparison of GS10 and Reciprocal Rank : Both GS10 and RR are 1.0 if a desired page is found at rank 1. At rank 2, GS10 is just 7 points lower (0.93), whereas RR is 50 points lower (0.50). At rank 3, GS10 is another 7 points lower (0.86), whereas RR is 17 points lower (0.33). At rank 10, GS10 is 0.50, whereas RR is 0.10. GS10 is greater than RR for ranks 2 to 52 and lower for ranks 53 and beyond.
Connection of GS10 to Success@10 : GS10 is considered a generalization of Success@10 because it rounds to 1 for r·10 and to 0 for r>10. (Similarly, GS30 is considered a generalization of Success@30 because it rounds to 1 for r·30 and to 0 for r>30.) 2.5.2
Secondary Recall Measures
“Secondary recall” is retrieval of the additional relevant items for a topic (after the first one).
Secondary recall measures place most of their weight on these additional relevant items.
² Precision@n: For a topic, “precision” is the percentage of retrieved documents which are
relevant. “Precision@n” is the precision after n documents have been retrieved. This paper
lists Precision@10 (P10) for all runs.
² Average Precision (AP): For a topic, AP is the average of the precision after each relevant
document is retrieved (using zero as the precision for relevant documents which are not
retrieved). By convention, AP is based on the first 1000 retrieved documents for the topic.
The score ranges from 0.0 (no relevant documents found) to 1.0 (all relevant documents
found at the top of the list). “Mean Average Precision” (MAP) is the mean of the average
precision scores over all of the topics (i.e. all topics are weighted equally).
² Geometric MAP (GMAP): GMAP (introduced in [
For tables comparing 2 diagnostic runs (such as Table 3), the columns are as follows: ² “Expt” specifies the experiment. The language code is given, followed by the labels of the 2 runs being compared. (We abbreviate “lexstem” to “lex”, “algstem” to “alg”, “4gram” to “4gr” and “lexall” and “algall” to “all”.) The difference is the first run minus the second run. For example, “DE-lex-none” specifies the difference of subtracting the scores of the German ‘none’ run from the German ‘lexstem’ run (of Table 2). ² “¢GS10” is the difference of the mean GS10 scores of the two runs being compared (and “¢MAP” is the difference of the mean average precision scores). ² “95% Conf” is an approximate 95% confidence interval for the difference (calculated from plus/minus twice the standard error of the mean difference). If zero is not in the interval, the result is “statistically significant” (at the 5% level), i.e. the feature is unlikely to be of neutral impact (on average), though if the average difference is small (e.g. <0.020) it may still be too minor to be considered “significant” in the magnitude sense. ² “vs.” is the number of topics on which the first run scored higher, lower and tied (respectively) compared to the second run. These numbers should always add to the number of topics. ² “3 Extreme Diffs (Topic)” lists 3 of the individual topic differences, each followed by the topic number in brackets. The first difference is the largest one of any topic (based on the absolute value). The third difference is the largest difference in the other direction (so the first and third differences give the range of differences observed in this experiment). The middle difference is the largest of the remaining differences (based on the absolute value). 3 3.1
Table 3 shows the impact of stemming for the 4 languages. For instance, it shows that the mean increase in GenS@10 was statistically significant for German, and the mean increases in MAP were statistically significant for all 4 languages.
Table 3 also shows that there were large impacts from stemming on particular topics for German, French and English in both the GenS@10 and MAP measures (we look at some examples in the later sections).
Like last year, for Persian, even on individual topics there was relatively little impact from stemming. We notice in Table 2 that the Success@10 rate was relatively high for Persian (50 out of 50) even without stemming, and that relevant documents were plentiful (89 per topic on average as per Table 1), but we have not done sufficient analysis to understand why the stemming impact was so minor across Persian topics. Expt
FR-lex-none EN-lex-none FA-alg-none ¢GS10 weight from three 4-grams (‘nabb’, ‘abba’, ‘bbau’) and it did not retrieve a relevant record until rank 509. 4
For each language, we submitted 4 experimental runs in June 2009 for official assessment. In the identifiers (e.g. “otFA09tdz”), ‘t’ and ‘d’ indicate that the Title and Description field of the topic were used (respectively), and ‘e’ indicates that query expansion from blind feedback on the first 3 rows was used (weight of one-half on the original query, and one-sixth each on the 3 expanded rows). The ‘z’ code indicates that special sampling was done, as described below. From the Description field for German, French and English, instruction words such as “find”, “relevant” and “document” were automatically removed (based on looking at some older topic lists, not this year’s topics; this step was skipped for Persian as we didn’t have time to update our lists this year).
Details of the submitted approaches: ² “t”: Just the Title field of the topic was used. Same as the “lexstem” run of Section 2.4 for
German, French and English, and same as the “algstem” run of Section 2.4 for Persian. ² “td”: Same as “t” except that the Description field of the topic was additionally used. ² “tde”: Same as “td” except that blind feedback (based on the first 3 rows of the “td” query) was used to expand the query. ² “tdz”: Depth-10000 sampling run based on the “td” run as described below. EN-tde-td FA-tde-td FR-tde-td DE-tde-td EN-tde-td FA-tde-td FR-tde-td DE-tde-td ¡0:007 ¡0:015 ¡0:023 ¡0:039 The submitted tdz run for each language was actually a depth probe run from sampling the td run for the language.
The base td run was retrieved to depth 10000 for each topic. The first 100 rows of the submitted tdz run contained the following rows of the base run in the following order:
The remainder of the sample run was padded with the top-ranked remaining rows from the base run until 1000 rows had been retrieved (i.e. rows 11, 12, 13, 14, 16, ..., 962 of the base run).
This ordering (e.g. the placement of the sample from depth 10000 before the sample from depth 15) was chosen because of uncertainty of how deep the judging would be. As long as the top-37 were judged for each topic, we would have sampling to depth 10000 (because in the above list, you can count that after 37 samples that depth 10000 is reached). The extra sample points, if judged, would just improve the accuracy (because they are just additional sample points from the top 10000, not deeper sample points).
Our sample run (i.e. the tdz run) for each language was submitted to the CLEF organizers for assessing in June 2009. We assigned it highest judging precedence of all our submitted runs.
When we received the relevance judgments and analyzed them in August 2009, we checked the judging depth of our sample runs. We found that the top-60 rows were judged for each topic for each German, French and English, and the top-80 rows were judged for each topic for Persian.
Tables 10, 11, 12 and 13 show the results of the sampling for each language. The columns are as follows: ² “Depth Range”: The range of depths being sampled. The 11 depth ranges cover from 1 to 10000. ² “Samples”: The depths of the sample points from the depth range. The samples are always uniformly spaced. They always end at the last point of the depth range. The total number of sample points (over the 11 rows of the table) adds to 60 for German, French and English and adds to 80 for Persian. ² “Precision”: Estimated precision of the depth range (R/(R+N+U+X)). ² “Wgt”: The weight of each sample point. The weight is equal to the difference in ranks between sample points, i.e. each sample point can be thought of as representing this number of rows, which is itself plus the preceding unsampled rows. ² “EstRel/Topic”: Estimated number of relevant items retrieved per topic for this depth range.
This is the Precision multiplied by the size of the depth range. Or equivalently, it is (R * Wgt) / 50.
Because each sample point is at the deep end of the range of rows it represents, the sampling should tend to underestimate precision for each depth range (assuming that precision tends to fall with depth, which appears to be the case for all 4 languages).
Table 14 shows the sums of the estimated number of relevant items per topic over all depth ranges in its first row (i.e. it is the sum of the EstRel/Topic entries in the last column of the corresponding table from Tables 10-13). The official number of relevant items per topic for each language is listed in the second row. The final row of the table just divides the official number of relevant items by the estimated number in the first 10000 retrieved (e.g. for German, 31.2/50.2=62%). This number should tend to be an overestimate of the percentage of all relevant items that are judged (on average per topic) because there may be relevant items that were not matched by the query in the first 10000 rows.
However, the sampling was very coarse at the deeper ranks, e.g. for French, 1 relevant item out of 200 samples in the 6001-10000 range led to an estimate of 20 relevant items per topic in this range. If the sampling had turned up 0 or 2 relevant items, a minor difference, the estimate would have been 0 or 40 relevant items per topic in this range, leading to a substantially different sum (118.5 or 158.5 instead of 138.5). We should compute confidence intervals for these estimates, but have not yet done so. Also, there is a lot of variance across topics, which we have not yet analyzed.
These preliminary estimates of judging coverage for the CLEF 2009 collections are similar to
last year’s estimates [
On older TREC collections of approximately 500,000 documents which used depth-100 pooling,
[
We described evaluation experiments conducted by submitting retrieval runs for the monolingual German, French, English and Persian (Farsi) information retrieval tasks of the Ad Hoc Track of the Cross-Language Evaluation Forum (CLEF) 2009. We conducted diagnostic experiments with different techniques for matching word variations, comparing the performance on the robust Generalized Success@10 measure and the non-robust Mean Average Precision measure. The measures generally agreed on the mean benefits of morphological techniques such as decompounding and stemming, but generally disagreed on the blind feedback technique. Also, for each language, we submitted a sample of the first 10000 retrieved items to investigate the frequency of relevant items at deeper ranks than the official judging depth of 60 for German, French and English and 80 for Persian. The results suggest that, on average, the percentage of relevant items assessed was less than 62% for German, 27% for French, 35% for English and 22% for Persian. We analyzed a few individual topics for which the different retrieval techniques produced large differences in the scores and found that the judgments were sufficient to gain insight into the reasons for the for
IR
retrieval resource