We describe evaluation experiments conducted by submitting retrieval runs for the monolingual Bulgarian, Czech and Hungarian information retrieval tasks of the AdHoc Track of the Cross-Language Evaluation Forum (CLEF) 2007. In the ad hoc retrieval tasks, the system was given 50 natural language queries, and the goal was to find all of the relevant documents (with high precision) in a particular document set. We conducted diagnostic experiments with different techniques for matching word variations and handling stopwords, comparing the performance on the robust Generalized Success@10 measure and the non-robust mean average precision measure. The measures generally agreed on the mean benefits of morphological techniques such as stemming, but generally disagreed on the blind feedback technique, though not all of the mean differences were statistically significant. Also, for each language, we submitted a sample of the first 10000 retrieved items to investigate the frequency of relevant items at deeper ranks than the official judging depth (of 60 for Czech and 80 for Bulgarian and Hungarian). The results suggest that, on average, the percentage of relevant items assessed was less than 60% for Czech, 70% for Bulgarian and 85% for Hungarian.
http://www.opentext.com/ Livelink ECM - eDOCS SearchServerTM (formerly known as Hummingbird SearchServerTM) is a toolkit for developing enterprise search and retrieval applications. The SearchServer kernel is also embedded in other components of the Livelink ECM - eDOCS Suite1.
1Livelink, Open TextTM and SearchServerTM are trademarks or registered trademarks of Open Text Corporation in the United States of America, Canada, the European Union and/or other countries. This list of trademarks is not exhaustive. Other trademarks, registered trademarks, product names, company names, brands and service names mentioned herein are property of Open Text Corporation or other respective owners.
Language
Czech Hungarian
SearchServer works in Unicode internally [
This paper describes experimental work with SearchServer for the task of finding relevant documents for natural language queries in various European languages using the CLEF 2007 AdHoc Track test collections. 2 2.1
The CLEF 2007 Ad-Hoc Track document sets consisted of tagged (SGML-formatted) news articles in 3 different languages: Bulgarian, Czech and Hungarian. Table 1 gives the sizes.
The CLEF organizers created 50 natural language “topics” (numbered 401-450) and translated them into many languages. Sometimes topics are discarded for some languages because of a lack of relevant documents. Table 1 gives the final number of topics for each language and their average number of relevant documents (along with the lowest and highest number of relevant documents of any topic). For more information on the CLEF test collections, see the track overview paper. 2.2
Our indexing approach was mostly the same as last year [
For some experiments, some stop words were excluded from indexing (e.g. words like “the”,
“by” and “of” in English). For these experiments, the stop word lists for Bulgarian, Czech and
Hungarian were based on Savoy’s lists [
By default, the SearchServer index supports both exact matching (after some Unicode-based normalizations, such as decompositions and conversion to upper-case) and morphological matching (e.g. inflections, derivations and compounds, depending on the linguistic component used). 2.3
We experimented with the SearchServer CONTAINS predicate. Our test application specified SearchSQL to perform a boolean-OR of the query words. For example, for Czech topic 405 whose Title was “Astma u dˇetstv´ı” (Childhood Asthma), a corresponding SearchSQL query would be: SELECT RELEVANCE(’2:4’) AS REL, DOCNO FROM CLEF07CS WHERE FT_TEXT CONTAINS ’Astma’|’u’|’dˇetstvı´’ ORDER BY REL DESC;
Most aspects of the SearchServer relevance value calculation are the same as described last
year [
For the diagnostic runs listed in Tables 2, the run names consist of a language code (“BG” for
Bulgarian, “CS” for Czech and “HU” for Hungarian) followed by one of the following labels:
² “none”: No linguistic variations from stemming were matched. Just the surface forms were
searched on (after case-normalization).
² “stem”: Same as “none” except that linguistic variations from stemming were matched. We
thank Jacques Savoy for providing experimental algorithmic stemmers [
For Czech, our port of the stemmer was accent-insensitive.
² “all”: Same as “stem” except that a separate index was used which did not stop any words
from being indexed.
² “4gram”: Same as “all” except that the run used a different index which primarily consisted
of the 4-grams of terms, e.g. the word ‘search’ would produce index terms of ‘sear’, ‘earc’
and ‘arch’. No stemming was done; searching used the IS_ABOUT predicate (instead of the
CONTAINS predicate) with morphological options disabled to search for the 4-grams of the
query terms. For Bulgarian, we did not index the breve accent for the 4-gram runs (unlike
2 years ago [
2.5
Traditionally, different retrieval measures have been used for “ad hoc” tasks, which seek relevant items for a topic, than for “known-item” tasks, which seek a particular known document. However, we argue that the known-item measures are not only applicable to ad hoc tasks, but that they are often preferable. For many ad hoc tasks, e.g. finding answer documents for questions, just one relevant item is needed. Also, the traditional ad hoc measures encourage retrieval of duplicate relevants, which does not correspond to user benefit.
The traditional known-item measures are very coarse, e.g. Success@10 is 1 or 0 for each topic,
while reciprocal rank cannot produce a value between 1.0 and 0.5. Two years ago, we began
investigating a new measure, Generalized Success@10 (GS10) (introduced as “First Relevant Score”
(FRS) in [
Primary Recall Measures “Primary recall” is retrieval of the first relevant item for a topic. Primary recall measures include the following: ² Generalized Success@30 (GS30): For a topic, GS30 is 1:0241¡r where r is the rank of the first row for which a desired page is found, or zero if a desired page was not found. ² Generalized Success@10 (GS10): For a topic, GS10 is 1:081¡r where r is the rank of the first row for which a desired page is found, or zero if a desired page was not found. ² Success@n (S@n): For a topic, Success@n is 1 if a desired page is found in the first n rows, 0 otherwise. This paper lists Success@1 (S1) and Success@10 (S10) for all runs. ² Reciprocal Rank (RR): For a topic, RR is 1r where r is the rank of the first row for which a desired page is found, or zero if a desired page was not found. “Mean Reciprocal Rank” (MRR) is the mean of the reciprocal ranks over all the topics.
Interpretation of Generalized Success@n: GS30 and GS10 are estimates of the percentage of potential result list reading the system saved the user to get to the first relevant item, assuming that users are less and less likely to continue reading as they get deeper into the result list. Comparison of GS10 and Reciprocal Rank : Both GS10 and RR are 1.0 if a desired page is found at rank 1. At rank 2, GS10 is just 7 points lower (0.93), whereas RR is 50 points lower (0.50). At rank 3, GS10 is another 7 points lower (0.86), whereas RR is 17 points lower (0.33). At rank 10, GS10 is 0.50, whereas RR is 0.10. GS10 is greater than RR for ranks 2 to 52 and lower for ranks 53 and beyond.
Connection of GS10 to Success@10 : GS10 is considered a generalization of Success@10 because it rounds to 1 for r·10 and to 0 for r>10. (Similarly, GS30 is considered a generalization of Success@30 because it rounds to 1 for r·30 and to 0 for r>30.) 2.5.2
Secondary Recall Measures
“Secondary recall” is retrieval of the additional relevant items for a topic (after the first one).
Secondary recall measures place most of their weight on these additional relevant items.
² Precision@n: For a topic, “precision” is the percentage of retrieved documents which are
relevant. “Precision@n” is the precision after n documents have been retrieved. This paper
lists Precision@10 (P10) for all runs.
² Average Precision (AP): For a topic, AP is the average of the precision after each relevant
document is retrieved (using zero as the precision for relevant documents which are not
retrieved). By convention, AP is based on the first 1000 retrieved documents for the topic.
The score ranges from 0.0 (no relevants found) to 1.0 (all relevants found at the top of the
list). “Mean Average Precision” (MAP) is the mean of the average precision scores over all
of the topics (i.e. all topics are weighted equally).
² Geometric MAP (GMAP): GMAP (introduced in [
For each language, we submitted 4 experimental runs in May 2007 for official assessment. In the identifiers (e.g. “otBG07tde”), ‘t’, ‘d’ and ‘n’ indicate that the Title, Description and Narrative field of the topic were used (respectively), and ‘e’ indicates that query expansion from blind feedback on the first 3 rows was used (weight of one-half on the original query, and one-sixth each on the 3 expanded rows). The ‘z’ code indicates that special sampling was done, as described below. From the Description and Narrative fields for most languages, instruction words such as “find”, “relevant” ² “tdnz”: Depth-10000 sampling run based on the “tdn” run as described below.
Table 6 lists the mean scores for the submitted runs. BG-tde-td CS-tde-td HU-tde-td BG-tde-td CS-tde-td HU-tde-td ¡0:010 ¡0:011 0.001 The submitted tdnz run for each language was actually a depth probe run from sampling the tdn run for the language (the tdn run was not itself submitted).
The base tdn run was retrieved to depth 10000 for each topic. The first 100 rows of the submitted tdnz run contained the following rows of the base tdn run in the following order: ² “Samples”: The depths of the sample points from the depth range. The samples are always uniformly spaced. They always end at the last point of the depth range. The total number of sample points (over the 11 rows of the table) adds to 60 for Czech and 80 for Bulgarian and Hungarian. ² “Precision”: Estimated precision of the depth range (R/(R+N+U)). ² “Wgt”: The weight of each sample point. The weight is equal to the difference in ranks between sample points, i.e. each sample point can be thought of as representing this number of rows, which is itself plus the preceding unsampled rows. The weights are higher in some cases for Czech than for Bulgarian and Hungarian because we have fewer sample points for Czech (60 instead of 80). ² “EstRel/Topic”: Estimated number of relevant items retrieved per topic for this depth range.
This is the Precision multiplied by the size of the depth range. Or equivalently, it is (R * Wgt) / 50.
Because each sample point is at the deep end of the range of rows it represents, the sampling should tend to underestimate precision for each depth range (assuming that precision tends to fall with depth, which appears to be the case for all 3 languages).
Table 11 shows the sums of the estimated number of relevant items per topic over all depth ranges in its first row. The official number of relevant items per topic for each language is listed in the second row. The final row of the table just divides the official number of relevant items by the estimated number in the first 10000 retrieved (e.g. for Bulgarian, 20.2/29.3=69%). This number should tend to be an overestimate of the percentage of all relevant items that are judged (on average per topic) because there may be relevant items that were not matched by the query in the first 10000 rows.
However, the sampling was very coarse at the deeper ranks, e.g. for Czech, 1 relevant item out of 300 samples in the 3001-6000 range led to an estimate of 10 relevant items per topic in this range. If the sampling had turned up 0 or 2 relevant items, a minor difference, the estimate would have been 0 or 20 relevant items per topic in this range, leading to a substantially different sum (17.4 or 37.4 instead of 27.4). We should compute confidence intervals for these estimates, but have not yet done so. Also, there is a lot of variance across topics, which we have not yet analyzed.
These preliminary estimates of judging coverage for the CLEF 2007 collections (55% for Czech,
69% for Bulgarian, 83% for Hungarian) are much higher than the estimates we produced for the
TREC 2006 Legal and Terabyte collections using a similar approach (18% for TREC Legal and
36% for TREC Terabyte) [
These incompleteness results are similar to what [
Fortunately, [
IR
Multilingual information retrieval resource