This paper describes the WebCLEF 2007 task. The task definition-which goes beyond traditional navigational queries and is concerned with undirected information search goals-combines insights gained at previous editions of WebCLEF and of the WiQA pilot that was run at CLEF 2006. We detail the task, the assessment procedure and the results achieved by the participants.
1.1
Task model Our hypothetical user is a knowledgable person, perhaps even an an expert, writing a survey article on a specicfi topic with a clear goal and audience, for example, a Wikipedia article, or a state of the art survey, or an article in a scientific journal. She needs to locate items of information to be included in the article and wants to use an automatic system to help with this. The user does not have immediate access to offline libraries and only uses online sources.
The user formulates her information need (the topic) by specifying: • a short topic title (e.g., the title of the survey article), • a free text description of the goals and the intended audience of the article, • a list of languages in which the user is willing to accept the found information, • an optional list of Google retrieval queries that can be used to locate the relevant information; the queries may use site restrictions (see examples below) to express the user’s preferences. Here’s an example of an information need: • topic title: Significance testing • description: I want to write a survey (about 10 screen pages) for undergraduate students on statistical signicfiance testing, with an overview of the ideas, common methods and critiques.
I will assume some basic knowledge of statistics.
• language(s): English
• known source(s): http://en.wikipedia.org/wiki/Statistical_hypothesis_testing ;
http://en.wikipedia.org/wiki/Statistical_significance
• retrieval queries: signicfiance testing; site:mathworld.wolfram.com signicfiance testing;
significance testing pdf; significance testing site:en.wikipedia.org
Defined in this way, the task model corresponds to addressing undirected informational search
goals, that are reported to account for over 23% of web queries [
Each participating team was asked to develop several topics and subsequently assess responses of all participating systems for the created topics and 1.2
Data collection In order to keep the idealized task as close as possible to the real-world scenario (i.e., there are many relevant documents) but still tractable (i.e., the size of the collection is manageable), our collection is defined per topic. Specicfially, for each topic, the subcollection for the topic contains the following set of documents along with their URLs: • all “known” sources speciefid for the topic; • the top 1000 (or less, depending at the actual availability) hits from Google for each of the retrieval queries speciefid in the topic, or for the topic title if the queries are not specified; • for each online document included in the collection, its URL, the original content retrieved from the URL and the plain text conversion of the content are provided. The plain text conversion is only available for HTML, PDF and Postscript documents. For each document, the subcollection also provides its origin: which query or queries were used to locate it and at which rank(s) in the Google result list it was found. 1.3
System response For each topic description, a response of an automatic system consists of a ranked list of plain text snippets extracted from the sub-collection of the topic. Each snippet should indicate what document of the sub-collection it comes from. 2
In order to comply with the task model, the manual assessment of the responses of the systems
was done by the topic creators. The assessment procedure was somewhat similar to assessing
answers to OTHER questions at TREC 2006 Question Answering task [
The assessment was to be blind. For a given topic, all responses of all system were pooled into anonymized sequence of text segments. To limit the amount of required assessments, for each topic only first 7,000 characters of each response were included (according to the ranking of the snippets in the response). The cut-off point 7,000 was chosen so that for at least half of the submitted runs the length of the responses was at least 7,000 for all topics. For the pool created in this way for each topic, the assessor was asked to make a list of nuggets, atomic facts, that, according to the assessor, should be included in the article for the topic. A nugget may be to character spans in the responses, so that all spans linked to one nugget express this atomic fact. Different character spans in one snippet in the response may be linked to more than one nugget. The assessors used a GUI to mark character spans in the responses and link each span to the nugget it expresses (if any). Assessors could also mark character spans as “known” if they expressed fact relevant for the topic but alredy present in one of the known sources.
Figure 1 shows the assessment interface for the topic “iPhone opinions”. Snippets (left bottom of the figure) are separated by grey horisontal lines. Note that only part of the rfist snippet is marked as relevant and the second snippet contain two marked spans that are linked to distinct nuggets. This example illustrates the many-to-many relation between nuggets and character spans in the response.
Similar to INEX [
Table 1 gives the statistics for the 30 test topics and for the assessments of the topics.1 3
The evaluation measures for the task are based on standard precision and recall. For a given response R (a ranked list of text snippets) of a system S for a topic T we define: 1Full definition of the test topics is available from http://ilps.science.uva.nl/WebCLEF/WebCLEF2007/Topics. #o a n p k s l p ta ip to sn #o u n o k s s e g a u g r o s s e s s
T I , PT ,IT ,IT , T T
E E A
B
B B #rak an 16 4 2 1 1 n N ,P ,P ,E ,E ,E ,N ,N ,N ,N ,N ,N ,N a N N ,E N N N N N N N L L L L L N N N N N N N N N N N N N N L E N E SE ,E ,E E E E E E N N N N N E E E E E E E E E E E E E E
S S
E A A A
C C C C C D D D D D E E E E E E E E E E E E E E r e h t o s e g a u g n s t e an R ic n l o app ita - s n t s g u a ire cd ”G o o s f o n ’s ro w co le f d v
n d a f m liittceopT ireogagyhnBBT flfliiItrrsezaoaovyunnnpSdubAmm lifttrrrrececeeeoooabuuhhpudBTTm ´iliiftrssscceeceaooaaaaonpdSBmm ´´fiiiiiittttrsssceceeceoaooaadbSnnumm ıfi´iittrssssscceecceeaoaavnpn sagpuunOmM l)(saaooydndBBm lseeeoaaynhnBDm iiifrrceeanphETAm ilseceaaGGm lijttrrsseeceeaaooaaavvkkdnnhnnw lilifttrrssseeeaoaaaoynnnnpuEm iltsgahnnhE lirrsseceeegagoaohnbnpunE iitrssevpuh ijrszoauuYO iiliItttrrrsssseeoaaggaogadnnuuVVH illisseaoaovydndHM iliilffItrrrrsseeeoaooaoovnnpndCmm litreav llifttt()rreeeecaoaaaduudnhTAmmM iiiseooonnpnhP iillttrssececeeoaooanhnnudhdhTN iiiliiittttrssssseagaooaavunnbnunRD rse ii’ltrrrrssssseceegao”a”agvnhbnnPAD ili:-trecceceoaaagadudnnunpnhHmm lilittttrrseeeceeaaaaovndhnbnnPwm iiiilfttttrrrseeeecaooaaonpnnuRDMiltre”aauddpN iii’Itttsee”gaooa”ooaavvknnbndhBN liiliiiftttrrseeaoaagoagydbudDm il’frrssceoooaoakhdbnndnM
Note that the evaluation measures described above differ slightly from the measures originally proposed in the task description. 2 The original measures were based on the fact that spans are linked to nuggets by assessors: as described in section 2, different spans linked to one nugget are assumed to bear approximately the same factual content. Then, in addition to character-based measures above, a nugget-based recall can be defined based on the number of nuggets (rather than lengths of character spans) found by a system. However, an analysis of the assessments showed that some assessors used nuggets in a way not intended by the assessment guidelines: namely, to group related rather than synonymous character spans. We believe that this misinterpretation of the assessment guidelines indicates that the guidelines are overly complicated and need to be simpliefid in future edition of the task. As a consequence, we did not use nugget-based measures for evaluation. 4
In total, 12 runs were submitted from 4 research groups. To provide a baseline for the task, we created an artificial run: for each topic, a response of the baseline was created as the ranked list of at most 1000 snippets provided by Google in response to retrieval queries from the topic definition. Note that the Google web search engine was designed for a task very different from WebCLEF 2007 (namely, for the task of web page nfiding), and therefore the evaluation results of our baseline can in no way be interpreted as an indication of Google’s performance.
Table 2 shows the submitted runs with the basic statistics: the average length (the number of bytes) of the snippets in the run, the averate number of snippets in the response for one topic, and the average total length of response per topic.
2See http://ilps.science.uva.nl/WebCLEF/WebCLEF2007/Tasks/\#Evaluation_measures. We described WebCLEF 2007. This was the rfist year in which a new task was being assessed, one aimed at undirected information search goals. While the number of participants was limited, we believe the track was a success, as most submitted runs outperformed the Google-based baseline. For the best runs, in top 7,000 bytes per topic about 1/5 of the text was found relevant and important by the assessors.
The WebCLEF 2007 evaluation also raised several important issues. The task denfiition did not specify the exact size of a system’s response for a topic, which has make a comparison across systems problematic. Furthermore, assessor’s guidelines appeared to be overly complicated: not all assessors used nuggets as was intented by the organizers.
Our plans for future work include a detailed per-topic analysis of runs and a study of re-usability of the results of the assessments for improving the systems. 7
Valentin Jijkoun was supported by the Netherlands Organisation for Scientific Research (NWO) under project numbers 220-80-001, 600.065.120 and 612.000.106. Maarten de Rijke was supported