possible, to how a normal user would pro eed, in luding through some intera tion and adaptation Filtering task [Robertson and Soboro, 2002℄ (adaptive ltering tra k has been run from 2000 to 2002), with a parti ular interest in the orresponden e of the proto ol with the ground truth of the ontext of use of ltering systems by real professional users. Even if the ampaign is mainly The INFILE ampaign an mainly be seen as a ross-lingual pursuit of the TREC 2002 Adaptive dened as "events", with a ne granularity level, and often temporally restri ted, whereas in ompetitive intelligen e (CI) professionals. In this goal, we asked CI professionals to write the ontext, the evaluation proto ol of the ampaign has been designed with a parti ular attention to Information ltering has many appli ations (routing, ategorization, email ltering, anti-spamtrained to have good performan e on both tasks [Yang et al., 2005℄. an indu e dieren t te hniques, even if some studies show that some models an be e ien tly a te hnologi al oriented evaluation pro ess, we adapted the proto ol and the metri s, as lose as INFILE (similar to TREC 2002) topi s are of long-term interest and supposed to be stable, whi h topi s a ording to their experien e in the domain. of his system. ming). In the INFILE ampaign, we onsider the ontext of ompetitive intelligen e: in this Other related ampaigns are the Topi Dete tion and Tra king (TDT) ampaigns from 1998 to 2004 [Fis us and Wheatley, 2004℄. However, in the TDT ampaigns, fo us was mainly on topi s when the previous do ument has been ltered; 5. the system an retrieve a new do ument (ba k to step 2) that an only be retrieved 1. a parti ipant system onne ts to the server from whi h its gets a run identier: if a dieren t run identiers; 3. the system lters the do ument, i.e. it asso iates the do ument with one or several parti ipant wants to submit several runs, the system must onne t several times to get 4. for adaptive systems, a relevan e feedba k an be provided for ltered do uments; proles, or dis ard it; 2. the system retrieves one do ument; A dry run has been organized from June 26th to July 3rd to he k the te hni al viability of A simulated relevan e feedba k is provided for adaptive systems: the idea is again to have a on a do ument asso iated with a prole by the system, there is no relevan e feedba k on dis arded of feedba ks is then limited to 50, from the advi e taken from CI professionals. This tends to give automated pro ess, it is also the only feedba k authorized: relevan e feedba k an only be asked Furthermore, we assume that a CI professional would not have an innite patien e: the number task. re eives the do uments found relevant to a prole in a orresponding mailbox or dire tory and simulation of a realist behavior of the CI professional. In a real pro ess, the CI professional the proto ol. The o ial ampaign has been run from July 7th to July 26th. do uments. to be trained, but it seemed right for the organizers to put systems in a the ontext of a realisti he an read the do ument and de ide to remove it if it was a ltering error. In the INFILE more interest to systems with qui k adaptivity, than to systems that needs a large amount of data For INFILE, we sele ted 3 languages (Arabi , English and Fren h) and a 3 years period (2004For ea h language, the 100,000 do uments have been sele ted in the following way: Sin e we provide a real-time simulated feedba k to the parti ipants, we need to have the test. News arti les are en oded in XML format and follow the News Markup Language (NewsML) but are not ne essarily translations from a language to another, sin e the same information is 2006) whi h represents a olle tion of about one and half millions newswires for around 10 GB, Although AFP is the largest Fren h news agen y, it transmits news in other languages su h as spe i ations 6. generally ompletely rewritten from one language to another to mat h the interest of the audien e from whi h 100,000 do uments of ea h language have been sele ted to be used for the ltering in the orresponding ountry.
The INFILE orpus is provided by the Agen e Fran e Presse (AFP) for resear h purpose. AFP is English, Arabi , Spanish, German and Portuguese. Newswires are available in dieren t languages the oldest news agen y in the world and one of the three largest with Asso iated Press and Reuters. identi ation of relevant do uments prior to the ampaign, as in [Soboro and Robertson, 2002℄. one query taking all elds and one query taking all elds but the sample ( onsidering that the sample may introdu e more noise than other elds). This gives a pool of 28 runs.
Ea h sear h engine is queried independently using the 5 dieren t elds of the topi s, plus • in English, but has been modied to also deal with Fren h. The three other engines work in the three languages (English, Fren h, Arabi ).
The whole olle tion has been indexed with 4 dieren t sear h engines: Lu • ene7, Indri8, and our own sear h engine developed at CEA LIST. Zettair is originally only working Zettair9 of Experts model: the rst 10 do uments of ea h run are taken as rst pool and assessed. used: relevant or not relevant. The assessment pro ess has been performed using a Mixture The relevan e of retrieved do uments is judged by human two riteria being • assessors10, are proportional to the s ore)11.
Then, a s ore is omputed for ea h run and ea h topi a ording to the urrent assessments and a next pool is reated by merging the runs using a weighted sum of s ores (where weights (F-measure is then the harmoni mean of the two values).
We used the standard value whi h gives the same importan e to pre ision and re all α = 0.5, perform poorly. To be able to average the value, the measure is s aled as follows: where is the importan e given to a relevant do ument retrieved and is the ost of a not w1 w2 relevant do ument retrieved.
Linear utility is bounded positively (to 1 for a perfe t ltering), but unbounded negatively (negative values depend on the number of relevant do uments for a prole). Hen e, the average value on all proles would give too mu h importan e to the few proles on whi h a systems would 1 F =
α P1 + (1 − α) R1 max( umuax , umin) − umin un =
1 − umin cdet = cmiss × Pmiss × Ptopic + cfalse × Pfalse × (1 − Ptopic) 0.006 0.009 det_10_0.1 0.009 0.052 0.071 0.056 0.082 0.324 0.231 re all F_0.5 it allows equalizing the ontribution of the proles, whose dieren es are supposed to be the main few runs, this measure is not really relevant. by the interest of being at the utting edge of a domain, and not missing the rst information to is the ost of a false alarm cfalse gen y table and ompute the s ores on the resulting table. The rst method is preferred be ause the system uniquely retrieves (among parti ipants). It gives more importan e to systems that use In order to measure the adaptivity of the systems, the measures are also omputed at dieren t is the a priori probability that a do ument is relevant to a given prole. Ptopic sour e of varian e in measures. times in the pro ess, ea h 10,000 do uments, and an evolution urve of the dieren t values a ross To ompute average s ores, the values are rst omputed for ea h prole and then averaged. (MRR) used for instan e in Question Answering Evaluation [Voorhees, 1999℄, but is not omputed on the ranked list of retrieved do uments but on the hronologi al list of the relevant do uments.
The se ond one is an anti ipation measure, designed to give more interest to systems that an Additionally, we proposed two following experimental measures. The rst one is an originality In the INFILE ampaign, we used the values and (a ording cmiss = 10, cfalse = 0.1 Ptopic = 0.001 nd the rst do ument in a given prole. This measure is motivated in ompetitive intelligen e of the do uments), averaged on all proles. The measure is similar to the mean re ipro al rank be rea tive. It is measured by the inverse rank of the rst relevant do ument dete ted (in the list Another way of averaging would be to sum up the values for all proles in ea h ell of the ontinmeasure, dened as a omparative measure orresponding to the number of relevant do uments time is presented. innovative and promising te hnologies that retrieve "di ult" do uments. Sin e we only had too to an estimation of the average ratio of relevant do uments in the orpus). [Voorhees, 1999℄ Voorhees, E. (1999). The tre -8 question answering tra k report. In Pro eedings of the Eighth Text REtrieval Conferen e (TREC-8). NIST. to simulate the in oming of newswires do uments and the intera tion of a user, with a simulated feedba k. Due to delays in the implementation of this setup, the ampaign has been postponed or ommuni ation around this evaluation, or the la k of interest in the subje t. the proto ol is the element that has dis ouraged parti ipants, or if it was the la k of information in July. Only one team parti ipated in the ampaign, whi h at least validated the viability of the intera tive approa h hosen. For the future of this tra k, it has to be veried if the omplexity of http://www.nist.gov/spee h/tests/tdt/1998/do /tdt2.eval.plan.98.v3.7.pdf. [NIST, 1998℄ NIST (1998). The topi dete tion and tra king phase 2 (tdt2) evaluation plan. report. In Pro eedings of the Eighth Text REtrieval Conferen e (TREC-8). NIST. [Hull and Roberston, 1999℄ Hull, D. and Roberston, S. (1999). The tre -8 ltering tra k nal olle tion for tre 2002. In Pro eedings of The Eleventh Text Retrieval Conferen e (TREC 2002). NIST. [Soboro and Robertson, 2002℄ Soboro, I. and Robertson, S. (2002). Building a ltering test