-

Appendix C Results of the Robust Track

Giorgio Maria Di Nunzio

dinunzio@dei.unipd.it 0

Nicola Ferro

ferro@dei.unipd.it 0 0 Department of Information Engineering University of Padua Italy

746 787

Introduction 3

Results for CLEF 2009 Ad-hoc Robust Track 3. Individual Experiment Results and Graphs This section provides the individual results for each official experiment. For each experiment the following tables and graphs are shown: - Overall statistics and information - Interpolated recall vs precision averages plot - Average precision statistics and box plot - Average precision comparison to median plot - Document cutoff levels vs precision at DCL plot - R-Precision statistics and box plot - R-Precision comparison to median plot Topics are identified with DOIs, as well as the experiments. The prefix for the DOI of a topic is 10.2452. The following example shows how to build the DOI for a topic given its number: for topic 200-AH, the corresponding DOI is 10.2452/200-AH List of Submitted Experiments 7

TD TD TD TD TD TDN TDN TD TDN TD TD TD TD TD TD TDN T TDN TD TD TDN TD TDN TD T TD TD TD TD TD TDN T TDN TD TD TDN TD TDN TD TD reina reina reina reina uniba uniba uniba uniba uniba 10.2415/AH-ROBUST-WSD-BILI-X2EN-TEST

CLEF2009.UNIBA.UNIBACROSSWSDTDN AH-ROBUST-WSD-BILI-X2EN-TESTCLEF2009 es es

es es es es es es es

TDN T TDN TD TDN TDN TDN TD TDN Track Overview Results and Graphs 12 Ad−Hoc Robust Monolingual English Test Task Top 5 Participants − Standard Recall Levels vs Mean Interpolated Precision 100% 90% 80% 1 1 141−AH142−AH143−AH144−AH145−AH146−AH147−AH148−AH149−AH150−AH151−AH152−AH153−AH154−AH155−AH156−AH157−AH158−AH159−AH160−AH161−AH162−AH163−AH164−AH165−AH 166−AH167−AH168−AH169−AH170−AH171−AH172−AH173−AH174−AH175−AH176−AH177−AH178−AH179−AH180−AH181−AH182−AH183−AH184−AH185−AH186−AH187−AH188−AH189−AH190−AH 191−AH192−AH193−AH194−AH195−AH196−AH197−AH198−AH199−AH200−AH251−AH252−AH253−AH254−AH255−AH256−AH257−AH258−AH259−AH260−AH261−AH262−AH263−AH264−AH265−AH 266−AH267−AH268−AH269−AH270−AH271−AH272−AH273−AH274−AH275−AH276−AH277−AH278−AH279−AH280−AH281−AH282−AH283−AH284−AH285−AH286−AH287−AH288−AH289−AH290−AH 291−AH292−AH293−AH294−AH295−AH296−AH297−AH298−AH299−AH300−AH301−AH302−AH303−AH304−AH305−AH306−AH307−AH308−AH309−AH310−AH311−AH312−AH313−AH314−AH315−AH 316−AH317−AH318−AH319−AH320−AH321−AH322−AH323−AH324−AH325−AH326−AH327−AH328−AH329−AH330−AH331−AH332−AH333−AH334−AH335−AH336−AH337−AH338−AH339−AH340−AH darmstadt [Experiment DA_4; MAP 45.09%; Not Pooled] reina [Experiment ROB2; MAP 44.52%; Not Pooled] uniba [Experiment UNIBAKRF; MAP 42.50%; Not Pooled] geneva [Experiment ISIENNATTDN; MAP 41.71%; Not Pooled] know−center [Experiment ASSO; MAP 41.70%; Not Pooled] darmstadt [Experiment DA_4; MAP 45.09%; Not Pooled] reina [Experiment ROB2; MAP 44.52%; Not Pooled] uniba [Experiment UNIBAKRF; MAP 42.50%; Not Pooled] geneva [Experiment ISIENNATTDN; MAP 41.71%; Not Pooled] know−center [Experiment ASSO; MAP 41.70%; Not Pooled]

Track Overview Results and Graphs AH-ROBUST-MONO-EN-TEST-CLEF2009 10.2455/TUKEY_T_TEST.91EB3343B3844844935B940F5AEA7190

Ad−Hoc Robust Monolingual English Test Task − Tukey T test with "top group" highlighted

30 100 200 Retrieved Documents (logarithmic scale) 500 1000

Ad−Hoc Robust Monolingual English Test Task Top 5 Participants − Comparison to Median R−Precision by Topic (Topics 141−AH to 165−AH)

Ad−Hoc Robust Monolingual English Test Task Top 5 Participants − Retrieved documents vs Mean Precision 100% 90% 80% 70% 60% 50% 40% 30% 20% 10% 0%

5 1 1 −1 141−AH142−AH143−AH144−AH145−AH146−AH147−AH148−AH149−AH150−AH151−AH152−AH153−AH154−AH155−AH156−AH157−AH158−AH159−AH160−AH161−AH162−AH163−AH164−AH165−AH 166−AH167−AH168−AH169−AH170−AH171−AH172−AH173−AH174−AH175−AH176−AH177−AH178−AH179−AH180−AH181−AH182−AH183−AH184−AH185−AH186−AH187−AH188−AH189−AH190−AH 191−AH192−AH193−AH194−AH195−AH196−AH197−AH198−AH199−AH200−AH251−AH252−AH253−AH254−AH255−AH256−AH257−AH258−AH259−AH260−AH261−AH262−AH263−AH264−AH265−AH 266−AH267−AH268−AH269−AH270−AH271−AH272−AH273−AH274−AH275−AH276−AH277−AH278−AH279−AH280−AH281−AH282−AH283−AH284−AH285−AH286−AH287−AH288−AH289−AH290−AH 291−AH292−AH293−AH294−AH295−AH296−AH297−AH298−AH299−AH300−AH301−AH302−AH303−AH304−AH305−AH306−AH307−AH308−AH309−AH310−AH311−AH312−AH313−AH314−AH315−AH 316−AH317−AH318−AH319−AH320−AH321−AH322−AH323−AH324−AH325−AH326−AH327−AH328−AH329−AH330−AH331−AH332−AH333−AH334−AH335−AH336−AH337−AH338−AH339−AH340−AH darmstadt [Experiment DA_4; R−Prec 43.01%; Not Pooled] reina [Experiment ROB2; R−Prec 43.54%; Not Pooled] uniba [Experiment UNIBAKRF; R−Prec 41.28%; Not Pooled] geneva [Experiment ISIENNATTDN; R−Prec 40.43%; Not Pooled] know−center [Experiment ASSO; R−Prec 40.13%; Not Pooled] darmstadt [Experiment DA_4; R−Prec 43.01%; Not Pooled] reina [Experiment ROB2; R−Prec 43.54%; Not Pooled] uniba [Experiment UNIBAKRF; R−Prec 41.28%; Not Pooled] geneva [Experiment ISIENNATTDN; R−Prec 40.43%; Not Pooled] know−center [Experiment ASSO; R−Prec 40.13%; Not Pooled] geneva [Experiment ISIENNATTDN; R−Prec 40.43%; Not Pooled] geneva [Experiment ISIENNATDOMTDN; R−Prec 40.43%; Not Pooled] s t n e irm geneva [Experiment NAT_TDN_WEB; R−Prec 40.25%; Not Pooled] e p x E know−center [Experiment ASSO; R−Prec 40.13%; Not Pooled] uniba [Experiment UNIBAKTD; R−Prec 39.40%; Not Pooled]

reina [Experiment ROB5; R−Prec 39.12%; Not Pooled] know−center [Experiment SYNSET; R−Prec 38.87%; Not Pooled] know−center [Experiment SYNONYM; R−Prec 38.87%; Not Pooled] valencia [Experiment NLEL0901; R−Prec 38.72%; Not Pooled] ixa [Experiment ENENNOWSD; R−Prec 37.13%; Not Pooled] geneva [Experiment ISILEMTD; R−Prec 36.92%; Not Pooled]

reina [Experiment ROB3; R−Prec 36.49%; Not Pooled] valencia [Experiment NLEL0906; R−Prec 32.98%; Not Pooled] ufrgs [Experiment UFRGS_MONO; R−Prec 31.30%; Not Pooled] geneva [Experiment NUS_WSD_TD; R−Prec 29.53%; Not Pooled] 0% 10% 20% 30% 40% 60% 70% 80% 90%

100% 50%

R−Precision

Ad−Hoc Robust Monolingual English Test Task − Tukey T test with "top group" highlighted reina

Docs Cutoff Levels Precision at DCL (%)

m u N 100% 90% 80% 70% 60% 40% 30% 20% 10% n o i is 50% c e r P 0% 5 10 0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100% 0 0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100% −1 1 −1

Ad−Hoc Robust WSD Bilingual English Test Task − Comparison to Median R−Precision by Topic (Topics 141−AH to 165−AH) 141−AH 142−AH 143−AH 144−AH 145−AH 146−AH 147−AH 148−AH 149−AH 150−AH 151−AH 152−AH 153−AH 154−AH 155−AH 156−AH 157−AH 158−AH 159−AH 160−AH 161−AH 162−AH 163−AH 164−AH 165−AH

Topic Identifier

Ad−Hoc Robust WSD Bilingual English Test Task − Comparison to Median R−Precision by Topic (Topics 166−AH to 190−AH) 166−AH 167−AH 168−AH 169−AH 170−AH 171−AH 172−AH 173−AH 174−AH 175−AH 176−AH 177−AH 178−AH 179−AH 180−AH 181−AH 182−AH 183−AH 184−AH 185−AH 186−AH 187−AH 188−AH 189−AH 190−AH

Topic Identifier

Ad−Hoc Robust WSD Bilingual English Test Task − Comparison to Median R−Precision by Topic (Topics 191−AH to 265−AH) 191−AH 192−AH 193−AH 194−AH 195−AH 196−AH 197−AH 198−AH 199−AH 200−AH 251−AH 252−AH 253−AH 254−AH 255−AH 256−AH 257−AH 258−AH 259−AH 260−AH 261−AH 262−AH 263−AH 264−AH 265−AH

Topic Identifier

Ad−Hoc Robust WSD Bilingual English Test Task − Comparison to Median R−Precision by Topic (Topics 266−AH to 290−AH) 266−AH 267−AH 268−AH 269−AH 270−AH 271−AH 272−AH 273−AH 274−AH 275−AH 276−AH 277−AH 278−AH 279−AH 280−AH 281−AH 282−AH 283−AH 284−AH 285−AH 286−AH 287−AH 288−AH 289−AH 290−AH

Topic Identifier

Ad−Hoc Robust WSD Bilingual English Test Task − Comparison to Median R−Precision by Topic (Topics 291−AH to 315−AH) 291−AH 292−AH 293−AH 294−AH 295−AH 296−AH 297−AH 298−AH 299−AH 300−AH 301−AH 302−AH 303−AH 304−AH 305−AH 306−AH 307−AH 308−AH 309−AH 310−AH 311−AH 312−AH 313−AH 314−AH 315−AH

Topic Identifier

Ad−Hoc Robust WSD Bilingual English Test Task − Comparison to Median R−Precision by Topic (Topics 316−AH to 340−AH) 316−AH 317−AH 318−AH 319−AH 320−AH 321−AH 322−AH 323−AH 324−AH 325−AH 326−AH 327−AH 328−AH 329−AH 330−AH 331−AH 332−AH 333−AH 334−AH 335−AH 336−AH 337−AH 338−AH 339−AH 340−AH

Topic Identifier

Ad−Hoc Robust WSD Bilingual English Test Task − Comparison to Median R−Precision by Topic (Topics 341−AH to 350−AH) 341−AH reina

Precision averages (%) for individual queries 10 20 30 40 50 60 70 80 90 100 uniba for 160 : Total of over all false bilingual keyword

BM25 lucene +

BM25 lucene 60% 40% 30% 20% 10% 0% 0% 50% 10% 20% 30% 40% 60% 70% 80% 90% 100% 0 0% 0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% −1 1 −1

Ad−Hoc Robust WSD Bilingual English Test Task − Comparison to Median Average Precision by Topic (Topics 141−AH to 165−AH) 141−AH 142−AH 143−AH 144−AH 145−AH 146−AH 147−AH 148−AH 149−AH 150−AH 151−AH 152−AH 153−AH 154−AH 155−AH 156−AH 157−AH 158−AH 159−AH 160−AH 161−AH 162−AH 163−AH 164−AH 165−AH

Topic Identifier

Ad−Hoc Robust WSD Bilingual English Test Task − Comparison to Median Average Precision by Topic (Topics 166−AH to 190−AH) 166−AH 167−AH 168−AH 169−AH 170−AH 171−AH 172−AH 173−AH 174−AH 175−AH 176−AH 177−AH 178−AH 179−AH 180−AH 181−AH 182−AH 183−AH 184−AH 185−AH 186−AH 187−AH 188−AH 189−AH 190−AH

Topic Identifier

Ad−Hoc Robust WSD Bilingual English Test Task − Comparison to Median Average Precision by Topic (Topics 191−AH to 265−AH) 191−AH 192−AH 193−AH 194−AH 195−AH 196−AH 197−AH 198−AH 199−AH 200−AH 251−AH 252−AH 253−AH 254−AH 255−AH 256−AH 257−AH 258−AH 259−AH 260−AH 261−AH 262−AH 263−AH 264−AH 265−AH

Topic Identifier

Ad−Hoc Robust WSD Bilingual English Test Task − Comparison to Median Average Precision by Topic (Topics 266−AH to 290−AH) 266−AH 267−AH 268−AH 269−AH 270−AH 271−AH 272−AH 273−AH 274−AH 275−AH 276−AH 277−AH 278−AH 279−AH 280−AH 281−AH 282−AH 283−AH 284−AH 285−AH 286−AH 287−AH 288−AH 289−AH 290−AH

Topic Identifier

Ad−Hoc Robust WSD Bilingual English Test Task − Comparison to Median Average Precision by Topic (Topics 291−AH to 315−AH) 291−AH 292−AH 293−AH 294−AH 295−AH 296−AH 297−AH 298−AH 299−AH 300−AH 301−AH 302−AH 303−AH 304−AH 305−AH 306−AH 307−AH 308−AH 309−AH 310−AH 311−AH 312−AH 313−AH 314−AH 315−AH

Topic Identifier

Ad−Hoc Robust WSD Bilingual English Test Task − Comparison to Median Average Precision by Topic (Topics 316−AH to 340−AH) 316−AH 317−AH 318−AH 319−AH 320−AH 321−AH 322−AH 323−AH 324−AH 325−AH 326−AH 327−AH 328−AH 329−AH 330−AH 331−AH 332−AH 333−AH 334−AH 335−AH 336−AH 337−AH 338−AH 339−AH 340−AH

Topic Identifier

Ad−Hoc Robust WSD Bilingual English Test Task − Comparison to Median Average Precision by Topic (Topics 341−AH to 350−AH) 341−AH 342−AH 343−AH uniba Precision averages (%) for individual queries uniba

Docs

−1 1 −1

Topic Identifier

Ad−Hoc Robust WSD Bilingual English Test Task − Comparison to Median R−Precision by Topic (Topics 341−AH to 350−AH) 341−AH 342−AH 343−AH 344−AH 347−AH 348−AH 349−AH 350−AH 100% 90% 80% 70% 60% 40% 30% 20% 10% 0% 5 10 15 20 30 100 200 500 1000

Retrieved Documents (logarithmic scale) uniba Precision averages (%) for individual queries 10 20 30 40 50 60 70 80 90 100 uniba for 160 : Total of over all false 100% 90% 80% 70% 60% 40% 30% 20% 10% 0% 0% 50% 10% 20% 30% 40% 60% 70% 80% 90% 100% 0 0% 0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% −1 1 −1

Topic Identifier

Ad−Hoc Robust WSD Bilingual English Test Task − Comparison to Median Average Precision by Topic (Topics 341−AH to 350−AH) 341−AH 342−AH 343−AH 344−AH 347−AH 348−AH 349−AH

350−AH uniba Precision averages (%) for individual queries retrieved, where

R = uniba

Docs

55% 60% 65% 70% 75% 80% 85% 90% 1 −1 1 −1

Topic Identifier

Retrieved Documents (logarithmic scale)

Ad−Hoc Robust WSD Bilingual English Test Task − Box plot of the Topics of the Experiment 0 0% 0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% uniba Precision averages (%) for individual queries 10 20 30 40 50 60 70 80 90 100 uniba for 160 : Total of over all false 100% 90% 80% 70% 60% 40% 30% 20% 10% 0% 0% 50% 10% 20% 30% 40% 60% 70% 80% 90% 100%

Ad−Hoc Robust WSD Bilingual English Test Task − Box plot of the Topics of the Experiment 0 0% 0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 1 −1 1 −1

Topic Identifier

Ad−Hoc Robust WSD Bilingual English Test Task − Comparison to Median Average Precision by Topic (Topics 341−AH to 350−AH) 341−AH 342−AH 343−AH 344−AH 347−AH 348−AH 349−AH 350−AH 345−AH 346−AH

Topic Identifier uniba Precision averages (%) for individual queries (precision after

R retrieved, where

R = uniba

Docs

DCL

Ad−Hoc Robust WSD Bilingual English Test Task − Retrieved documents vs Mean Precision UNIBACROSSWSDNL0901 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 1 −1 1 −1

Topic Identifier

Ad−Hoc Robust WSD Bilingual English Test Task − Comparison to Median R−Precision by Topic (Topics 341−AH to 350−AH) 341−AH 342−AH 343−AH 344−AH 347−AH 348−AH 349−AH 350−AH 345−AH 346−AH

Topic Identifier 100% 90% 80% 70% 60% 40% 30% 20% 10% 0%

Retrieved Documents (logarithmic scale)

Ad−Hoc Robust WSD Bilingual English Test Task − Box plot of the Topics of the Experiment 40 0 0% 0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% Ad−Hoc Robust WSD Bilingual English Test Task − Distribution of the Topics of the Experiment uniba Precision averages (%) for individual queries 10 20 30 40 50 60 70 80 90 100 uniba for Total of over all

Mean

title+description (engine: UNIBACROSSWSDTD f o r a l l 100% 90% 80% 70% 60% 40% 30% 20% 10% 0% 0% 50% 10% 20% 30% 40% 60% 70% 80% 90% 100%

Ad−Hoc Robust WSD Bilingual English Test Task − Box plot of the Topics of the Experiment 150 0 0% 0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 1 −1 1 −1

Topic Identifier

Topic Identifier uniba Precision averages (%) for individual queries (precision after

R retrieved, where

R = uniba

Docs

DCL

Ad−Hoc Robust WSD Bilingual English Test Task − Retrieved documents vs Mean Precision UNIBACROSSWSDTD 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 1 −1 −1

Topic Identifier

Topic Identifier 100% 90% 80% 70% 60% 40% 30% 20% 10% 0%

Retrieved Documents (logarithmic scale)

Ad−Hoc Robust WSD Bilingual English Test Task − Box plot of the Topics of the Experiment 150 fT 50 0 0% 0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% Ad−Hoc Robust WSD Bilingual English Test Task − Distribution of the Topics of the Experiment uniba Precision averages (%) for individual queries 10 40 70 100 uniba for Total of over all

Mean AUTOMATIC

160,000 4,327 1,927 (engine: title, description, false UNIBACROSSWSDTDN f o r a l l 100% 90% 80% 70% 60% 40% 30% 20% 10% 0% 0% 50% 10% 20% 30% 40% 60% 70% 80% 90% 100%

Ad−Hoc Robust WSD Bilingual English Test Task − Box plot of the Topics of the Experiment 150 fT 50 o r e b m u N 0 0% 0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 1 −1 −1

Topic Identifier

Topic Identifier uniba Precision averages (%) for individual queries (precision after

R retrieved, where

R = uniba

Docs

DCL

Ad−Hoc Robust WSD Bilingual English Test Task − Retrieved documents vs Mean Precision UNIBACROSSWSDTDN 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 1 −1 −1

Topic Identifier

Topic Identifier 100% 90% 80% 70% 60% 40% 30% 20% 10% 0%

Retrieved Documents (logarithmic scale)

AH-ROBUST-WSD-BILI-X2EN-TESTCLEF2009 AH-ROBUST-WSD-BILI-X2EN-TESTCLEF2009 AH-ROBUST-WSD-BILI-X2EN-TESTCLEF2009 AH-ROBUST-WSD-BILI-X2EN-TESTCLEF2009 AH-ROBUST-WSD-BILI-X2EN-TESTCLEF2009 AH-ROBUST-WSD-BILI-X2EN-TESTCLEF2009 AH-ROBUST-WSD-BILI-X2EN-TESTCLEF2009 AH-ROBUST-WSD-BILI-X2EN-TESTCLEF2009