-

Appendix B Results of the Core Tracks Ad-hoc Robust Tasks

Giorgio Maria Di Nunzio

dinunzio@dei.unipd.it 0

Nicola Ferro

ferro@dei.unipd.it 0 0 Department of Information Engineering University of Padua Italy

329 370

Introduction 3

Results for CLEF 2007 Ad-hoc Robust Track 3. Individual Experiment Results and Graphs This section provides the individual results for each official experiment. For each experiment the following tables and graphs are shown: - Overall statistics and information - Interpolated recall vs precision averages plot - Average precision statistics and box plot - Average precision comparison to median plot - Document cutoff levels vs precision at DCL plot - R-Precision statistics and box plot - R-Precision comparison to median plot Topics are identified with DOIs, as well as the experiments. The prefix for the DOI of a topic is 10.2452. The following example shows how to build the DOI for a topic given its number: for topic 200-AH, the corresponding DOI is 10.2452/200-AH List of Submitted Experiments 7 daedalus xldb xldb xldb xldb unine Track Overview Results and Graphs 11 Ad−Hoc Robust Monolingual French Test Task Top 5 Participants − Standard Recall Levels vs Mean Interpolated Precision 0%

0% 1 0.5 e c n re 0 e iff D −0.5 0.5 e c n re 0 e iff D −0.5 −1 1 −1 1 0.5 e c n re 0 e iff D −0.5 −1 1 0.5 e c n re 0 e iff D −0.5 −1 unine [Experiment UNINEFR1; MAP 42.13%; Not Pooled] reina [Experiment REINAFRTDET; MAP 38.04%; Not Pooled] jaen [Experiment UJARTFR1; MAP 34.76%; Not Pooled] daedalus [Experiment FRFSFR22S; MAP 29.91%; Not Pooled] hildesheim [Experiment HIMOFRBRF2; MAP 27.31%; Not Pooled] unine [Experiment UNINEFR1; MAP 42.13%; Not Pooled] reina [Experiment REINAFRTDET; MAP 38.04%; Not Pooled] jaen [Experiment UJARTFR1; MAP 34.76%; Not Pooled] daedalus [Experiment FRFSFR22S; MAP 29.91%; Not Pooled] hildesheim [Experiment HIMOFRBRF2; MAP 27.31%; Not Pooled] 251−AH 252−AH 253−AH 254−AH 255−AH 256−AH 257−AH 258−AH 259−AH 260−AH 261−AH 262−AH 263−AH 264−AH 265−AH 266−AH 267−AH 268−AH 269−AH 270−AH 271−AH 272−AH 273−AH 274−AH 275−AH 276−AH 277−AH 278−AH 279−AH 280−AH 281−AH 282−AH 283−AH 284−AH 285−AH 286−AH 287−AH 288−AH 289−AH 290−AH 291−AH 292−AH 293−AH 294−AH 295−AH 296−AH 297−AH 298−AH 299−AH 300−AH 301−AH 302−AH 303−AH 304−AH 305−AH 306−AH 307−AH 308−AH 309−AH 310−AH 311−AH 312−AH 313−AH 314−AH 315−AH 316−AH 317−AH 318−AH 319−AH 320−AH 321−AH 322−AH 323−AH 324−AH 325−AH 326−AH 327−AH 328−AH 329−AH 330−AH 331−AH 332−AH 333−AH 334−AH 335−AH 336−AH 337−AH 338−AH 339−AH 340−AH 341−AH 342−AH 343−AH 344−AH 345−AH 346−AH 347−AH 348−AH 349−AH 350−AH

Ad−Hoc Robust Monolingual French Test Task − Box Plot of the Topics daedalus [Experiment FRFSFR22S; MAP 29.91%; Not Pooled] hildesheim [Experiment HIMOFRBRF2; MAP 27.31%; Not Pooled] hildesheim [Experiment HIMOFRBRF; MAP 26.34%; Not Pooled] hildesheim [Experiment HIMOFRBASE; MAP 25.84%; Not Pooled] 0% 10% 20% 30% 40% 50% 60%

Average Precision 70% 80% 90%

100% 10.2455/TUKEY_T_TEST.66736BFA0C417F72BA727E3EF6324986

Ad−Hoc Robust Monolingual French Test Task − Tukey T test with "top group" highlighted

UNINEFR1

5 e c n e r e iff D e c n e r e iff D e c n e r e iff D e c n e r e iff D 1 0 1 0 1 0 1 0 0.5 −0.5 −1 0.5 −0.5 −1 0.5 −0.5 −1 0.5 −0.5 −1 unine [Experiment UNINEFR1; R−Prec 39.29%; Not Pooled] reina [Experiment REINAFRTDET; R−Prec 37.22%; Not Pooled] jaen [Experiment UJARTFR1; R−Prec 33.66%; Not Pooled] daedalus [Experiment FRFSFR22S; R−Prec 29.18%; Not Pooled] hildesheim [Experiment HIMOFRBRF2; R−Prec 27.52%; Not Pooled] unine [Experiment UNINEFR1; R−Prec 39.29%; Not Pooled] reina [Experiment REINAFRTDET; R−Prec 37.22%; Not Pooled] jaen [Experiment UJARTFR1; R−Prec 33.66%; Not Pooled] daedalus [Experiment FRFSFR22S; R−Prec 29.18%; Not Pooled] hildesheim [Experiment HIMOFRBRF2; R−Prec 27.52%; Not Pooled] 251−AH 252−AH 253−AH254−AH 255−AH 256−AH257−AH 258−AH 259−AH260−AH 261−AH 262−AH 263−AH264−AH 265−AH 266−AH267−AH 268−AH 269−AH270−AH 271−AH 272−AH273−AH 274−AH 275−AH 276−AH 277−AH 278−AH279−AH 280−AH 281−AH282−AH 283−AH 284−AH285−AH 286−AH 287−AH 288−AH289−AH 290−AH 291−AH292−AH 293−AH 294−AH295−AH 296−AH 297−AH298−AH 299−AH 300−AH 301−AH 302−AH 303−AH304−AH 305−AH 306−AH307−AH 308−AH 309−AH310−AH 311−AH 312−AH 313−AH314−AH 315−AH 316−AH317−AH 318−AH 319−AH320−AH 321−AH 322−AH323−AH 324−AH 325−AH 326−AH 327−AH 328−AH329−AH 330−AH 331−AH332−AH 333−AH 334−AH335−AH 336−AH 337−AH 338−AH339−AH 340−AH 341−AH342−AH 343−AH 344−AH345−AH 346−AH 347−AH348−AH 349−AH 350−AH Ad−Hoc Robust Monolingual French Test Task − Box Plot of the Topics reina [Experiment REINAFRTT; R−Prec 34.38%; Not Pooled]

jaen [Experiment UJARTFR1; R−Prec 33.66%; Not Pooled] reina [Experiment REINAFRTET; R−Prec 33.32%; Not Pooled] daedalus [Experiment FRFSFR22S; R−Prec 29.18%; Not Pooled] hildesheim [Experiment HIMOFRBRF2; R−Prec 27.52%; Not Pooled] hildesheim [Experiment HIMOFRBRF; R−Prec 26.87%; Not Pooled] hildesheim [Experiment HIMOFRBASE; R−Prec 25.44%; Not Pooled] 0% 10% 20% 30% 40% 60% 70% 80% 90%

100% 50%

R−Precision

Ad−Hoc Robust Monolingual French Test Task − Tukey T test with "top group" highlighted

UNINEFR1

xldb

Overall

statistics for 150 queries : Total number of documents over all queries

Retrieved Relevant Relevant retrieved

fT 50 0 0% 100% 90% 80% 70% 60% 40% 30% 20% 10% 0% 0% 50% 10% 20% 30% 40% 60% 70% 80% 90% 100%

Ad−Hoc Robust Monolingual Portuguese Test Task − Box plot of the Topics of the Experiment 0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90%

Ad−Hoc Robust Monolingual Portuguese Test Task − Distribution of the Topics of the Experiment 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% e 0.5 cn re 0 e f iD−0.5 −1 1 −1 1 −1 1 −1 1 −1 1 −1

Ad−Hoc Robust Monolingual Portuguese Test Task − Comparison to Median Average Precision by Topic (Topics 201−AH to 225−AH) 201−AH 202−AH 203−AH 204−AH 205−AH 206−AH 207−AH 208−AH 209−AH 210−AH 211−AH 212−AH 213−AH 214−AH 215−AH 216−AH 217−AH 218−AH 219−AH 220−AH 221−AH 222−AH 223−AH 224−AH 225−AH

Topic Identifier

Ad−Hoc Robust Monolingual Portuguese Test Task − Comparison to Median Average Precision by Topic (Topics 226−AH to 250−AH) 226−AH 227−AH 228−AH 229−AH 230−AH 231−AH 232−AH 233−AH 234−AH 235−AH 236−AH 237−AH 238−AH 239−AH 240−AH 241−AH 242−AH 243−AH 244−AH 245−AH 246−AH 247−AH 248−AH 249−AH 250−AH

Topic Identifier

Ad−Hoc Robust Monolingual Portuguese Test Task − Comparison to Median Average Precision by Topic (Topics 251−AH to 275−AH) 251−AH 252−AH 253−AH 254−AH 255−AH 256−AH 257−AH 258−AH 259−AH 260−AH 261−AH 262−AH 263−AH 264−AH 265−AH 266−AH 267−AH 268−AH 269−AH 270−AH 271−AH 272−AH 273−AH 274−AH 275−AH

Topic Identifier

Ad−Hoc Robust Monolingual Portuguese Test Task − Comparison to Median Average Precision by Topic (Topics 276−AH to 300−AH) 276−AH 277−AH 278−AH 279−AH 280−AH 281−AH 282−AH 283−AH 284−AH 285−AH 286−AH 287−AH 288−AH 289−AH 290−AH 291−AH 292−AH 293−AH 294−AH 295−AH 296−AH 297−AH 298−AH 299−AH 300−AH

Topic Identifier

Ad−Hoc Robust Monolingual Portuguese Test Task − Comparison to Median Average Precision by Topic (Topics 301−AH to 325−AH) 301−AH 302−AH 303−AH 304−AH 305−AH 306−AH 307−AH 308−AH 309−AH 310−AH 311−AH 312−AH 313−AH 314−AH 315−AH 316−AH 317−AH 318−AH 319−AH 320−AH 321−AH 322−AH 323−AH 324−AH 325−AH

Topic Identifier

Ad−Hoc Robust Monolingual Portuguese Test Task − Comparison to Median Average Precision by Topic (Topics 326−AH to 350−AH) 326−AH 327−AH 328−AH 329−AH 330−AH 331−AH 332−AH 333−AH 334−AH 335−AH 336−AH 337−AH 338−AH 339−AH 340−AH 341−AH 342−AH 343−AH 344−AH 345−AH 346−AH 347−AH 348−AH 349−AH 350−AH Topic Identifier xldb after

R document retrieved, where

R = xldb

Docs Cutoff Levels

fT 50 0 0% 100% 90% 80% 70% 60% 40% 30% 20% 10% 0% 5 10 15 20 30 100 200 500 1000

Retrieved Documents (logarithmic scale)

Ad−Hoc Robust Monolingual Portuguese Test Task − Box plot of the Topics of the Experiment 0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% ce 0.5 n re 0 e f iD−0.5 −1

Ad−Hoc Robust Monolingual Portuguese Test Task − Comparison to Median R−Precision by Topic (Topics 201−AH to 225−AH) 201−AH 202−AH 203−AH 204−AH 205−AH 206−AH 207−AH 208−AH 209−AH 210−AH 211−AH 212−AH 213−AH 214−AH 215−AH 216−AH 217−AH 218−AH 219−AH 220−AH 221−AH 222−AH 223−AH 224−AH 225−AH

Topic Identifier

Ad−Hoc Robust Monolingual Portuguese Test Task − Comparison to Median R−Precision by Topic (Topics 226−AH to 250−AH) 226−AH 227−AH 228−AH 229−AH 230−AH 231−AH 232−AH 233−AH 234−AH 235−AH 236−AH 237−AH 238−AH 239−AH 240−AH 241−AH 242−AH 243−AH 244−AH 245−AH 246−AH 247−AH 248−AH 249−AH 250−AH

Topic Identifier

Ad−Hoc Robust Monolingual Portuguese Test Task − Comparison to Median R−Precision by Topic (Topics 251−AH to 275−AH) 251−AH 252−AH 253−AH 254−AH 255−AH 256−AH 257−AH 258−AH 259−AH 260−AH 261−AH 262−AH 263−AH 264−AH 265−AH 266−AH 267−AH 268−AH 269−AH 270−AH 271−AH 272−AH 273−AH 274−AH 275−AH

Topic Identifier

Ad−Hoc Robust Monolingual Portuguese Test Task − Comparison to Median R−Precision by Topic (Topics 276−AH to 300−AH) 276−AH 277−AH 278−AH 279−AH 280−AH 281−AH 282−AH 283−AH 284−AH 285−AH 286−AH 287−AH 288−AH 289−AH 290−AH 291−AH 292−AH 293−AH 294−AH 295−AH 296−AH 297−AH 298−AH 299−AH 300−AH

Topic Identifier

Ad−Hoc Robust Monolingual Portuguese Test Task − Comparison to Median R−Precision by Topic (Topics 301−AH to 325−AH) 301−AH 302−AH 303−AH 304−AH 305−AH 306−AH 307−AH 308−AH 309−AH 310−AH 311−AH 312−AH 313−AH 314−AH 315−AH 316−AH 317−AH 318−AH 319−AH 320−AH 321−AH 322−AH 323−AH 324−AH 325−AH

Topic Identifier

Ad−Hoc Robust Monolingual Portuguese Test Task − Comparison to Median R−Precision by Topic (Topics 326−AH to 350−AH) 326−AH 327−AH 328−AH 329−AH 330−AH 331−AH 332−AH 333−AH 334−AH 335−AH 336−AH 337−AH 338−AH 339−AH 340−AH 341−AH 342−AH 343−AH 344−AH 345−AH 346−AH 347−AH 348−AH 349−AH 350−AH Topic Identifier xldb

Precision averages (%) for individual queries 0% 0% 10% 20% 30% 40% 60% 70% 80% 90%

100% EN2FRTST4GRINTDICEU001 EN2FRTST4GRINTDICEU001 EN2FRTST4GRINTDICEU001 EN2FRTST4GRINTDICEU001

Ad−Hoc Robust Bilingual Test Task, French target col ection(s) − Box plot of the Topics of the Experiment

Ad−Hoc Robust Bilingual Test Task, French target col ection(s) − Distribution of the Topics of the Experiment 00% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%

Average Precision

Ad−Hoc Robust Bilingual Test Task, French target colection(s) − Comparison to Median Average Precision by Topic (Topics 251−AH to 275−AH) 1 0.5 Ad−Hoc Robust Bilingual Test Task, French target collection(s) − Retrieved documents vs Mean Precision 100% 0%5 10 15 20 Precision averages (%) for individual queries 251-AH 40.00 276-AH 29.41 301-AH 36.00 326-AH 0.00 252-AH 0.00 277-AH 37.50 302-AH 0.00 327-AH 0.00 253-AH 3.66 278-AH 0.00 303-AH 4.00 328-AH 35.71 254-AH 34.78 279-AH 25.00 304-AH 20.69 329-AH 60.00 255-AH 0.00 280-AH 3.57 305-AH 10.53 330-AH 100.00 256-AH 57.14 281-AH 0.00 306-AH 0.00 331-AH 10.00 257-AH 8.82 282-AH 0.00 307-AH 28.57 332-AH 0.00 258-AH 0.00 283-AH 36.84 308-AH 57.14 333-AH 73.91 259-AH 0.00 284-AH 45.95 309-AH 28.57 334-AH 50.00 260-AH 25.00 285-AH 35.00 310-AH 3.85 335-AH 60.00 261-AH 0.00 286-AH 0.00 311-AH 31.86 336-AH 0.00 262-AH 25.00 287-AH 8.06 312-AH 16.67 337-AH 68.15 263-AH 36.36 288-AH 6.25 313-AH 10.14 338-AH 30.77 264-AH 67.57 289-AH 28.57 314-AH 7.69 339-AH 62.50 265-AH 0.00 290-AH 19.44 315-AH 20.83 340-AH 50.00 266-AH 6.67 291-AH 8.70 316-AH 45.89 341-AH 30.00 267-AH 0.00 292-AH 0.00 317-AH 60.00 342-AH 0.00 268-AH 33.33 293-AH 57.89 318-AH 0.00 343-AH 21.88 269-AH 16.00 294-AH 0.00 319-AH 43.75 344-AH 44.44 270-AH 60.00 295-AH 17.78 320-AH 0.00 345-AH 12.50 271-AH 84.62 296-AH 0.00 321-AH 0.00 346-AH 0.00 272-AH 0.00 297-AH 51.35 322-AH 0.00 347-AH 20.00 273-AH 16.39 298-AH 15.49 323-AH 17.50 348-AH 60.87 274-AH 20.00 299-AH 0.00 324-AH 15.00 349-AH 66.67 275-AH 17.50 300-AH 0.00 325-AH 0.00 350-AH 37.50

Ad−Hoc Robust Bilingual Test Task, French target col ection(s) − Box plot of the Topics of the Experiment

Ad−Hoc Robust Bilingual Test Task, French target col ection(s) − Distribution of the Topics of the Experiment 00% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%

R−Precision

Ad−Hoc Robust Bilingual Test Task, French target colection(s) − Comparison to Median R−Precision by Topic (Topics 251−AH to 275−AH) 1 0.5 EN2FRTST4GRINTDICEU001 EN2FRTST4GRINTDICEU001 EN2FRTST4GRINTDICEU001

Priority Query Construction Source Language Topic Fields Pooled

EN2FR (Test); bidir. alignment; Log-Likelihood; N=1 0% 0% 100% Precision averages (%) for individual queries 251-AH 24.03 276-AH 15.58 301-AH 41.17 326-AH 0.00 252-AH 0.18 277-AH 41.60 302-AH 3.10 327-AH 0.00 253-AH 0.25 278-AH 0.59 303-AH 4.34 328-AH 59.98 254-AH 14.07 279-AH 14.88 304-AH 8.43 329-AH 83.50 255-AH 0.00 280-AH 1.56 305-AH 10.69 330-AH 100.00 256-AH 79.52 281-AH 0.33 306-AH 1.74 331-AH 16.72 257-AH 7.49 282-AH 2.26 307-AH 23.57 332-AH 0.00 258-AH 0.00 283-AH 37.07 308-AH 58.50 333-AH 71.10 259-AH 10.91 284-AH 46.39 309-AH 24.36 334-AH 55.34 260-AH 12.41 285-AH 28.33 310-AH 2.08 335-AH 50.69 261-AH 0.00 286-AH 1.80 311-AH 13.16 336-AH 2.63 262-AH 30.41 287-AH 5.41 312-AH 23.34 337-AH 67.41 263-AH 5.40 288-AH 9.95 313-AH 5.87 338-AH 30.86 264-AH 69.03 289-AH 16.67 314-AH 7.35 339-AH 62.10 265-AH 6.50 290-AH 10.41 315-AH 18.39 340-AH 48.70 266-AH 2.55 291-AH 3.92 316-AH 43.43 341-AH 7.11 267-AH 0.87 292-AH 0.60 317-AH 57.16 342-AH 0.06 268-AH 75.56 293-AH 59.72 318-AH 0.00 343-AH 15.68 269-AH 8.99 294-AH 0.00 319-AH 44.04 344-AH 50.48 270-AH 62.33 295-AH 9.37 320-AH 0.86 345-AH 21.43 271-AH 82.38 296-AH 1.27 321-AH 0.00 346-AH 0.00 272-AH 4.53 297-AH 53.98 322-AH 3.59 347-AH 11.73 273-AH 15.51 298-AH 12.67 323-AH 9.11 348-AH 57.98 274-AH 49.85 299-AH 0.00 324-AH 0.91 349-AH 58.64 275-AH 17.54 300-AH 0.00 325-AH 0.04 350-AH 54.61

Ad−Hoc Robust Bilingual Test Task, French target col ection(s) − Box plot of the Topics of the Experiment

Ad−Hoc Robust Bilingual Test Task, French target col ection(s) − Distribution of the Topics of the Experiment 00% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%

Average Precision

Ad−Hoc Robust Bilingual Test Task, French target colection(s) − Comparison to Median Average Precision by Topic (Topics 251−AH to 275−AH) 1 0.5 EN2FRTST4GRINTLOGLU001 EN2FRTST4GRINTLOGLU001 EN2FRTST4GRINTLOGLU001 10.2415/AH-ROBUST-BILI-X2FR-TEST-CLEF2007.COLESUN.EN2FRTST4GRINTLOGLU001 AH-ROBUST-BILI-X2FR-TEST-CLEF2007

Ad−Hoc Robust Bilingual Test Task, French target collection(s) − Retrieved documents vs Mean Precision 100% 0%5 10 15 20 Precision averages (%) for individual queries 251-AH 35.00 276-AH 23.53 301-AH 44.00 326-AH 0.00 252-AH 0.00 277-AH 37.50 302-AH 0.00 327-AH 0.00 253-AH 3.66 278-AH 0.00 303-AH 4.00 328-AH 50.00 254-AH 11.59 279-AH 25.00 304-AH 17.24 329-AH 80.00 255-AH 0.00 280-AH 3.57 305-AH 0.00 330-AH 100.00 256-AH 71.43 281-AH 0.00 306-AH 0.00 331-AH 10.00 257-AH 11.76 282-AH 0.00 307-AH 28.57 332-AH 0.00 258-AH 0.00 283-AH 36.84 308-AH 71.43 333-AH 72.83 259-AH 0.00 284-AH 43.24 309-AH 28.57 334-AH 66.67 260-AH 25.00 285-AH 37.50 310-AH 7.69 335-AH 65.00 261-AH 0.00 286-AH 0.00 311-AH 26.55 336-AH 0.00 262-AH 25.00 287-AH 8.06 312-AH 16.67 337-AH 65.19 263-AH 9.09 288-AH 12.50 313-AH 11.59 338-AH 30.77 264-AH 67.57 289-AH 28.57 314-AH 7.69 339-AH 57.50 265-AH 0.00 290-AH 11.11 315-AH 29.17 340-AH 57.14 266-AH 6.67 291-AH 8.70 316-AH 45.89 341-AH 20.00 267-AH 0.00 292-AH 0.00 317-AH 60.00 342-AH 0.00 268-AH 66.67 293-AH 70.18 318-AH 0.00 343-AH 15.62 269-AH 18.00 294-AH 0.00 319-AH 43.75 344-AH 44.44 270-AH 60.00 295-AH 15.56 320-AH 0.00 345-AH 12.50 271-AH 76.92 296-AH 0.00 321-AH 0.00 346-AH 0.00 272-AH 0.00 297-AH 51.35 322-AH 0.00 347-AH 20.00 273-AH 16.39 298-AH 14.08 323-AH 17.50 348-AH 60.87 274-AH 50.00 299-AH 0.00 324-AH 5.00 349-AH 66.67 275-AH 22.50 300-AH 0.00 325-AH 0.00 350-AH 37.50

Ad−Hoc Robust Bilingual Test Task, French target col ection(s) − Box plot of the Topics of the Experiment

Ad−Hoc Robust Bilingual Test Task, French target col ection(s) − Distribution of the Topics of the Experiment 00% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%

R−Precision

Ad−Hoc Robust Bilingual Test Task, French target colection(s) − Comparison to Median R−Precision by Topic (Topics 251−AH to 275−AH) 1 0.5 EN2FRTST4GRINTLOGLU001 EN2FRTST4GRINTLOGLU001 EN2FRTST4GRINTLOGLU001

Priority Query Construction Source Language Topic Fields Pooled 3 AUTOMATIC English

title, description false EN2FR (Test); bidir. alignment; Mutual Information; N=10 Ad−Hoc Robust Bilingual Test Task, French target collection(s) − Standard Recall Levels vs Mean Interpolated Precision 100% 0% 0% 100% Precision averages (%) for individual queries 251-AH 39.27 276-AH 11.94 301-AH 27.17 326-AH 0.00 252-AH 0.53 277-AH 52.30 302-AH 2.25 327-AH 0.00 253-AH 0.07 278-AH 16.38 303-AH 1.23 328-AH 37.59 254-AH 66.52 279-AH 12.65 304-AH 17.14 329-AH 53.70 255-AH 0.00 280-AH 0.43 305-AH 6.45 330-AH 4.72 256-AH 80.70 281-AH 0.26 306-AH 0.47 331-AH 15.02 257-AH 4.51 282-AH 0.16 307-AH 2.13 332-AH 0.00 258-AH 0.00 283-AH 2.14 308-AH 0.06 333-AH 52.92 259-AH 3.73 284-AH 22.30 309-AH 7.68 334-AH 49.26 260-AH 17.74 285-AH 21.97 310-AH 1.22 335-AH 11.22 261-AH 0.00 286-AH 2.47 311-AH 25.38 336-AH 0.44 262-AH 0.00 287-AH 4.30 312-AH 2.27 337-AH 57.37 263-AH 10.22 288-AH 1.23 313-AH 5.42 338-AH 26.63 264-AH 66.86 289-AH 0.39 314-AH 11.31 339-AH 58.28 265-AH 11.07 290-AH 3.25 315-AH 2.84 340-AH 35.07 266-AH 1.75 291-AH 3.81 316-AH 43.36 341-AH 3.01 267-AH 0.01 292-AH 1.54 317-AH 11.84 342-AH 1.68 268-AH 13.18 293-AH 42.85 318-AH 0.28 343-AH 5.39 269-AH 0.14 294-AH 0.00 319-AH 4.07 344-AH 49.89 270-AH 69.84 295-AH 35.65 320-AH 4.16 345-AH 10.21 271-AH 38.15 296-AH 0.56 321-AH 0.00 346-AH 0.00 272-AH 8.64 297-AH 60.32 322-AH 1.91 347-AH 0.11 273-AH 7.77 298-AH 4.45 323-AH 6.08 348-AH 28.62 274-AH 10.37 299-AH 0.00 324-AH 2.97 349-AH 14.85 275-AH 43.62 300-AH 0.00 325-AH 0.61 350-AH 50.06

Ad−Hoc Robust Bilingual Test Task, French target col ection(s) − Box plot of the Topics of the Experiment

Ad−Hoc Robust Bilingual Test Task, French target col ection(s) − Distribution of the Topics of the Experiment 00% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%

Average Precision

Ad−Hoc Robust Bilingual Test Task, French target colection(s) − Comparison to Median Average Precision by Topic (Topics 251−AH to 275−AH) 1 0.5 EN2FRTST4GRINTMIU010 EN2FRTST4GRINTMIU010 EN2FRTST4GRINTMIU010 10.2415/AH-ROBUST-BILI-X2FR-TEST-CLEF2007.COLESUN.EN2FRTST4GRINTMIU010 AH-ROBUST-BILI-X2FR-TEST-CLEF2007

Ad−Hoc Robust Bilingual Test Task, French target collection(s) − Retrieved documents vs Mean Precision 100% 0%5 10 15 20 Docs Cutoff Levels Precision at DCL (%) 5 docs 19.20 10 docs 18.80 15 docs 17.53 20 docs 16.45 30 docs 14.93 100 docs 9.44 200 docs 6.28 500 docs 3.28 1000 docs 1.87 R-Precision (precision after R document retrieved, where R = Relevant retrieved)

15.94 Precision averages (%) for individual queries 251-AH 45.00 276-AH 17.65 301-AH 24.00 326-AH 0.00 252-AH 0.00 277-AH 50.00 302-AH 0.00 327-AH 0.00 253-AH 0.00 278-AH 30.77 303-AH 0.00 328-AH 42.86 254-AH 63.77 279-AH 25.00 304-AH 24.14 329-AH 40.00 255-AH 0.00 280-AH 3.57 305-AH 0.00 330-AH 0.00 256-AH 71.43 281-AH 0.00 306-AH 0.00 331-AH 10.00 257-AH 8.82 282-AH 0.00 307-AH 0.00 332-AH 0.00 258-AH 0.00 283-AH 5.26 308-AH 0.00 333-AH 56.52 259-AH 0.00 284-AH 21.62 309-AH 0.00 334-AH 50.00 260-AH 25.00 285-AH 30.00 310-AH 3.85 335-AH 0.00 261-AH 0.00 286-AH 0.00 311-AH 34.51 336-AH 0.00 262-AH 0.00 287-AH 11.29 312-AH 0.00 337-AH 57.78 263-AH 18.18 288-AH 0.00 313-AH 7.25 338-AH 30.77 264-AH 62.16 289-AH 0.00 314-AH 7.69 339-AH 57.50 265-AH 21.43 290-AH 5.56 315-AH 8.33 340-AH 57.14 266-AH 6.67 291-AH 4.35 316-AH 40.58 341-AH 0.00 267-AH 0.00 292-AH 0.00 317-AH 0.00 342-AH 10.00 268-AH 0.00 293-AH 40.35 318-AH 0.00 343-AH 6.25 269-AH 0.00 294-AH 0.00 319-AH 12.50 344-AH 44.44 270-AH 65.00 295-AH 42.22 320-AH 0.00 345-AH 8.33 271-AH 46.15 296-AH 0.00 321-AH 0.00 346-AH 0.00 272-AH 0.00 297-AH 54.05 322-AH 0.00 347-AH 0.00 273-AH 13.11 298-AH 7.04 323-AH 20.00 348-AH 32.61 274-AH 10.00 299-AH 0.00 324-AH 5.00 349-AH 16.67 275-AH 42.50 300-AH 0.00 325-AH 0.00 350-AH 37.50

Ad−Hoc Robust Bilingual Test Task, French target col ection(s) − Box plot of the Topics of the Experiment

Ad−Hoc Robust Bilingual Test Task, French target col ection(s) − Distribution of the Topics of the Experiment 00% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%

R−Precision

Ad−Hoc Robust Bilingual Test Task, French target colection(s) − Comparison to Median R−Precision by Topic (Topics 251−AH to 275−AH) 1 0.5 EN2FRTST4GRINTMIU010 EN2FRTST4GRINTMIU010 EN2FRTST4GRINTMIU010 reina 10.2415/AH-ROBUST-BILI-X2FR-TEST-CLEF2007.REINA.REINAE2FTDET Merging MT of TD English topics, expansion using windows of terms 0% 0% 10% 20% 30% 40% 60% 70% 80% 90% 100% Precision averages (%) for individual queries 251-AH 11.44 276-AH 16.27 301-AH 57.88 326-AH 0.00 252-AH 9.71 277-AH 51.99 302-AH 9.73 327-AH 0.00 253-AH 24.96 278-AH 35.42 303-AH 20.95 328-AH 57.96 254-AH 74.22 279-AH 9.21 304-AH 16.19 329-AH 79.44 255-AH 0.00 280-AH 23.87 305-AH 14.87 330-AH 100.00 256-AH 72.30 281-AH 0.47 306-AH 10.44 331-AH 10.58 257-AH 11.26 282-AH 4.33 307-AH 31.14 332-AH 0.00 258-AH 0.00 283-AH 50.39 308-AH 75.74 333-AH 87.51 259-AH 45.89 284-AH 61.76 309-AH 28.04 334-AH 73.29 260-AH 33.83 285-AH 65.11 310-AH 3.57 335-AH 52.40 261-AH 0.00 286-AH 39.67 311-AH 14.41 336-AH 5.26 262-AH 9.25 287-AH 59.50 312-AH 30.20 337-AH 78.41 263-AH 42.53 288-AH 25.84 313-AH 19.01 338-AH 41.29 264-AH 81.26 289-AH 89.66 314-AH 15.88 339-AH 72.23 265-AH 20.87 290-AH 8.21 315-AH 3.75 340-AH 58.31 266-AH 12.59 291-AH 7.06 316-AH 44.02 341-AH 100.00 267-AH 31.47 292-AH 1.40 317-AH 20.43 342-AH 60.25 268-AH 71.93 293-AH 76.24 318-AH 50.00 343-AH 28.63 269-AH 13.27 294-AH 0.00 319-AH 44.04 344-AH 46.52 270-AH 70.51 295-AH 54.73 320-AH 1.25 345-AH 47.76 271-AH 52.01 296-AH 24.43 321-AH 0.00 346-AH 0.00 272-AH 8.26 297-AH 59.80 322-AH 8.30 347-AH 39.84 273-AH 52.40 298-AH 36.67 323-AH 23.53 348-AH 70.73 274-AH 77.76 299-AH 6.31 324-AH 1.23 349-AH 76.17 275-AH 12.98 300-AH 23.97 325-AH 1.70 350-AH 48.82 REINAE2FTDET REINAE2FTDET REINAE2FTDET

REINAE2FTDET

Ad−Hoc Robust Bilingual Test Task, French target col ection(s) − Box plot of the Topics of the Experiment

Ad−Hoc Robust Bilingual Test Task, French target col ection(s) − Distribution of the Topics of the Experiment 00% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%

Average Precision

Ad−Hoc Robust Bilingual Test Task, French target colection(s) − Comparison to Median Average Precision by Topic (Topics 251−AH to 275−AH) 1 0.5 reina 10.2415/AH-ROBUST-BILI-X2FR-TEST-CLEF2007.REINA.REINAE2FTDET AH-ROBUST-BILI-X2FR-TEST-CLEF2007 0%5 10 15 20 Docs Cutoff Levels Precision at DCL (%) 5 docs 43.80 10 docs 38.70 15 docs 33.27 20 docs 30.25 30 docs 26.47 100 docs 14.26 200 docs 8.81 500 docs 4.11 1000 docs 2.22 R-Precision (precision after R document retrieved, where R = Relevant retrieved)

33.38 Precision averages (%) for individual queries 251-AH 25.00 276-AH 17.65 301-AH 56.00 326-AH 0.00 252-AH 16.67 277-AH 50.00 302-AH 0.00 327-AH 0.00 253-AH 29.27 278-AH 38.46 303-AH 24.00 328-AH 50.00 254-AH 68.12 279-AH 5.00 304-AH 24.14 329-AH 60.00 255-AH 0.00 280-AH 32.14 305-AH 15.79 330-AH 100.00 256-AH 71.43 281-AH 0.00 306-AH 0.00 331-AH 0.00 257-AH 23.53 282-AH 12.50 307-AH 14.29 332-AH 0.00 258-AH 0.00 283-AH 42.11 308-AH 85.71 333-AH 82.61 259-AH 40.00 284-AH 54.05 309-AH 28.57 334-AH 66.67 260-AH 43.75 285-AH 60.00 310-AH 15.38 335-AH 45.00 261-AH 0.00 286-AH 25.00 311-AH 19.47 336-AH 0.00 262-AH 25.00 287-AH 54.84 312-AH 33.33 337-AH 71.85 263-AH 36.36 288-AH 31.25 313-AH 27.54 338-AH 46.15 264-AH 74.32 289-AH 78.57 314-AH 7.69 339-AH 62.50 265-AH 21.43 290-AH 13.89 315-AH 8.33 340-AH 64.29 266-AH 13.33 291-AH 13.04 316-AH 47.34 341-AH 100.00 267-AH 33.33 292-AH 0.00 317-AH 20.00 342-AH 60.00 268-AH 66.67 293-AH 73.68 318-AH 0.00 343-AH 40.62 269-AH 20.00 294-AH 0.00 319-AH 43.75 344-AH 33.33 270-AH 60.00 295-AH 53.33 320-AH 0.00 345-AH 37.50 271-AH 53.85 296-AH 16.67 321-AH 0.00 346-AH 0.00 272-AH 0.00 297-AH 56.76 322-AH 14.29 347-AH 40.00 273-AH 52.46 298-AH 40.85 323-AH 27.50 348-AH 63.04 274-AH 80.00 299-AH 15.15 324-AH 5.00 349-AH 66.67 275-AH 12.50 300-AH 15.38 325-AH 0.00 350-AH 62.50

Ad−Hoc Robust Bilingual Test Task, French target col ection(s) − Box plot of the Topics of the Experiment

Ad−Hoc Robust Bilingual Test Task, French target col ection(s) − Distribution of the Topics of the Experiment 00% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%

R−Precision

Ad−Hoc Robust Bilingual Test Task, French target colection(s) − Comparison to Median R−Precision by Topic (Topics 251−AH to 275−AH) 1 0.5 REINAE2FTDET REINAE2FTDET REINAE2FTDET reina 10.2415/AH-ROBUST-BILI-X2FR-TEST-CLEF2007.REINA.REINAE2FTDNT 5

AUTOMATIC English title, description, narrative false Mergin MT of EN-ti+de+na topics, no expansion

Ad−Hoc Robust Bilingual Test Task, French target collection(s) − Standard Recall Levels vs Mean Interpolated Precision 100% 0% 0% 10% 20% 30% 40% 60% 70% 80% 90% 100% Precision averages (%) for individual queries 251-AH 7.67 276-AH 14.40 301-AH 53.47 326-AH 0.00 252-AH 30.22 277-AH 47.90 302-AH 7.60 327-AH 0.00 253-AH 19.04 278-AH 23.11 303-AH 16.95 328-AH 49.99 254-AH 68.25 279-AH 9.01 304-AH 21.53 329-AH 85.29 255-AH 0.00 280-AH 21.00 305-AH 20.91 330-AH 100.00 256-AH 75.42 281-AH 0.56 306-AH 8.01 331-AH 44.68 257-AH 10.66 282-AH 15.88 307-AH 53.68 332-AH 0.00 258-AH 0.00 283-AH 26.91 308-AH 72.19 333-AH 84.08 259-AH 55.48 284-AH 63.65 309-AH 17.68 334-AH 75.79 260-AH 36.97 285-AH 70.98 310-AH 15.11 335-AH 52.72 261-AH 33.89 286-AH 56.81 311-AH 26.57 336-AH 100.00 262-AH 6.70 287-AH 56.79 312-AH 44.37 337-AH 82.91 263-AH 42.36 288-AH 23.80 313-AH 14.74 338-AH 38.45 264-AH 65.07 289-AH 72.67 314-AH 16.51 339-AH 62.15 265-AH 19.90 290-AH 14.06 315-AH 7.58 340-AH 43.45 266-AH 22.81 291-AH 7.63 316-AH 40.38 341-AH 100.00 267-AH 27.85 292-AH 6.44 317-AH 43.25 342-AH 51.28 268-AH 68.99 293-AH 71.61 318-AH 4.17 343-AH 34.19 269-AH 12.34 294-AH 0.00 319-AH 56.03 344-AH 47.52 270-AH 66.94 295-AH 54.70 320-AH 2.21 345-AH 52.92 271-AH 34.97 296-AH 33.13 321-AH 0.00 346-AH 0.00 272-AH 9.67 297-AH 58.08 322-AH 12.60 347-AH 24.50 273-AH 49.70 298-AH 31.37 323-AH 25.68 348-AH 70.52 274-AH 62.36 299-AH 4.39 324-AH 6.61 349-AH 70.78 275-AH 42.26 300-AH 21.32 325-AH 2.02 350-AH 46.65

Ad−Hoc Robust Bilingual Test Task, French target col ection(s) − Box plot of the Topics of the Experiment 0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%

Average Precision

Ad−Hoc Robust Bilingual Test Task, French target col ection(s) − Distribution of the Topics of the Experiment 00% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%

Average Precision

Ad−Hoc Robust Bilingual Test Task, French target colection(s) − Comparison to Median Average Precision by Topic (Topics 251−AH to 275−AH) 1 0.5 REINAE2FTDNT REINAE2FTDNT REINAE2FTDNT reina 10.2415/AH-ROBUST-BILI-X2FR-TEST-CLEF2007.REINA.REINAE2FTDNT AH-ROBUST-BILI-X2FR-TEST-CLEF2007 20% 10% 0%5

30 100 200 Retrieved Documents (logarithmic scale) Docs Cutoff Levels Precision at DCL (%) 5 docs 42.80 10 docs 38.30 15 docs 33.40 20 docs 30.20 30 docs 26.13 100 docs 14.69 200 docs 8.90 500 docs 4.20 1000 docs 2.24 R-Precision (precision after R document retrieved, where R = Relevant retrieved)

35.28 Precision averages (%) for individual queries 251-AH 15.00 276-AH 17.65 301-AH 48.00 326-AH 0.00 252-AH 16.67 277-AH 62.50 302-AH 0.00 327-AH 0.00 253-AH 24.39 278-AH 23.08 303-AH 20.00 328-AH 42.86 254-AH 63.77 279-AH 10.00 304-AH 27.59 329-AH 80.00 255-AH 0.00 280-AH 21.43 305-AH 26.32 330-AH 100.00 256-AH 71.43 281-AH 0.00 306-AH 0.00 331-AH 30.00 257-AH 20.59 282-AH 12.50 307-AH 71.43 332-AH 0.00 258-AH 0.00 283-AH 21.05 308-AH 85.71 333-AH 77.17 259-AH 40.00 284-AH 51.35 309-AH 28.57 334-AH 66.67 260-AH 40.62 285-AH 62.50 310-AH 19.23 335-AH 50.00 261-AH 33.33 286-AH 50.00 311-AH 34.51 336-AH 100.00 262-AH 0.00 287-AH 53.23 312-AH 33.33 337-AH 74.07 263-AH 36.36 288-AH 25.00 313-AH 24.64 338-AH 46.15 264-AH 59.46 289-AH 64.29 314-AH 15.38 339-AH 65.00 265-AH 21.43 290-AH 19.44 315-AH 16.67 340-AH 50.00 266-AH 33.33 291-AH 17.39 316-AH 42.51 341-AH 100.00 267-AH 41.67 292-AH 0.00 317-AH 40.00 342-AH 50.00 268-AH 66.67 293-AH 75.44 318-AH 0.00 343-AH 28.12 269-AH 24.00 294-AH 0.00 319-AH 50.00 344-AH 44.44 270-AH 55.00 295-AH 55.56 320-AH 0.00 345-AH 41.67 271-AH 30.77 296-AH 33.33 321-AH 0.00 346-AH 0.00 272-AH 0.00 297-AH 56.76 322-AH 14.29 347-AH 20.00 273-AH 54.10 298-AH 39.44 323-AH 30.00 348-AH 63.04 274-AH 60.00 299-AH 9.09 324-AH 10.00 349-AH 55.56 275-AH 42.50 300-AH 23.08 325-AH 0.00 350-AH 50.00

Ad−Hoc Robust Bilingual Test Task, French target col ection(s) − Box plot of the Topics of the Experiment 0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%

R−Precision

Ad−Hoc Robust Bilingual Test Task, French target col ection(s) − Distribution of the Topics of the Experiment 00% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%

R−Precision

Ad−Hoc Robust Bilingual Test Task, French target colection(s) − Comparison to Median R−Precision by Topic (Topics 251−AH to 275−AH) 1 0.5 ce reen 0 if D−0.5 0.5 ce reen 0 if D−0.5 1 1 0.5 REINAE2FTDNT REINAE2FTDNT REINAE2FTDNT reina 10.2415/AH-ROBUST-BILI-X2FR-TEST-CLEF2007.REINA.REINAE2FTDT Mergin MT of T+D EN topics, no expansion Ad−Hoc Robust Bilingual Test Task, French target collection(s) − Standard Recall Levels vs Mean Interpolated Precision 100% 0% 0% 10% 20% 30% 40% 60% 70% 80% 90% 100% Precision averages (%) for individual queries 251-AH 14.26 276-AH 15.88 301-AH 52.87 326-AH 0.00 252-AH 9.27 277-AH 51.39 302-AH 18.05 327-AH 0.00 253-AH 22.22 278-AH 49.57 303-AH 22.18 328-AH 47.74 254-AH 67.42 279-AH 8.96 304-AH 23.33 329-AH 79.44 255-AH 0.00 280-AH 23.77 305-AH 16.09 330-AH 100.00 256-AH 73.52 281-AH 0.59 306-AH 11.11 331-AH 13.09 257-AH 8.83 282-AH 4.70 307-AH 42.46 332-AH 0.00 258-AH 0.00 283-AH 47.52 308-AH 75.30 333-AH 86.51 259-AH 54.08 284-AH 64.95 309-AH 24.69 334-AH 74.40 260-AH 39.08 285-AH 63.58 310-AH 2.62 335-AH 52.25 261-AH 0.00 286-AH 24.36 311-AH 16.40 336-AH 5.56 262-AH 5.57 287-AH 57.65 312-AH 37.38 337-AH 74.28 263-AH 42.60 288-AH 23.01 313-AH 16.67 338-AH 30.25 264-AH 73.02 289-AH 84.42 314-AH 16.07 339-AH 72.09 265-AH 19.83 290-AH 12.04 315-AH 5.60 340-AH 51.15 266-AH 15.53 291-AH 9.80 316-AH 43.11 341-AH 100.00 267-AH 30.79 292-AH 1.61 317-AH 14.36 342-AH 58.34 268-AH 72.55 293-AH 70.53 318-AH 33.33 343-AH 26.42 269-AH 10.77 294-AH 0.00 319-AH 43.15 344-AH 49.42 270-AH 68.60 295-AH 55.49 320-AH 1.33 345-AH 40.26 271-AH 61.08 296-AH 22.98 321-AH 0.00 346-AH 0.00 272-AH 9.06 297-AH 56.61 322-AH 10.82 347-AH 30.59 273-AH 50.10 298-AH 36.04 323-AH 20.15 348-AH 67.72 274-AH 76.24 299-AH 9.28 324-AH 2.85 349-AH 74.14 275-AH 8.96 300-AH 19.98 325-AH 1.85 350-AH 50.96

Ad−Hoc Robust Bilingual Test Task, French target col ection(s) − Box plot of the Topics of the Experiment 0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%

Average Precision

Ad−Hoc Robust Bilingual Test Task, French target col ection(s) − Distribution of the Topics of the Experiment 00% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%

Average Precision

Ad−Hoc Robust Bilingual Test Task, French target colection(s) − Comparison to Median Average Precision by Topic (Topics 251−AH to 275−AH) 1 0.5 ce reen 0 if D−0.5 0.5 ce reen 0 if D−0.5 1 1 0.5 ce reen 0 if D−0.5 1 0.5 ce reen 0 if D−0.5 20 REINAE2FTDT REINAE2FTDT REINAE2FTDT reina 10.2415/AH-ROBUST-BILI-X2FR-TEST-CLEF2007.REINA.REINAE2FTDT AH-ROBUST-BILI-X2FR-TEST-CLEF2007

Ad−Hoc Robust Bilingual Test Task, French target collection(s) − Retrieved documents vs Mean Precision 100% 90% 80% 70% 60% 40% 30% 20% 10% 0%5

30 100 200 Retrieved Documents (logarithmic scale) Docs Cutoff Levels Precision at DCL (%) 5 docs 43.40 10 docs 37.70 15 docs 33.47 20 docs 29.95 30 docs 25.60 100 docs 14.12 200 docs 8.79 500 docs 4.11 1000 docs 2.24 R-Precision (precision after R document retrieved, where R = Relevant retrieved)

32.87 Precision averages (%) for individual queries 251-AH 25.00 276-AH 17.65 301-AH 44.00 326-AH 0.00 252-AH 16.67 277-AH 62.50 302-AH 25.00 327-AH 0.00 253-AH 23.17 278-AH 46.15 303-AH 32.00 328-AH 42.86 254-AH 65.22 279-AH 10.00 304-AH 27.59 329-AH 60.00 255-AH 0.00 280-AH 28.57 305-AH 15.79 330-AH 100.00 256-AH 71.43 281-AH 0.00 306-AH 0.00 331-AH 0.00 257-AH 17.65 282-AH 12.50 307-AH 42.86 332-AH 0.00 258-AH 0.00 283-AH 36.84 308-AH 85.71 333-AH 81.52 259-AH 40.00 284-AH 54.05 309-AH 28.57 334-AH 66.67 260-AH 43.75 285-AH 55.00 310-AH 7.69 335-AH 45.00 261-AH 0.00 286-AH 25.00 311-AH 22.12 336-AH 0.00 262-AH 0.00 287-AH 56.45 312-AH 33.33 337-AH 67.41 263-AH 36.36 288-AH 25.00 313-AH 20.29 338-AH 46.15 264-AH 67.57 289-AH 78.57 314-AH 7.69 339-AH 62.50 265-AH 21.43 290-AH 13.89 315-AH 12.50 340-AH 57.14 266-AH 13.33 291-AH 13.04 316-AH 45.41 341-AH 100.00 267-AH 41.67 292-AH 0.00 317-AH 20.00 342-AH 60.00 268-AH 66.67 293-AH 75.44 318-AH 0.00 343-AH 28.12 269-AH 14.00 294-AH 0.00 319-AH 37.50 344-AH 33.33 270-AH 55.00 295-AH 51.11 320-AH 0.00 345-AH 33.33 271-AH 61.54 296-AH 16.67 321-AH 0.00 346-AH 0.00 272-AH 0.00 297-AH 56.76 322-AH 14.29 347-AH 40.00 273-AH 50.82 298-AH 39.44 323-AH 25.00 348-AH 65.22 274-AH 70.00 299-AH 15.15 324-AH 10.00 349-AH 61.11 275-AH 10.00 300-AH 15.38 325-AH 0.00 350-AH 62.50

Ad−Hoc Robust Bilingual Test Task, French target col ection(s) − Box plot of the Topics of the Experiment 0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%

R−Precision

Ad−Hoc Robust Bilingual Test Task, French target col ection(s) − Distribution of the Topics of the Experiment 00% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%

R−Precision

Priority Query Construction Source Language Topic Fields Pooled 3 AUTOMATIC English

title false Mergin MT of EN-title topics, with expansion using windows of terms Ad−Hoc Robust Bilingual Test Task, French target collection(s) − Standard Recall Levels vs Mean Interpolated Precision 100% 90% 80% 70% 60% 40% 30% 20% 10% 0% 0% 10% 20% 30% 40% 60% 70% 80% 90% 100% Precision averages (%) for individual queries 251-AH 7.30 276-AH 13.06 301-AH 57.35 326-AH 0.00 252-AH 32.39 277-AH 42.51 302-AH 18.81 327-AH 0.00 253-AH 46.95 278-AH 38.32 303-AH 20.37 328-AH 57.47 254-AH 70.30 279-AH 9.18 304-AH 22.05 329-AH 61.37 255-AH 0.00 280-AH 23.89 305-AH 36.28 330-AH 100.00 256-AH 23.72 281-AH 0.23 306-AH 9.84 331-AH 10.08 257-AH 13.89 282-AH 0.23 307-AH 40.19 332-AH 0.00 258-AH 0.00 283-AH 65.39 308-AH 51.69 333-AH 91.60 259-AH 38.26 284-AH 59.79 309-AH 7.02 334-AH 63.57 260-AH 24.63 285-AH 67.09 310-AH 6.86 335-AH 52.43 261-AH 0.00 286-AH 60.99 311-AH 12.87 336-AH 25.00 262-AH 0.14 287-AH 58.72 312-AH 39.90 337-AH 84.87 263-AH 1.33 288-AH 12.49 313-AH 23.54 338-AH 32.02 264-AH 73.02 289-AH 89.76 314-AH 14.71 339-AH 71.57 265-AH 21.02 290-AH 20.44 315-AH 11.35 340-AH 56.53 266-AH 20.83 291-AH 4.89 316-AH 1.84 341-AH 71.87 267-AH 11.79 292-AH 1.68 317-AH 9.50 342-AH 28.46 268-AH 70.83 293-AH 72.03 318-AH 100.00 343-AH 25.43 269-AH 14.74 294-AH 0.00 319-AH 31.03 344-AH 47.83 270-AH 74.13 295-AH 65.35 320-AH 9.74 345-AH 52.95 271-AH 6.01 296-AH 27.66 321-AH 0.00 346-AH 0.00 272-AH 18.06 297-AH 61.18 322-AH 7.39 347-AH 43.30 273-AH 40.62 298-AH 24.24 323-AH 13.56 348-AH 64.49 274-AH 70.29 299-AH 1.04 324-AH 0.00 349-AH 73.35 275-AH 4.40 300-AH 15.99 325-AH 6.06 350-AH 58.48

Ad−Hoc Robust Bilingual Test Task, French target col ection(s) − Box plot of the Topics of the Experiment 0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%

Average Precision

Ad−Hoc Robust Bilingual Test Task, French target col ection(s) − Distribution of the Topics of the Experiment 00% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%

Average Precision

Ad−Hoc Robust Bilingual Test Task, French target colection(s) − Comparison to Median Average Precision by Topic (Topics 251−AH to 275−AH) 1 0.5 ce reen 0 if D−0.5 0.5 ce reen 0 if D−0.5 1 1 0.5 ce reen 0 if D−0.5 1 0.5 ce reen 0 if D−0.5 20 REINAE2FTET REINAE2FTET REINAE2FTET reina 10.2415/AH-ROBUST-BILI-X2FR-TEST-CLEF2007.REINA.REINAE2FTET AH-ROBUST-BILI-X2FR-TEST-CLEF2007

Ad−Hoc Robust Bilingual Test Task, French target collection(s) − Retrieved documents vs Mean Precision 100% 90% 80% 70% 60% 40% 30% 20% 10% 0%5

30 100 200 Retrieved Documents (logarithmic scale) Docs Cutoff Levels Precision at DCL (%) 5 docs 40.40 10 docs 35.00 15 docs 30.73 20 docs 28.20 30 docs 24.43 100 docs 13.70 200 docs 8.28 500 docs 3.77 1000 docs 2.03 R-Precision (precision after R document retrieved, where R = Relevant retrieved)

31.33 Precision averages (%) for individual queries 251-AH 15.00 276-AH 17.65 301-AH 48.00 326-AH 0.00 252-AH 33.33 277-AH 25.00 302-AH 25.00 327-AH 0.00 253-AH 50.00 278-AH 46.15 303-AH 32.00 328-AH 57.14 254-AH 66.67 279-AH 0.00 304-AH 24.14 329-AH 60.00 255-AH 0.00 280-AH 32.14 305-AH 36.84 330-AH 100.00 256-AH 14.29 281-AH 0.00 306-AH 0.00 331-AH 0.00 257-AH 20.59 282-AH 0.00 307-AH 42.86 332-AH 0.00 258-AH 0.00 283-AH 63.16 308-AH 42.86 333-AH 85.87 259-AH 40.00 284-AH 56.76 309-AH 0.00 334-AH 66.67 260-AH 31.25 285-AH 60.00 310-AH 15.38 335-AH 55.00 261-AH 0.00 286-AH 75.00 311-AH 16.81 336-AH 0.00 262-AH 0.00 287-AH 54.84 312-AH 33.33 337-AH 78.52 263-AH 0.00 288-AH 25.00 313-AH 33.33 338-AH 38.46 264-AH 68.92 289-AH 78.57 314-AH 0.00 339-AH 62.50 265-AH 28.57 290-AH 16.67 315-AH 12.50 340-AH 50.00 266-AH 20.00 291-AH 8.70 316-AH 7.73 341-AH 80.00 267-AH 16.67 292-AH 0.00 317-AH 20.00 342-AH 20.00 268-AH 66.67 293-AH 66.67 318-AH 100.00 343-AH 28.12 269-AH 20.00 294-AH 0.00 319-AH 31.25 344-AH 33.33 270-AH 65.00 295-AH 55.56 320-AH 14.29 345-AH 37.50 271-AH 0.00 296-AH 33.33 321-AH 0.00 346-AH 0.00 272-AH 16.67 297-AH 59.46 322-AH 0.00 347-AH 40.00 273-AH 37.70 298-AH 26.76 323-AH 17.50 348-AH 65.22 274-AH 70.00 299-AH 0.00 324-AH 0.00 349-AH 55.56 275-AH 10.00 300-AH 15.38 325-AH 20.00 350-AH 37.50

Ad−Hoc Robust Bilingual Test Task, French target col ection(s) − Box plot of the Topics of the Experiment 0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%

R−Precision

Ad−Hoc Robust Bilingual Test Task, French target col ection(s) − Distribution of the Topics of the Experiment 00% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%

R−Precision

Priority Query Construction Source Language Topic Fields Pooled 4 AUTOMATIC English title false Mergin MT of EN-title topics, no expansion

Ad−Hoc Robust Bilingual Test Task, French target collection(s) − Standard Recall Levels vs Mean Interpolated Precision 100% 90% 80% 70% 60% 40% 30% 20% 10% 0% 0% 10% 20% 30% 40% 60% 70% 80% 90% 100% Precision averages (%) for individual queries 251-AH 9.10 276-AH 13.36 301-AH 48.33 326-AH 0.00 252-AH 23.61 277-AH 57.09 302-AH 22.79 327-AH 0.00 253-AH 40.93 278-AH 46.31 303-AH 15.63 328-AH 39.03 254-AH 63.53 279-AH 9.10 304-AH 19.41 329-AH 61.89 255-AH 0.00 280-AH 24.65 305-AH 32.22 330-AH 100.00 256-AH 18.90 281-AH 0.25 306-AH 16.20 331-AH 12.57 257-AH 9.85 282-AH 0.30 307-AH 57.33 332-AH 0.00 258-AH 0.00 283-AH 56.95 308-AH 42.69 333-AH 88.46 259-AH 35.67 284-AH 57.35 309-AH 6.56 334-AH 61.73 260-AH 31.38 285-AH 63.47 310-AH 4.51 335-AH 50.07 261-AH 0.00 286-AH 44.65 311-AH 12.61 336-AH 5.88 262-AH 0.15 287-AH 53.43 312-AH 42.97 337-AH 77.81 263-AH 2.16 288-AH 11.70 313-AH 36.02 338-AH 21.54 264-AH 54.36 289-AH 85.11 314-AH 13.52 339-AH 69.96 265-AH 18.19 290-AH 36.20 315-AH 9.49 340-AH 47.69 266-AH 12.04 291-AH 5.21 316-AH 4.22 341-AH 68.57 267-AH 13.39 292-AH 2.85 317-AH 5.09 342-AH 25.32 268-AH 72.92 293-AH 72.65 318-AH 50.00 343-AH 19.42 269-AH 10.74 294-AH 0.00 319-AH 32.62 344-AH 51.21 270-AH 71.12 295-AH 61.11 320-AH 8.32 345-AH 43.72 271-AH 10.60 296-AH 35.92 321-AH 0.00 346-AH 0.00 272-AH 17.57 297-AH 56.40 322-AH 5.62 347-AH 31.25 273-AH 25.54 298-AH 24.79 323-AH 9.98 348-AH 61.39 274-AH 66.73 299-AH 1.02 324-AH 0.00 349-AH 69.64 275-AH 4.68 300-AH 17.25 325-AH 1.47 350-AH 49.71

Ad−Hoc Robust Bilingual Test Task, French target col ection(s) − Box plot of the Topics of the Experiment 0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%

Average Precision

Ad−Hoc Robust Bilingual Test Task, French target col ection(s) − Distribution of the Topics of the Experiment 00% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%

Average Precision

Ad−Hoc Robust Bilingual Test Task, French target colection(s) − Comparison to Median Average Precision by Topic (Topics 251−AH to 275−AH) 1 0.5 ce reen 0 if D−0.5 0.5 ce reen 0 if D−0.5 1 1 0.5 ce reen 0 if D−0.5 1 0.5 ce reen 0 if D−0.5 20 REINAE2FTT REINAE2FTT REINAE2FTT reina 10.2415/AH-ROBUST-BILI-X2FR-TEST-CLEF2007.REINA.REINAE2FTT AH-ROBUST-BILI-X2FR-TEST-CLEF2007

Ad−Hoc Robust Bilingual Test Task, French target collection(s) − Retrieved documents vs Mean Precision 100% 90% 80% 70% 60% 40% 30% 20% 10% 0%5

30 100 200 Retrieved Documents (logarithmic scale) Docs Cutoff Levels Precision at DCL (%) 5 docs 39.00 10 docs 32.10 15 docs 29.07 20 docs 26.95 30 docs 23.73 100 docs 13.25 200 docs 8.03 500 docs 3.74 1000 docs 2.07 R-Precision (precision after R document retrieved, where R = Relevant retrieved)

29.00 Precision averages (%) for individual queries 251-AH 15.00 276-AH 5.88 301-AH 40.00 326-AH 0.00 252-AH 33.33 277-AH 50.00 302-AH 25.00 327-AH 0.00 253-AH 41.46 278-AH 46.15 303-AH 32.00 328-AH 57.14 254-AH 62.32 279-AH 5.00 304-AH 20.69 329-AH 60.00 255-AH 0.00 280-AH 28.57 305-AH 42.11 330-AH 100.00 256-AH 14.29 281-AH 0.00 306-AH 25.00 331-AH 0.00 257-AH 11.76 282-AH 0.00 307-AH 57.14 332-AH 0.00 258-AH 0.00 283-AH 47.37 308-AH 28.57 333-AH 82.61 259-AH 40.00 284-AH 51.35 309-AH 14.29 334-AH 66.67 260-AH 34.38 285-AH 60.00 310-AH 11.54 335-AH 50.00 261-AH 0.00 286-AH 25.00 311-AH 16.81 336-AH 0.00 262-AH 0.00 287-AH 54.84 312-AH 50.00 337-AH 66.67 263-AH 0.00 288-AH 18.75 313-AH 44.93 338-AH 38.46 264-AH 52.70 289-AH 78.57 314-AH 0.00 339-AH 60.00 265-AH 14.29 290-AH 44.44 315-AH 8.33 340-AH 50.00 266-AH 13.33 291-AH 17.39 316-AH 8.70 341-AH 80.00 267-AH 8.33 292-AH 0.00 317-AH 0.00 342-AH 20.00 268-AH 66.67 293-AH 77.19 318-AH 0.00 343-AH 21.88 269-AH 14.00 294-AH 0.00 319-AH 31.25 344-AH 33.33 270-AH 60.00 295-AH 53.33 320-AH 14.29 345-AH 37.50 271-AH 7.69 296-AH 33.33 321-AH 0.00 346-AH 0.00 272-AH 16.67 297-AH 59.46 322-AH 0.00 347-AH 30.00 273-AH 24.59 298-AH 28.17 323-AH 17.50 348-AH 60.87 274-AH 60.00 299-AH 0.00 324-AH 0.00 349-AH 55.56 275-AH 12.50 300-AH 15.38 325-AH 0.00 350-AH 37.50

Ad−Hoc Robust Bilingual Test Task, French target col ection(s) − Box plot of the Topics of the Experiment 0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%

R−Precision

Ad−Hoc Robust Bilingual Test Task, French target col ection(s) − Distribution of the Topics of the Experiment 00% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%

R−Precision

Overall statistics for 100 queries :

Total number of documents over all queries

Retrieved Relevant Relevant retrieved

49,265 2,415 1,890 PL2 & LM Ad−Hoc Robust Bilingual Test Task, French target collection(s) − Standard Recall Levels vs Mean Interpolated Precision 100% 90% 80% 70% 60% 40% 30% 20% 10% 0% 0% 50%

Recall 10% 20% 30% 40% 60% 70% 80% 90% 100% Precision averages (%) for individual queries 251-AH 8.33 276-AH 17.35 301-AH 68.38 326-AH 0.00 252-AH 8.17 277-AH 56.58 302-AH 15.14 327-AH 0.00 253-AH 0.00 278-AH 55.77 303-AH 10.65 328-AH 68.07 254-AH 51.40 279-AH 9.15 304-AH 0.00 329-AH 40.24 255-AH 0.00 280-AH 26.62 305-AH 40.61 330-AH 100.00 256-AH 68.56 281-AH 0.23 306-AH 23.69 331-AH 10.61 257-AH 8.48 282-AH 0.00 307-AH 58.63 332-AH 0.00 258-AH 0.00 283-AH 35.52 308-AH 74.65 333-AH 80.38 259-AH 45.49 284-AH 63.35 309-AH 2.67 334-AH 43.39 260-AH 46.00 285-AH 71.23 310-AH 0.00 335-AH 53.85 261-AH 0.00 286-AH 28.17 311-AH 25.16 336-AH 2.44 262-AH 0.28 287-AH 48.39 312-AH 23.26 337-AH 79.67 263-AH 2.67 288-AH 14.05 313-AH 30.35 338-AH 30.83 264-AH 76.56 289-AH 83.57 314-AH 11.08 339-AH 63.85 265-AH 17.69 290-AH 75.56 315-AH 0.23 340-AH 66.60 266-AH 0.16 291-AH 8.15 316-AH 0.00 341-AH 100.00 267-AH 49.08 292-AH 1.07 317-AH 10.20 342-AH 53.27 268-AH 48.79 293-AH 76.70 318-AH 0.61 343-AH 22.49 269-AH 24.54 294-AH 4.33 319-AH 72.89 344-AH 42.35 270-AH 72.24 295-AH 57.57 320-AH 32.63 345-AH 88.29 271-AH 8.53 296-AH 24.97 321-AH 0.00 346-AH 0.00 272-AH 6.82 297-AH 52.26 322-AH 3.15 347-AH 60.54 273-AH 49.27 298-AH 56.02 323-AH 7.82 348-AH 75.96 274-AH 82.98 299-AH 1.88 324-AH 0.00 349-AH 58.93 275-AH 68.37 300-AH 13.30 325-AH 0.06 350-AH 69.83 UNINEBILFR1 UNINEBILFR1 UNINEBILFR1 UNINEBILFR1

Ad−Hoc Robust Bilingual Test Task, French target col ection(s) − Box plot of the Topics of the Experiment 0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%

Average Precision

Ad−Hoc Robust Bilingual Test Task, French target col ection(s) − Distribution of the Topics of the Experiment 00% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%

Average Precision

Ad−Hoc Robust Bilingual Test Task, French target colection(s) − Comparison to Median Average Precision by Topic (Topics 251−AH to 275−AH) 1 0.5 ce reen 0 if D−0.5 0.5 ce reen 0 if D−0.5 1 1 0.5 ce reen 0 if D−0.5 1 0.5 ce reen 0 if D−0.5 30 t n iem25 r e p xE20 e h ftso15 cp i o fT10 o r e b m 5 u

Ad−Hoc Robust Bilingual Test Task, French target collection(s) − Retrieved documents vs Mean Precision 100%

Interquartile range Mean Mean With No Outliers

Std With No Outliers

Ad−Hoc Robust Bilingual Test Task, French target col ection(s) − Distribution of the Topics of the Experiment −1 276−AH 277−AH 278−AH 279−AH 280−AH 281−AH 282−AH 283−AH 284−AH 285−AH 286−AH 287−AH 288−AH 289−AH 290−AH 291−AH 292−AH 293−AH 294−AH 295−AH 296−AH 297−AH 298−AH 299−AH 300−AH

Topic Identifier

Ad−Hoc Robust Bilingual Test Task, French target col ection(s) − Comparison to Median R−Precision by Topic (Topics 301−AH to 325−AH) −1 301−AH 302−AH 303−AH 304−AH 305−AH 306−AH 307−AH 308−AH 309−AH 310−AH 311−AH 312−AH 313−AH 314−AH 315−AH 316−AH 317−AH 318−AH 319−AH 320−AH 321−AH 322−AH 323−AH 324−AH 325−AH

Topic Identifier

Ad−Hoc Robust Bilingual Test Task, French target col ection(s) − Comparison to Median R−Precision by Topic (Topics 326−AH to 350−AH) UNINEBILFR1 UNINEBILFR1