Appendix A Results of the Core Tracks Ad-hoc Bilingual and Monolingual Tasks

-

Introduction 3

Results for CLEF 2007 Ad-hoc Track 3. Individual Experiment Results and Graphs This section provides the individual results for each official experiment. For each experiment the following tables and graphs are shown: - Overall statistics and information - Interpolated recall vs precision averages plot - Average precision statistics and box plot - Average precision comparison to median plot - Document cutoff levels vs precision at DCL plot - R-Precision statistics and box plot - R-Precision comparison to median plot Topics are identified with DOIs, as well as the experiments. The prefix for the DOI of a topic is 10.2452. The following example shows how to build the DOI for a topic given its number: for topic 200-AH, the corresponding DOI is 10.2452/200-AH List of Submitted Experiments 7 bg bg bg bg bg bg bg bg bg bg bg bg bg bg bg bg cs cs cs cs cs cs cs cs cs hu am cs cs cs cs cs cs cs cs cs cs cs cs cs

TD TDN TD TD TD TD TD T TD TD TD TD TD TDN TD TDN TD TD TDN TDN TD TD TD TD TDN TDN TD TD TDN T TD TD TDN TD TDN TDN TD TD TDN

MANUAL

MANUAL

yes yes unine unine unine

AH-MONO-CS-CLEF2007

cs cs cs hu hu hu hu hu hu hu hu hu hu hu hu hu hu hu hu hu hu hu en en en en en en hu en en en en en en en en en en en

TD TD TDN TD TDN TD TD TD TDN TDN TD TD TD TDN T TD TD TD TD TD TDN TDN T TD T T T T T T D TD T TD TDN TD TDN TDN TDN TD TD TDN T TD TDN TDN TDN TD TD T TD TD TD TD TD TD TD TD T T T T TD TD TD TD TD TD TD TD TD TD TD TD TD TD TD TD TD TDN United States 10.2415/AH-BILI-X2EN-CLEF2007.JHU

APL.APLBIIDENTD4

id id id id id id id hi om om om om te bn hi te id id id id bn bn bn hi hi hi hi hi hi hi zh zh zh zh zh zh zh zh zh zh

TD TD TD TD TD T T TD TDN T TD TDN TD TDN TDN TD TD TD TD TD T TD TDN T TD TDN TDN TDN TD TD T TD T TD T TD T TD T TD Track Overview Results and Graphs 14

Track

Ad−Hoc Monolingual Bulgarian Task Top 5 Participants − Standard Recall Levels vs Mean Interpolated Precision 0%

0% 1 0.8 0.6 0.4 0.2 −0.4 −0.6 −0.8 −1 1 0.8 0.6 0.4 0.2 −0.4 −0.6 −0.8 −1 50% Recall 10% 20% 30% 40% 60% 70% 80% 90% 100% Ad−Hoc Monolingual Bulgarian Task Top 5 Participants − Comparison to Median Average Precision by Topic (Topics 401−AH to 425−AH) unine [Experiment UNINEBG4; MAP 44.22%; Pooled] jhu−apl [Experiment APLMOBGTD4; MAP 36.57%; Not Pooled] opentext [Experiment OTBG07TDE; MAP 35.02%; Not Pooled] alicante [Experiment IRNBUEXP2N; MAP 29.81%; Pooled] daedalus [Experiment BGFSBG2S; MAP 27.19%; Pooled] 401−AH 402−AH 403−AH 404−AH 405−AH 406−AH 407−AH 408−AH 409−AH 410−AH 411−AH 412−AH 413−AH 414−AH 415−AH 416−AH 417−AH 418−AH 419−AH 420−AH 421−AH 422−AH 423−AH 424−AH 425−AH 426−AH 427−AH 428−AH 429−AH 430−AH 431−AH 432−AH 433−AH 434−AH 435−AH 436−AH 437−AH 438−AH 439−AH 440−AH 441−AH 442−AH 443−AH 444−AH 445−AH 446−AH 447−AH 448−AH 449−AH 450−AH Ad−Hoc Monolingual Bulgarian Task − Box Plot of the Topics jhu−apl [Experiment APLMOBGTD4; MAP 36.57%; Not Pooled] opentext [Experiment OTBG07TDE; MAP 35.02%; Not Pooled] jhu−apl [Experiment APLMOBGTDN4; MAP 34.93%; Pooled] s t n e m i r e p x E jhu−apl [Experiment APLMOBGTD5; MAP 34.01%; Pooled]

opentext [Experiment OTBG07TD; MAP 33.13%; Pooled] alicante [Experiment IRNBUEXP2N; MAP 29.81%; Pooled]

opentext [Experiment OTBG07T; MAP 29.48%; Pooled] daedalus [Experiment BGFSBG2S; MAP 27.19%; Pooled] alicante [Experiment IRNBUEXP3; MAP 26.35%; Pooled] alicante [Experiment IRNBUEXP2; MAP 26.05%; Not Pooled] opentext [Experiment OTBG07TDNZ; MAP 25.27%; Pooled] alicante [Experiment IRNBUNEXP; MAP 21.19%; Not Pooled] 0% 10% 20% 30% 70% 80% 90% 100% 40% 50% 60%

Average Precision 10.2455/TUKEY_T_TEST.9633F4C12FC97855A347B6AF7DD9B7A5 Ad−Hoc Monolingual Bulgarian Task − Tukey T test with "top group" highlighted UNINEBG4 UNINEBG1 UNINEBG2

UNINEBG3 APLMOBGTD4

OTBG07TDE APLMOBGTDN4 s t n e irm APLMOBGTD5 e p x E

OTBG07TD

OTBG07T IRNBUEXP2N

BGFSBG2S OTBG07TDNZ

IRNBUEXP3 IRNBUEXP2 IRNBUNEXP 0.35

Track

0.8 0.6 0.4 0.2 −0.4 −0.6 −0.8 −1 1 0.8 0.6 0.4 0.2 −0.4 −0.6 −0.8 −1 401−AH 402−AH 403−AH 404−AH 405−AH 406−AH 407−AH 408−AH 409−AH 410−AH 411−AH 412−AH 413−AH 414−AH 415−AH 416−AH 417−AH 418−AH 419−AH 420−AH 421−AH 422−AH 423−AH 424−AH 425−AH 426−AH 427−AH 428−AH 429−AH 430−AH 431−AH 432−AH 433−AH 434−AH 435−AH 436−AH 437−AH 438−AH 439−AH 440−AH 441−AH 442−AH 443−AH 444−AH 445−AH 446−AH 447−AH 448−AH 449−AH 450−AH unine [Experiment UNINEBG4; R−Prec 40.54%; Pooled] jhu−apl [Experiment APLMOBGTD4; R−Prec 35.74%; Not Pooled] opentext [Experiment OTBG07TDE; R−Prec 34.02%; Not Pooled] alicante [Experiment IRNBUEXP2N; R−Prec 29.74%; Pooled] daedalus [Experiment BGFSBG2S; R−Prec 27.84%; Pooled]

Ad−Hoc Bilingual English Task − Distribution of the Topics of the Experiment 00% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%

Average Precision 1 0.8 0.6 0.4 ce 0.2 reen 0 ifD−0.2 −0.4 −0.6 −0.8 1 0.8 0.6 0.4 ce 0.2 reen 0 ifD−0.2 −0.4 −0.6 −0.8

Ad−Hoc Bilingual English Task − Distribution of the Topics of the Experiment 00% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%

R−Precision Precision averages (%) for individual queries

Ad−HocBilingualEnglishTask−ComparisontoMedianAveragePrecisionbyTopic(Topics401−AHto425−AH) 1 0.8 0.6 0.4 ce 0.2 reen 0 ifD−0.2 −0.4 −0.6 −0.8 1 0.8 0.6 0.4 ce 0.2 reen 0 ifD−0.2 −0.4 −0.6 −0.8 40

Ad−Hoc Bilingual English Task − Distribution of the Topics of the Experiment 00% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100% Average Precision

DSV_AMH_BLNG_RUN1 DSV_AMH_BLNG_RUN1 0%5 Ad−Hoc Bilingual English Task − Box plot of the Topics of the Experiment Precision averages (%) for individual queries

Ad−HocBilingualEnglishTask−ComparisontoMedianR−PrecisionbyTopic(Topics401−AHto425−AH) 1 0.8 0.6 0.4 ce 0.2 reen 0 ifD−0.2 −0.4 −0.6 −0.8 1 0.8 0.6 0.4 ce 0.2 reen 0 ifD−0.2 −0.4 −0.6 −0.8 40

Ad−Hoc Bilingual English Task − Distribution of the Topics of the Experiment 00% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100% R−Precision

DSV_AMH_BLNG_RUN1 DSV_AMH_BLNG_RUN1 0.0058 Maximal expansion of terms with pseudo relevance 0.1138 feedback 0%0% Ad−Hoc Bilingual English Task − Box plot of the Topics of the Experiment Precision averages (%) for individual queries

Ad−HocBilingualEnglishTask−ComparisontoMedianAveragePrecisionbyTopic(Topics401−AHto425−AH) 1 0.8 0.6 0.4 ce 0.2 reen 0 ifD−0.2 −0.4 −0.6 −0.8 1 0.8 0.6 0.4 ce 0.2 reen 0 ifD−0.2 −0.4 −0.6 −0.8 40

Ad−Hoc Bilingual English Task − Distribution of the Topics of the Experiment 00% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100% Average Precision

DSV_AMH_BLNG_RUN2 DSV_AMH_BLNG_RUN2 0%5 Ad−Hoc Bilingual English Task − Box plot of the Topics of the Experiment

R_PRECISION

Precision averages (%) for individual queries

Ad−HocBilingualEnglishTask−ComparisontoMedianR−PrecisionbyTopic(Topics401−AHto425−AH) 1 0.8 0.6 0.4 ce 0.2 reen 0 ifD−0.2 −0.4 −0.6 −0.8 1 0.8 0.6 0.4 ce 0.2 reen 0 ifD−0.2 −0.4 −0.6 −0.8 40

Ad−Hoc Bilingual English Task − Distribution of the Topics of the Experiment 00% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100% R−Precision

DSV_AMH_BLNG_RUN2 DSV_AMH_BLNG_RUN2 0%0% Ad−Hoc Bilingual English Task − Box plot of the Topics of the Experiment Precision averages (%) for individual queries

Ad−HocBilingualEnglishTask−ComparisontoMedianAveragePrecisionbyTopic(Topics401−AHto425−AH) 1 0.8 0.6 0.4 ce 0.2 reen 0 ifD−0.2 −0.4 −0.6 −0.8 1 0.8 0.6 0.4 ce 0.2 reen 0 ifD−0.2 −0.4 −0.6 −0.8 40

Ad−Hoc Bilingual English Task − Distribution of the Topics of the Experiment 00% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100% Average Precision

DSV_AMH_BLNG_RUN3 DSV_AMH_BLNG_RUN3 0%5

R_PRECISION

Precision averages (%) for individual queries

Ad−HocBilingualEnglishTask−ComparisontoMedianR−PrecisionbyTopic(Topics401−AHto425−AH) 1 0.8 0.6 0.4 ce 0.2 reen 0 ifD−0.2 −0.4 −0.6 −0.8 1 0.8 0.6 0.4 ce 0.2 reen 0 ifD−0.2 −0.4 −0.6 −0.8 40

Ad−Hoc Bilingual English Task − Distribution of the Topics of the Experiment 00% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100% R−Precision

DSV_AMH_BLNG_RUN3 DSV_AMH_BLNG_RUN3

Priority

Query Construction

50,000 Source Language

2,247 Topic Fields 943 Pooled 4

AUTOMATIC

Amharic

title, description false 0.0052 Maximal expansion of terms with pseudo relevance 0.1240 feedback 0%0% Ad−Hoc Bilingual English Task − Box plot of the Topics of the Experiment Precision averages (%) for individual queries

Ad−HocBilingualEnglishTask−ComparisontoMedianAveragePrecisionbyTopic(Topics401−AHto425−AH) 1 0.8 0.6 0.4 ce 0.2 reen 0 ifD−0.2 −0.4 −0.6 −0.8 1 0.8 0.6 0.4 ce 0.2 reen 0 ifD−0.2 −0.4 −0.6 −0.8 40

Ad−Hoc Bilingual English Task − Distribution of the Topics of the Experiment 00% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100% Average Precision

DSV_AMH_BLNG_RUN4 DSV_AMH_BLNG_RUN4 0%5 Ad−Hoc Bilingual English Task − Box plot of the Topics of the Experiment

R_PRECISION

Precision averages (%) for individual queries

Ad−HocBilingualEnglishTask−ComparisontoMedianR−PrecisionbyTopic(Topics401−AHto425−AH) 1 0.8 0.6 0.4 ce 0.2 reen 0 ifD−0.2 −0.4 −0.6 −0.8 1 0.8 0.6 0.4 ce 0.2 reen 0 ifD−0.2 −0.4 −0.6 −0.8 40

Ad−Hoc Bilingual English Task − Distribution of the Topics of the Experiment 00% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100% R−Precision

DSV_AMH_BLNG_RUN4 DSV_AMH_BLNG_RUN4

Priority

Query Construction

50,000 Source Language

2,247 Topic Fields 868 Pooled 5

AUTOMATIC

Amharic

title, description, narrative false 0.0083 First translation for WSD 0.1137 0%0% Ad−Hoc Bilingual English Task − Box plot of the Topics of the Experiment Precision averages (%) for individual queries

Ad−HocBilingualEnglishTask−ComparisontoMedianAveragePrecisionbyTopic(Topics401−AHto425−AH) 1 0.8 0.6 0.4 ce 0.2 reen 0 ifD−0.2 −0.4 −0.6 −0.8 1 0.8 0.6 0.4 ce 0.2 reen 0 ifD−0.2 −0.4 −0.6 −0.8 40

Ad−Hoc Bilingual English Task − Distribution of the Topics of the Experiment 00% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100% Average Precision

DSV_AMH_BLNG_RUN5 DSV_AMH_BLNG_RUN5 0%5 Ad−Hoc Bilingual English Task − Box plot of the Topics of the Experiment

R_PRECISION

Precision averages (%) for individual queries

Ad−HocBilingualEnglishTask−ComparisontoMedianR−PrecisionbyTopic(Topics401−AHto425−AH) 1 0.8 0.6 0.4 ce 0.2 reen 0 ifD−0.2 −0.4 −0.6 −0.8 1 0.8 0.6 0.4 ce 0.2 reen 0 ifD−0.2 −0.4 −0.6 −0.8

Ad−Hoc Bilingual English Task − Distribution of the Topics of the Experiment 00% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100% R−Precision

DSV_AMH_BLNG_RUN5 DSV_AMH_BLNG_RUN5

Priority

Query Construction

50,000 Source Language

2,247 Topic Fields 1,030 Pooled 6

AUTOMATIC

Amharic

title, description, narrative false 0.0099 First translation for WSD with pseudo relevance 0.1455 feedback 0%0% Ad−Hoc Bilingual English Task − Box plot of the Topics of the Experiment Precision averages (%) for individual queries

Ad−HocBilingualEnglishTask−ComparisontoMedianAveragePrecisionbyTopic(Topics401−AHto425−AH) 1 0.8 0.6 0.4 ce 0.2 reen 0 ifD−0.2 −0.4 −0.6 −0.8 1 0.8 0.6 0.4 ce 0.2 reen 0 ifD−0.2 −0.4 −0.6 −0.8 40

Ad−Hoc Bilingual English Task − Distribution of the Topics of the Experiment 00% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100% Average Precision

DSV_AMH_BLNG_RUN6 DSV_AMH_BLNG_RUN6 20% 10% 0%5 Ad−Hoc Bilingual English Task − Box plot of the Topics of the Experiment

R_PRECISION

Precision averages (%) for individual queries

Ad−HocBilingualEnglishTask−ComparisontoMedianR−PrecisionbyTopic(Topics401−AHto425−AH) 1 0.8 0.6 0.4 ce 0.2 reen 0 ifD−0.2 −0.4 −0.6 −0.8 1 0.8 0.6 0.4 ce 0.2 reen 0 ifD−0.2 −0.4 −0.6 −0.8 30 t n iem25 r e p xE20 e h ftso15 cp i o fT10 o r e b m 5 u N

Ad−Hoc Bilingual English Task − Distribution of the Topics of the Experiment 00% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100% R−Precision

DSV_AMH_BLNG_RUN6 DSV_AMH_BLNG_RUN6 Overall statistics for 50 queries : Total number of documents over all queries

Retrieved

Relevant

Relevant retrieved Geometric Mean Average Precision Binary Preference (BPREF) Priority

0.0128 Manual WSD 0.1299 7

MANUAL

Amharic title, description, narrative false

Precision averages (%) for individual queries Ad−Hoc Bilingual English Task − Box plot of the Topics of the Experiment 0%0% 1 0.8 0.6 0.4 ce 0.2 reen 0 ifD−0.2 −0.4 −0.6 −0.8 1 0.8 0.6 0.4 ce 0.2 reen 0 ifD−0.2 −0.4 −0.6 −0.8 30 t n iem25 r e p xE20 e h ftso15 cp i o fT10 o r e b m 5 u N 0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%

Average Precision

Ad−Hoc Bilingual English Task − Distribution of the Topics of the Experiment 00% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100% Average Precision

DSV_AMH_BLNG_RUN7 DSV_AMH_BLNG_RUN7 −1 401−AH402−AH403−AH404−AH405−AH406−AH407−AH408−AH409−AH410−AH411−AH412−AH413−AH414−AH415−AH416−AH417−AH418−AH419−AH420−AH421−AH422−AH423−AH424−AH425−AH

TopicIdentifier

Ad−HocBilingualEnglishTask−ComparisontoMedianAveragePrecisionbyTopic(Topics426−AHto450−AH) −1 426−AH427−AH428−AH429−AH430−AH431−AH432−AH433−AH434−AH435−AH436−AH437−AH438−AH439−AH440−AH441−AH442−AH443−AH444−AH445−AH446−AH447−AH448−AH449−AH450−AH

TopicIdentifier 20% 10% 0%5 Ad−Hoc Bilingual English Task − Box plot of the Topics of the Experiment

R_PRECISION

Precision averages (%) for individual queries

Ad−HocBilingualEnglishTask−ComparisontoMedianR−PrecisionbyTopic(Topics401−AHto425−AH) 1 0.8 0.6 0.4 ce 0.2 reen 0 ifD−0.2 −0.4 −0.6 −0.8 1 0.8 0.6 0.4 ce 0.2 reen 0 ifD−0.2 −0.4 −0.6 −0.8 30 t n iem25 r e p xE20 e h tfso15 cp i o fT10 o r e b m 5 u N 0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%

R−Precision

Ad−Hoc Bilingual English Task − Distribution of the Topics of the Experiment 00% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100% R−Precision

DSV_AMH_BLNG_RUN7 DSV_AMH_BLNG_RUN7 Overall statistics for 50 queries : Total number of documents over all queries

Retrieved

Relevant

Relevant retrieved

Priority

Query Construction

50,000 Source Language

2,247 Topic Fields 1,104 Pooled 8

MANUAL

Amharic

title, description, narrative false 0.0158 Manual WSD with pseudo relevance feedback 0.1560 0%0% Ad−Hoc Bilingual English Task − Box plot of the Topics of the Experiment Precision averages (%) for individual queries

Ad−HocBilingualEnglishTask−ComparisontoMedianAveragePrecisionbyTopic(Topics401−AHto425−AH) 1 0.8 0.6 0.4 ce 0.2 reen 0 ifD−0.2 −0.4 −0.6 −0.8 1 0.8 0.6 0.4 ce 0.2 reen 0 ifD−0.2 −0.4 −0.6 −0.8 40

Ad−Hoc Bilingual English Task − Distribution of the Topics of the Experiment 00% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100% Average Precision

DSV_AMH_BLNG_RUN8 DSV_AMH_BLNG_RUN8 Docs Cutoff Levels Precision at DCL (%) 5 docs 18.00 10 docs 17.40 15 docs 16.00 20 docs 14.40 30 docs 12.93 100 docs 8.50 200 docs 6.00 500 docs 3.58 1000 docs 2.21 R-Precision (precision after R document retrieved, where R = Relevant retrieved) 13.31 100% 90% 80% 70% 60% 40% 30% 20% 10% 0%5 Ad−Hoc Bilingual English Task − Box plot of the Topics of the Experiment

R_PRECISION

Precision averages (%) for individual queries

Ad−HocBilingualEnglishTask−ComparisontoMedianR−PrecisionbyTopic(Topics401−AHto425−AH) 1 0.8 0.6 0.4 ce 0.2 reen 0 ifD−0.2 −0.4 −0.6 −0.8 1 0.8 0.6 0.4 ce 0.2 reen 0 ifD−0.2 −0.4 −0.6 −0.8

Ad−Hoc Bilingual English Task − Distribution of the Topics of the Experiment 00% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100% R−Precision

DSV_AMH_BLNG_RUN8 DSV_AMH_BLNG_RUN8 Overall statistics for 50 queries : Total number of documents over all queries

Retrieved

Relevant

Relevant retrieved Geometric Mean Average Precision Binary Preference (BPREF)

Priority

Query Construction 50,000 Source Language 911 Topic Fields 727 Pooled 1

AUTOMATIC

English

title, description true 0%0% Ad−Hoc Bilingual Hungarian Task − Box plot of the Topics of the Experiment 2.57 22.26 3.91 6.61 0.63 47.16 33.94 53.40 92.86 21.34 0.00 63.63 3.30 0.92 33.90 71.38 80.05 3.79 26.64 6.42 0.00 85.36 79.35 52.86 22.45 Precision averages (%) for individual queries

Ad−HocBilingualHungarianTask−ComparisontoMedianAveragePrecisionbyTopic(Topics401−AHto425−AH) 1 0.8 0.6 0.4 ce 0.2 reen 0 ifD−0.2 −0.4 −0.6 −0.8 1 0.8 0.6 0.4 ce 0.2 reen 0 ifD−0.2 −0.4 −0.6 −0.8 0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%

Average Precision

Ad−Hoc Bilingual Hungarian Task − Distribution of the Topics of the Experiment 00% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100% Average Precision

APLBIENHUTD4 APLBIENHUTD4 Docs Cutoff Levels Precision at DCL (%) 5 docs 32.40 10 docs 32.80 15 docs 28.40 20 docs 25.20 30 docs 21.60 100 docs 10.02 200 docs 5.82 500 docs 2.71 1000 docs 1.45 R-Precision (precision after R document retrieved, where R = Relevant retrieved) 29.00 100% 90% 80% 70% 60% 40% 30% 20% 10% 0%5 Ad−Hoc Bilingual Hungarian Task − Box plot of the Topics of the Experiment

R_PRECISION

0.00 28.57 0.00 15.38 0.00 63.64 33.33 46.51 87.50 26.67 0.00 61.54 8.82 0.00 30.77 66.67 77.78 3.70 43.75 0.00 0.00 86.96 75.00 48.15 26.92 Precision averages (%) for individual queries

Ad−HocBilingualHungarianTask−ComparisontoMedianR−PrecisionbyTopic(Topics401−AHto425−AH) 1 0.8 0.6 0.4 ce 0.2 reen 0 ifD−0.2 −0.4 −0.6 −0.8 1 0.8 0.6 0.4 ce 0.2 reen 0 ifD−0.2 −0.4 −0.6 −0.8 0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%

R−Precision

Ad−Hoc Bilingual Hungarian Task − Distribution of the Topics of the Experiment 00% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100% R−Precision

APLBIENHUTD4 APLBIENHUTD4 Overall statistics for 50 queries : Total number of documents over all queries

Retrieved

Relevant

Relevant retrieved Geometric Mean Average Precision Binary Preference (BPREF)

Priority

Query Construction 50,000 Source Language 911 Topic Fields 706 Pooled 2

AUTOMATIC

English

title, description true 100% 90% 80% 70% 60% 40% 30% 20% 10% 0%0% Ad−Hoc Bilingual Hungarian Task − Box plot of the Topics of the Experiment 2.35 22.58 5.34 8.62 4.57 51.24 32.35 63.25 93.67 17.85 0.00 61.92 7.11 4.43 11.44 70.87 82.60 0.83 30.48 5.55 0.00 94.64 78.45 74.66 20.57 Precision averages (%) for individual queries

Ad−HocBilingualHungarianTask−ComparisontoMedianAveragePrecisionbyTopic(Topics401−AHto425−AH) 1 0.8 0.6 0.4 ce 0.2 reen 0 ifD−0.2 −0.4 −0.6 −0.8 1 0.8 0.6 0.4 ce 0.2 reen 0 ifD−0.2 −0.4 −0.6 −0.8 0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%

Average Precision

Ad−Hoc Bilingual Hungarian Task − Distribution of the Topics of the Experiment 00% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100% Average Precision

APLBIENHUTD5 APLBIENHUTD5 100% 90% 80% 70% 60% Ad−Hoc Bilingual Hungarian Task − Box plot of the Topics of the Experiment

Interquartile range

Mean

Mean With No Outliers

Std With No Outliers 401-AH 402-AH 403-AH 404-AH 405-AH 406-AH 407-AH 408-AH 409-AH 410-AH 411-AH 412-AH 413-AH 414-AH 415-AH 416-AH 417-AH 418-AH 419-AH 420-AH 421-AH 422-AH 423-AH 424-AH 425-AH

References