Appendix D Results of the Domain Specific Track Giorgio Maria Di Nunzio dinunzio@dei.unipd.it 0 Nicola Ferro ferro@dei.unipd.it 0 Department of Information Engineering University of Padua Italy 335 376 -

Introduction 3

Results for CLEF 2008 Domain Specific Track 3. Individual Experiment Results and Graphs This section provides the individual results for each official experiment. For each experiment the following tables and graphs are shown: - Overall statistics and information - Interpolated recall vs precision averages plot - Average precision statistics and box plot - Average precision comparison to median plot - Document cutoff levels vs precision at DCL plot - R-Precision statistics and box plot - R-Precision comparison to median plot Topics are identified with DOIs, as well as the experiments. The prefix for the DOI of a topic is 10.2452. The following example shows how to build the DOI for a topic given its number: for topic 200-AH, the corresponding DOI is 10.2452/200-AH List of Submitted Experiments 7

 United States 10.2415/DS-MONO-EN CLEF2008.CHESHIRE.BRKMOENTD United States 10.2415/DS-MONO-EN CLEF2008.CHESHIRE.BRKMOENTDN United States 10.2415/DS-MONO-DE CLEF2008.CHESHIRE.BRKMODETD United States 10.2415/DS-MONO-DE CLEF2008.CHESHIRE.BRKMODETDN 10.2415/DS-MONO-EN CLEF2008.UNINE.UNINEDSEN1

10.2415/DS-MONO-DE

 CLEF2008.CHEMNITZ.CUT_MERGED 10.2415/DS-MONO-DE CLEF2008.CHEMNITZ.CUT_MERGED_THES 10.2415/DS-MONO-DE CLEF2008.UNINE.UNINEDSDE1

10.2415/DS-MONO-DE

 CLEF2008.UNINE.UNINEDSDE2

10.2415/DS-MONO-DE

 CLEF2008.UNINE.UNINEDSDE3

10.2415/DS-MONO-DE

 CLEF2008.UNINE.UNINEDSDE4

10.2415/DS-MONO-RU

 CLEF2008.CHEMNITZ.CUT_MERGED

10.2415/DS-MONO-EN

 CLEF2008.AMSTERDAM.UAMSBASELINE 10.2415/DS-MONO-EN CLEF2008.AMSTERDAM.UAMSCONCEPTMODELS 10.2415/DS-MONO-EN CLEF2008.AMSTERDAM.UAMSPARSRELMODELS 10.2415/DS-MONO-EN CLEF2008.AMSTERDAM.UAMSRELMODELS 10.2415/DS-MONO-EN CLEF2008.CHEMNITZ.CUT_MERGED 10.2415/DS-MONO-ENCLEF2008.CHEMNITZ.CUT_MERGED_THES chemnitz

en en en en en en en en en en en en de de de de de de de de de de ru ru ru ru ru ru ru ru ru en en en en de de ru ru de

TN T TN TN TD TD TD TDN TD TDN TD TD TD TD TD TDN TD TDN TD TD TD TD TD TD TDN TD TDN TD TD TD TDN TD TD TD TD TD TDN TD TDN TD de de de de en en ru en en en en en en ru ru ru de de en en en en en de de en en ru ru

TD TD TD TD TD TDN TD TDN TD TD TDN TDN TD TD TD TD TD TDN TD TDN TD TD TD TD TDN TD TDN TD TDN Track Overview Results and Graphs 11

Track Overview Results and

Domain−Specific Monolingual English

Task Top 5 Participants − Standard Recall Levels vs Mean Interpolated Precision 100%

0%

0% 1 0.8 0.6 0.4 0.2 e c n re 0 e if D −0.2 −0.4 −0.6 −0.8 −1 chemnitz [Experiment CUT_MERGED; MAP amsterdam [Experiment UAMSCONCEPTMODELS; MAP 29.22%; Pooled]

amsterdam [Experiment UAMSRELMODELS; MAP 23.96%; Pooled] amsterdam [Experiment UAMSPARSRELMODELS; MAP 23.96%; Pooled] amsterdam [Experiment UAMSBASELINE; MAP 20.77%; Pooled] hug [Experiment HUGMONO; MAP 17.14%; Not Pooled] 0% 10% 20% 30% 70% 80% 90%

100% 40% 50% 60%

Average Precision 10.2455/TUKEY_T_TEST.960B6B5E536CA28011AB5EDCFC0FC38A

Domain−Specific Monolingual English Task − Tukey T test with "top group" highlighted s t n e m i r e p x E

 CUT_MERGED UNINEDSEN1 CUT_MERGED_THES BRKMOENTD BRKMOENTDN 976 969 UAMSCONCEPTMODELS UAMSPARSRELMODELS UAMSRELMODELS UAMSBASELINE

HUGMONO

 Track Overview Results and

Domain−Specific Monolingual English 100%

Task Top 5 Participants − Retrieved documents vs Mean Precision

−0.2 −0.4 −0.6 −0.8 −1 10 15 20 30

 Retrieved 100 Documents (logarithmic

200 scale) 500 1000 Domain−Specific Monolingual English Task Top 5 Participants − Comparison to Median R−Precision by Topic (Topics 201−DS to 225−DS) chemnitz [Experiment CUT_MERGED; R−Prec 41.07%; Pooled] unine [Experiment UNINEDSEN1; R−Prec 39.87%; Pooled] darmstadt [Experiment 976; R−Prec 37.76%; Pooled] cheshire [Experiment BRKMOENTD; R−Prec 35.31%; Pooled] amsterdam [Experiment UAMSCONCEPTMODELS; R−Prec 33.45%; Pooled] 201−DS 202−DS 203−DS 204−DS Domain−Specific Monolingual English Task − Box Plot of the Topics chemnitz [Experiment CUT_MERGED; R−Prec 41.07%; Pooled] chemnitz [Experiment CUT_MERGED_THES; R−Prec 40.89%; Pooled] unine [Experiment UNINEDSEN1; R−Prec 39.87%; Pooled] darmstadt [Experiment 976; R−Prec 37.76%; Pooled] darmstadt [Experiment 969; R−Prec 35.35%; Pooled]

Domain−Specific Monolingual English Task − Tukey T test with "top group" highlighted s t n e m i r e p x E

 CUT_MERGED CUT_MERGED_THES UNINEDSEN1 976 969 BRKMOENTD BRKMOENTDN UAMSCONCEPTMODELS UAMSPARSRELMODELS UAMSRELMODELS UAMSBASELINE

HUGMONO Domain−Specific Bilingual Russian Task − Distribution of the Topics of the Experiment −1 201−DS 202−DS 203−DS 204−DS 205−DS 206−DS 207−DS 208−DS 209−DS 210−DS 21 −DS 212−DS Topi2c13Id−eDnStifier214−DS 215−DS 216−DS 217−DS 218−DS 219−DS 2 0−DS 2 1−DS 2 2−DS 2 3−DS 2 4−DS 2 5−DS

Domain−Specific Bilingual Russian Task − Retrieved documents vs Mean Precision BRKBIENRUTD Precision averages (%) for individual queries

Domain−Specific Bilingual Russian Task − Distribution of the Topics of the Experiment −1 201−DS 202−DS 203−DS 204−DS 205−DS 206−DS 207−DS 208−DS 209−DS 210−DS 21 −DS 212−DS Topi2c13Id−eDnStifier214−DS 215−DS 216−DS 217−DS 218−DS 219−DS 2 0−DS 2 1−DS 2 2−DS 2 3−DS 2 4−DS 2 5−DS

Domain−Specific Bilingual Russian Task − Box plot of the Topics of the Experiment Domain−Specific Bilingual Russian Task − Distribution of the Topics of the Experiment −1 201−DS 202−DS 203−DS 204−DS 205−DS 206−DS 207−DS 208−DS 209−DS 210−DS 21 −DS 212−DS Topi2c13Id−eDnStifier214−DS 215−DS 216−DS 217−DS 218−DS 219−DS 2 0−DS 2 1−DS 2 2−DS 2 3−DS 2 4−DS 2 5−DS

Domain−Specific Bilingual Russian Task − Retrieved documents vs Mean Precision BRKBIENRUTDN Precision averages (%) for individual queries 0%5 Domain−Specific Bilingual Russian Task − Box plot of the Topics of the Experiment

Domain−Specific Bilingual Russian Task − Distribution of the Topics of the Experiment −1 201−DS 202−DS 203−DS 204−DS 205−DS 206−DS 207−DS 208−DS 209−DS 210−DS 21 −DS 212−DS Topi2c13Id−eDnStifier214−DS 215−DS 216−DS 217−DS 218−DS 219−DS 2 0−DS 2 1−DS 2 2−DS 2 3−DS 2 4−DS 2 5−DS

Domain−Specific Multilingual Task − Box plot of the Topics of the Experiment Domain−Specific Multilingual Task − Distribution of the Topics of the Experiment −1 201−DS 202−DS 203−DS 204−DS 205−DS 206−DS 207−DS 208−DS 209−DS 210−DS 21 −DS 212−DS Topi2c13Id−eDnStifier214−DS 215−DS 216−DS 217−DS 218−DS 219−DS 2 0−DS 2 1−DS 2 2−DS 2 3−DS 2 4−DS 2 5−DS Docs Cutoff Levels Precision at DCL (%) 5 docs 65.60 10 docs 64.40 15 docs 59.20 20 docs 56.60 30 docs 52.93 100 docs 41.08 200 docs 30.50 500 docs 18.83 1000 docs 11.83 R-Precision (precision after R document retrieved, where R = Relevant retrieved) 34.34 100% 90% 80% 70% 60% 0%5 Precision averages (%) for individual queries

Domain−Specific Multilingual Task − Box plot of the Topics of the Experiment

Domain−Specific Multilingual Task − Distribution of the Topics of the Experiment 00% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%

R−Precision 1 0.8 0.6 0.4 0.2 −1 201−DS 202−DS 203−DS 204−DS 205−DS 206−DS 207−DS 208−DS 209−DS 210−DS 21 −DS 212−DS Topi2c13Id−eDnStifier214−DS 215−DS 216−DS 217−DS 218−DS 219−DS 2 0−DS 2 1−DS 2 2−DS 2 3−DS 2 4−DS 2 5−DS Priority

Query Construction 25,000 Source Language 4,715 Topic Fields 2,977 Pooled 2 AUTOMATIC English title, description false

Domain−Specific Multilingual Task − Box plot of the Topics of the Experiment Domain−Specific Multilingual Task − Distribution of the Topics of the Experiment 00% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100% Average Precision Precision averages (%) for individual queries −1 201−DS 202−DS 203−DS 204−DS 205−DS 206−DS 207−DS 208−DS 209−DS 210−DS 21 −DS 212−DS Topi2c13Id−eDnStifier214−DS 215−DS 216−DS 217−DS 218−DS 219−DS 2 0−DS 2 1−DS 2 2−DS 2 3−DS 2 4−DS 2 5−DS Docs Cutoff Levels Precision at DCL (%) 5 docs 55.20 10 docs 57.60 15 docs 54.93 20 docs 52.60 30 docs 50.67 100 docs 36.16 200 docs 28.36 500 docs 18.39 1000 docs 11.91 R-Precision (precision after R document retrieved, where R = Relevant retrieved) 33.83 100% 90% 80% 70% 60% 0%5 R_PRECISION Precision averages (%) for individual queries

Domain−Specific Multilingual Task − Distribution of the Topics of the Experiment 5 t n e ie4 m r xp E tfh3 e so c ip2 o T f o r e b1 m u N 00% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%

R−Precision −1 201−DS 202−DS 203−DS 204−DS 205−DS 206−DS 207−DS 208−DS 209−DS 210−DS 21 −DS 212−DS Topi2c13Id−eDnStifier214−DS 215−DS 216−DS 217−DS 218−DS 219−DS 2 0−DS 2 1−DS 2 2−DS 2 3−DS 2 4−DS 2 5−DS AVERAGE_PRECISION Priority

Query Construction 25,000 Source Language 4,715 Topic Fields 2,721 Pooled

Domain−Specific Multilingual Task − Box plot of the Topics of the Experiment

Domain−Specific Multilingual Task − Distribution of the Topics of the Experiment 6 t iren5 m e xp E4 e h tfscop3 i o f2 T o r e bu1 m N 00% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%

Average Precision −1 201−DS 202−DS 203−DS 204−DS 205−DS 206−DS 207−DS 208−DS 209−DS 210−DS 21 −DS 212−DS Topi2c13Id−eDnStifier214−DS 215−DS 216−DS 217−DS 218−DS 219−DS 2 0−DS 2 1−DS 2 2−DS 2 3−DS 2 4−DS 2 5−DS Docs Cutoff Levels Precision at DCL (%) 5 docs 51.20 10 docs 48.00 15 docs 46.93 20 docs 45.60 30 docs 42.53 100 docs 33.40 200 docs 24.92 500 docs 16.33 1000 docs 10.88 R-Precision (precision after R document retrieved, where R = Relevant retrieved) 29.78 100% 90% 80% 70% 60% 0%5 Precision averages (%) for individual queries

Domain−Specific Multilingual Task − Box plot of the Topics of the Experiment

Domain−Specific Multilingual Task − Distribution of the Topics of the Experiment 00% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%

R−Precision 1 0.8 0.6 0.4 0.2 −1 201−DS 202−DS 203−DS 204−DS 205−DS 206−DS 207−DS 208−DS 209−DS 210−DS 21 −DS 212−DS Topi2c13Id−eDnStifier214−DS 215−DS 216−DS 217−DS 218−DS 219−DS 2 0−DS 2 1−DS 2 2−DS 2 3−DS 2 4−DS 2 5−DS Priority

Query Construction 25,000 Source Language 4,715 Topic Fields 1,617 Pooled 1 AUTOMATIC German title, description false 0%0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%

Recall

Domain−Specific Multilingual Task − Box plot of the Topics of the Experiment Domain−Specific Multilingual Task − Distribution of the Topics of the Experiment 00% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100% Average Precision Precision averages (%) for individual queries BRKMUDETD −1 201−DS 202−DS 203−DS 204−DS 205−DS 206−DS 207−DS 208−DS 209−DS 210−DS 21 −DS 212−DS Topi2c13Id−eDnStifier214−DS 215−DS 216−DS 217−DS 218−DS 219−DS 2 0−DS 2 1−DS 2 2−DS 2 3−DS 2 4−DS 2 5−DS Docs Cutoff Levels Precision at DCL (%) 5 docs 22.40 10 docs 16.00 15 docs 17.87 20 docs 17.80 30 docs 18.13 100 docs 15.12 200 docs 12.84 500 docs 9.48 1000 docs 6.47 R-Precision (precision after R document retrieved, where R = Relevant retrieved) 16.03 100% 90% 80% 70% 60% 0%5 Domain−Specific Multilingual Task − Box plot of the Topics of the Experiment R_PRECISION Precision averages (%) for individual queries

Domain−Specific Multilingual Task − Distribution of the Topics of the Experiment −1 201−DS 202−DS 203−DS 204−DS 205−DS 206−DS 207−DS 208−DS 209−DS 210−DS 21 −DS 212−DS Topi2c13Id−eDnStifier214−DS 215−DS 216−DS 217−DS 218−DS 219−DS 2 0−DS 2 1−DS 2 2−DS 2 3−DS 2 4−DS 2 5−DS Overall statistics for 25 queries : Total number of documents over all queries Retrieved Relevant Relevant retrieved Priority

Query Construction 25,000 Source Language 4,715 Topic Fields 1,490 Pooled 2 AUTOMATIC German title, description, narrative false 0.0231 Multilingual from German using TREC 2 and Blind 0.1673 Feedback on title and description and narrative 0%0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%

Recall

Domain−Specific Multilingual Task − Box plot of the Topics of the Experiment 0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%

Average Precision Domain−Specific Multilingual Task − Distribution of the Topics of the Experiment 00% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100% Average Precision Precision averages (%) for individual queries BRKMUDETDN −1 201−DS 202−DS 203−DS 204−DS 205−DS 206−DS 207−DS 208−DS 209−DS 210−DS 21 −DS 212−DS Topi2c13Id−eDnStifier214−DS 215−DS 216−DS 217−DS 218−DS 219−DS 2 0−DS 2 1−DS 2 2−DS 2 3−DS 2 4−DS 2 5−DS Docs Cutoff Levels Precision at DCL (%) 5 docs 12.80 10 docs 14.00 15 docs 16.80 20 docs 16.20 30 docs 15.33 100 docs 14.48 200 docs 11.44 500 docs 8.57 1000 docs 5.96 R-Precision (precision after R document retrieved, where R = Relevant retrieved) 14.04 100% 90% 80% 70% 60% 40% 30% 20% 10% 0%5 Domain−Specific Multilingual Task − Box plot of the Topics of the Experiment R_PRECISION Precision averages (%) for individual queries

Domain−Specific Multilingual Task − Distribution of the Topics of the Experiment 8 tn7 e m ire6 xp E tfscoep45 h i o3 T f o r2 e b m u1 N 00% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%

R−Precision BRKMUDETDN −1 201−DS 202−DS 203−DS 204−DS 205−DS 206−DS 207−DS 208−DS 209−DS 210−DS 21 −DS 212−DS Topi2c13Id−eDnStifier214−DS 215−DS 216−DS 217−DS 218−DS 219−DS 2 0−DS 2 1−DS 2 2−DS 2 3−DS 2 4−DS 2 5−DS Overall statistics for 25 queries : Total number of documents over all queries Retrieved Relevant Relevant retrieved Priority

Query Construction 25,000 Source Language 4,715 Topic Fields 1,834 Pooled

Domain−Specific Multilingual Task − Standard Recall Levels vs Mean Interpolated Precision BRKMUENTD 0%0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%

Recall

Domain−Specific Multilingual Task − Box plot of the Topics of the Experiment 0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%

Average Precision Domain−Specific Multilingual Task − Distribution of the Topics of the Experiment −0.2 −0.4 −0.6 −0.8 00% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100% Average Precision Precision averages (%) for individual queries BRKMUENTD −1 201−DS 202−DS 203−DS 204−DS 205−DS 206−DS 207−DS 208−DS 209−DS 210−DS 21 −DS 212−DS Topi2c13Id−eDnStifier214−DS 215−DS 216−DS 217−DS 218−DS 219−DS 2 0−DS 2 1−DS 2 2−DS 2 3−DS 2 4−DS 2 5−DS Docs Cutoff Levels Precision at DCL (%) 5 docs 16.80 10 docs 19.60 15 docs 19.73 20 docs 20.60 30 docs 22.67 100 docs 19.44 200 docs 16.86 500 docs 11.65 1000 docs 7.34 R-Precision (precision after R document retrieved, where R = Relevant retrieved) 19.54 100% 90% 80% 70% 60% 40% 30% 20% 10% 0%5

Domain−Specific Multilingual Task − Retrieved documents vs Mean Precision BRKMUENTD Domain−Specific Multilingual Task − Box plot of the Topics of the Experiment R_PRECISION Precision averages (%) for individual queries −0.2 −0.4 −0.6 −0.8 0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%

R−Precision

Domain−Specific Multilingual Task − Distribution of the Topics of the Experiment 6 t irne5 m e xp E4 e h ftcspo3 i o f2 T o r e bu1 m N 00% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%

R−Precision BRKMUENTD −1 201−DS 202−DS 203−DS 204−DS 205−DS 206−DS 207−DS 208−DS 209−DS 210−DS 21 −DS 212−DS Topi2c13Id−eDnStifier214−DS 215−DS 216−DS 217−DS 218−DS 219−DS 2 0−DS 2 1−DS 2 2−DS 2 3−DS 2 4−DS 2 5−DS Overall statistics for 25 queries : Total number of documents over all queries Retrieved Relevant Relevant retrieved Priority

Query Construction 25,000 Source Language 4,715 Topic Fields 1,774 Pooled 4 AUTOMATIC English title, description, narrative false Domain−Specific Multilingual Task − Standard Recall Levels vs Mean Interpolated Precision BRKMUENTDN 0%0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%

Recall

Domain−Specific Multilingual Task − Box plot of the Topics of the Experiment 0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%

Average Precision Domain−Specific Multilingual Task − Distribution of the Topics of the Experiment −0.2 −0.4 −0.6 −0.8 00% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100% Average Precision Precision averages (%) for individual queries BRKMUENTDN −1 201−DS 202−DS 203−DS 204−DS 205−DS 206−DS 207−DS 208−DS 209−DS 210−DS 21 −DS 212−DS Topi2c13Id−eDnStifier214−DS 215−DS 216−DS 217−DS 218−DS 219−DS 2 0−DS 2 1−DS 2 2−DS 2 3−DS 2 4−DS 2 5−DS Docs Cutoff Levels Precision at DCL (%) 5 docs 19.20 10 docs 20.40 15 docs 21.33 20 docs 22.80 30 docs 23.20 100 docs 20.56 200 docs 16.56 500 docs 11.30 1000 docs 7.10 R-Precision (precision after R document retrieved, where R = Relevant retrieved) 18.70 100% 90% 80% 70% 60% 40% 30% 20% 10% 0%5

Domain−Specific Multilingual Task − Retrieved documents vs Mean Precision BRKMUENTDN Domain−Specific Multilingual Task − Box plot of the Topics of the Experiment R_PRECISION Precision averages (%) for individual queries 9.09 21.15 16.50 26.97 0.91 17.70 6.82 30.89 8.82 16.07 20.34 22.32 1 0.8 0.6 0.4 0.2 ce reen 0 if D −0.2 −0.4 −0.6 −0.8 0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%

R−Precision

Domain−Specific Multilingual Task − Distribution of the Topics of the Experiment 6 t irne5 m e xp E4 e h ftscop3 i o f2 T o r e ub1 m N 00% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%

R−Precision BRKMUENTDN −1 201−DS 202−DS 203−DS 204−DS 205−DS 206−DS 207−DS 208−DS 209−DS 210−DS 21 −DS 212−DS Topi2c13Id−eDnStifier214−DS 215−DS 216−DS 217−DS 218−DS 219−DS 2 0−DS 2 1−DS 2 2−DS 2 3−DS 2 4−DS 2 5−DS Overall statistics for 25 queries : Total number of documents over all queries Retrieved Relevant Relevant retrieved Priority

Query Construction 25,000 Source Language 4,715 Topic Fields 1,385 Pooled 5 AUTOMATIC Russian title, description false 0.0209 Multilingual from Russian using TREC 2 and Blind 0.1509 Feedback on title and description 0%0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%

Recall

Domain−Specific Multilingual Task − Box plot of the Topics of the Experiment 0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%

Average Precision Domain−Specific Multilingual Task − Distribution of the Topics of the Experiment 2.48 13.05 0.18 2.27 0.08 3.93 2.78 22.62 2.88 0.24 0.09 8.91 1 0.8 0.6 0.4 0.2 ce reen 0 if D −0.2 −0.4 −0.6 −0.8 00% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100% Average Precision Precision averages (%) for individual queries BRKMURUTD −1 201−DS 202−DS 203−DS 204−DS 205−DS 206−DS 207−DS 208−DS 209−DS 210−DS 21 −DS 212−DS Topi2c13Id−eDnStifier214−DS 215−DS 216−DS 217−DS 218−DS 219−DS 2 0−DS 2 1−DS 2 2−DS 2 3−DS 2 4−DS 2 5−DS Docs Cutoff Levels Precision at DCL (%) 5 docs 15.20 10 docs 14.00 15 docs 12.27 20 docs 12.60 30 docs 12.80 100 docs 12.28 200 docs 10.52 500 docs 7.90 1000 docs 5.54 R-Precision (precision after R document retrieved, where R = Relevant retrieved) 12.19 100% 90% 80% 70% 60% 40% 30% 20% 10% 0%5

Domain−Specific Multilingual Task − Retrieved documents vs Mean Precision BRKMURUTD Domain−Specific Multilingual Task − Box plot of the Topics of the Experiment R_PRECISION Precision averages (%) for individual queries 201-DS 202-DS 203-DS 204-DS 205-DS 206-DS 207-DS 208-DS 209-DS 210-DS 211-DS 212-DS 213-DS 7.69 19.57 5.53 10.87 1.92 13.39 14.01 0.81 54.90 5.75 28.76 11.91 9.95 214-DS 215-DS 216-DS 217-DS 218-DS 219-DS 220-DS 221-DS 222-DS 223-DS 224-DS 225-DS 3.64 19.23 3.88 8.88 0.46 9.73 11.36 28.27 11.27 3.57 0.98 18.30 1 0.8 0.6 0.4 0.2 ce reen 0 if D −0.2 −0.4 −0.6 −0.8 0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%

R−Precision

Domain−Specific Multilingual Task − Distribution of the Topics of the Experiment 8 tn7 e m ire6 xp E ftcspoe45 h i o3 T f o r2 e b m u1 N 00% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%

R−Precision BRKMURUTD −1 201−DS 202−DS 203−DS 204−DS 205−DS 206−DS 207−DS 208−DS 209−DS 210−DS 21 −DS 212−DS Topi2c13Id−eDnStifier214−DS 215−DS 216−DS 217−DS 218−DS 219−DS 2 0−DS 2 1−DS 2 2−DS 2 3−DS 2 4−DS 2 5−DS Overall statistics for 25 queries : Total number of documents over all queries Retrieved Relevant Relevant retrieved Priority

Query Construction 25,000 Source Language 4,715 Topic Fields 1,459 Pooled 6 AUTOMATIC Russian title, description, narrative false 0.0226 Multilingual from Russian using TREC 2 and Blind 0.1506 Feedback on title and description and narrative 20% 10% 0%0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%

Recall

Domain−Specific Multilingual Task − Box plot of the Topics of the Experiment 0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%

Average Precision Domain−Specific Multilingual Task − Distribution of the Topics of the Experiment 0.99 8.08 0.19 1.72 0.02 4.22 2.65 41.40 3.71 0.65 0.21 7.74 1 0.8 0.6 0.4 0.2 ce reen 0 if D −0.2 −0.4 −0.6 −0.8 20 t n e m i rpe15 x E e h ftso10 cp i o T f o re 5 b m u N 00% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100% Average Precision Precision averages (%) for individual queries BRKMURUTDN −1 201−DS 202−DS 203−DS 204−DS 205−DS 206−DS 207−DS 208−DS 209−DS 210−DS 21 −DS 212−DS Topi2c13Id−eDnStifier214−DS 215−DS 216−DS 217−DS 218−DS 219−DS 2 0−DS 2 1−DS 2 2−DS 2 3−DS 2 4−DS 2 5−DS Docs Cutoff Levels Precision at DCL (%) 5 docs 10.40 10 docs 10.00 15 docs 12.53 20 docs 13.80 30 docs 14.80 100 docs 13.96 200 docs 11.88 500 docs 8.47 1000 docs 5.84 R-Precision (precision after R document retrieved, where R = Relevant retrieved) 100% 90% 80% 70% 60% Domain−Specific Multilingual Task − Box plot of the Topics of the Experiment Interquartile range Mean Mean With No Outliers Std With No Outliers Precision averages (%) for individual queries 201-DS 202-DS 203-DS 204-DS 205-DS 206-DS 207-DS 208-DS 209-DS 210-DS 211-DS 212-DS 213-DS 16.72 18.48 7.54 2.17 1.92 5.51 10.14 8.06 54.90 4.42 24.03 19.57 7.33 214-DS 215-DS 216-DS 217-DS 218-DS 219-DS 220-DS 221-DS 222-DS 223-DS 224-DS 225-DS 1 8 tn7 e m ire6 xp E tfsceop45 h i o3 T f o r2 e b m u1 N 00% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%

R−Precision BRKMURUTDN

 0.0064 Bilingual English to Russian using TREC2 with Blind 0.0775 Feedback on title and description 0.0047 Bilingual English to Russian using TREC2 with Blind 0.0526 Feedback on title and description and narrative 0.0142 Multilingual from German using TREC 2 and Blind 0.1978 Feedback on title and description 0.0526 Multilingual from English using TREC 2 and Blind 0.2062 Feedback on title and description 0.0501 Multilingual from English using TREC 2 and Blind 0.2023 Feedback on title and description and narrative