Introduction 3
Results for CLEF 2006 Domain Specific Tracks List of Submitted Experiments 7 Participant Track Overview Results and Graphs 11 v 40% A 30% 20% 10% 0%
0% 1 0.8 0.6 0.4 0.2 −0.2 −0.4 −0.6 −0.8 −1 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 Topic Identifier
Domain−Specific Monolingual English track − Box Plot of the Topics UniNEen3 [MAP 45.76%; Pooled] UniNEen2 [MAP 43.03%; Pooled]
UniNEen1 [MAP 42.92%; Pooled] BERK_MO_EN_T2FB [MAP 41.36%; Pooled]
TUCMIgirten2 [MAP 35.53%; Pooled] TUCMIgirten4 [MAP 35.38%; Pooled] TUCMIgirten1 [MAP 35.10%; Pooled] TUCMIgirten3 [MAP 34.50%; Pooled] 0% 10% 20% 30% 40% 50% 60%
Mean Average Precision
70% 80% 90% 100%
Domain−Specific Monolingual English track − Tukey T test with "top group" highlighted UniNEen3 UniNEen2
UniNEen1 tsnBERK_MO_EN_T2FB e m i r e xp TUCMIgirten2 E
TUCMIgirten4 TUCMIgirten1 TUCMIgirten3 40% 30% 20% 10% 0%
5 1 0.8 0.6 0.4 0.2 −0.2 −0.4 −0.6 −0.8 −1 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 Topic Identifier
Domain−Specific Monolingual English track − Box Plot of the Topics UniNEen3 [R−Prec 47.71%; Pooled] UniNEen2 [R−Prec 45.22%; Pooled]
UniNEen1 [R−Prec 44.80%; Pooled] BERK_MO_EN_T2FB [R−Prec 44.19%; Pooled]
TUCMIgirten4 [R−Prec 39.34%; Pooled] TUCMIgirten2 [R−Prec 39.33%; Pooled] TUCMIgirten1 [R−Prec 38.96%; Pooled] TUCMIgirten3 [R−Prec 38.52%; Pooled] 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
R−Precision UniNEen3 UniNEen2
Domain−Specific Monolingual English track − Tukey T test with "top group" highlighted
Domain−Specific Bilingual German track − Distribution of the Topics of the Experiment 00% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Mean Average Precision −1 151 152 153 154 155 156 157 158 159 160 161 162 163
TopicIdentifie1r64 165 166 167 168 169 170 171 172 173 174 175
Domain−Specific Bilingual German track − Box plot of the Topics of the Experiment
Domain−Specific Bilingual German track − Distribution of the Topics of the Experiment 8 tn7 e m ire6 xp E iftcspoe45 h fo3 T o r2 e b m u1 N 00% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Exact R−Precision
Domain−Specific Bilingual German track − Box plot of the Topics of the Experiment
Domain−Specific Bilingual German track − Distribution of the Topics of the Experiment 6 t iren5 m xep E4 e h iftcspo3 o f2 T o r e b m1 u N 00% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Mean Average Precision Docs Cutoff Levels Precision at DCL (%) 5 docs 50.40 10 docs 49.60 15 docs 46.93 20 docs 46.40 30 docs 43.73 100 docs 34.04 200 docs 27.20 500 docs 16.80 1000 docs 10.28 R-Precision (precision after R document retrieved, where R = Relevant retrieved) 30.38 Precision averages (%) for individual queries Topic 151 Topic 152 Topic 153 Topic 154 Topic 155 Topic 156 Topic 157 Topic 158 Topic 159 Topic 160 Topic 161 Topic 162 Topic 163 13.78 13.46 40.00 63.50 32.64 13.98 20.00 13.48 49.07 15.97 44.30 22.84 27.24
Topic 164 Topic 165 Topic 166 Topic 167 Topic 168 Topic 169 Topic 170 Topic 171 Topic 172 Topic 173 Topic 174 Topic 175 17.31 31.51 56.99 6.67 40.82 27.27 35.90 30.81 13.64 59.92 52.84 15.62 1 0.8 0.6 0.4 0.2 hagen
Domain−Specific Bilingual German track − Retrieved documents vs Precision
FUHegpyynl500 100% 90% 80% 70% 60% 0%5
Domain−Specific Bilingual German track − Box plot of the Topics of the Experiment
Domain−Specific Bilingual German track − Distribution of the Topics of the Experiment 5 t n e ir4 m xep E fth3 e so c ip2 o T f o r e b1 m u N 00% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Exact R−Precision 2 AUTOMATIC English title, description true Precision averages (%) for individual queries Topic 151 Topic 152 Topic 153 Topic 154 Topic 155 Topic 156 Topic 157 Topic 158 Topic 159 Topic 160 Topic 161 Topic 162 Topic 163 8.36 1.11 40.89 28.12 15.65 9.97 0.19 2.24 69.19 16.05 48.50 14.35 12.52
Topic 164 Topic 165 Topic 166 Topic 167 Topic 168 Topic 169 Topic 170 Topic 171 Topic 172 Topic 173 Topic 174 Topic 175 1.95 20.68 60.51 6.57 32.87 11.72 16.17 20.91 7.01 48.68 45.81 7.48 1 0.8 0.6 0.4 0.2
Domain−Specific Bilingual German track − Box plot of the Topics of the Experiment
Domain−Specific Bilingual German track − Distribution of the Topics of the Experiment 5 t n e ir4 m xpe E tfh3 e so c ip2 o T f o r e b1 m u N 00% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Mean Average Precision Docs Cutoff Levels Precision at DCL (%) 5 docs 44.80 10 docs 39.60 15 docs 39.47 20 docs 38.20 30 docs 37.33 100 docs 31.48 200 docs 25.86 500 docs 15.51 1000 docs 9.69 R-Precision (precision after R document retrieved, where R = Relevant retrieved) 27.81 Precision averages (%) for individual queries Topic 151 Topic 152 Topic 153 Topic 154 Topic 155 Topic 156 Topic 157 Topic 158 Topic 159 Topic 160 Topic 161 Topic 162 Topic 163 17.32 7.21 45.33 39.05 19.17 15.05 1.54 7.87 62.04 27.78 49.37 31.03 24.12
Topic 164 Topic 165 Topic 166 Topic 167 Topic 168 Topic 169 Topic 170 Topic 171 Topic 172 Topic 173 Topic 174 Topic 175 0.00 31.51 60.75 15.56 43.54 9.09 23.08 34.85 9.09 52.38 52.84 15.62 1 0.8 0.6 0.4 0.2 hagen
Domain−Specific Bilingual German track − Retrieved documents vs Precision
FUHegpyydl500 100% 90% 80% 70% 60% 0%5
Domain−Specific Bilingual German track − Box plot of the Topics of the Experiment
Domain−Specific Bilingual German track − Distribution of the Topics of the Experiment 5 t n e ir4 m xpe E tfh3 e so c ip2 o T f o r e b1 m u N 00% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Exact R−Precision 3 AUTOMATIC English title, description true Precision averages (%) for individual queries Topic 151 Topic 152 Topic 153 Topic 154 Topic 155 Topic 156 Topic 157 Topic 158 Topic 159 Topic 160 Topic 161 Topic 162 Topic 163 5.67 1.89 3.97 80.19 26.94 0.67 6.07 7.83 25.72 2.01 25.36 9.66 18.72
Topic 164 Topic 165 Topic 166 Topic 167 Topic 168 Topic 169 Topic 170 Topic 171 Topic 172 Topic 173 Topic 174 Topic 175 23.38 25.94 25.25 2.43 14.61 24.17 28.12 17.90 9.86 52.05 51.90 4.76 1 0.8 0.6 0.4 0.2
Domain−Specific Bilingual German track − Box plot of the Topics of the Experiment
Domain−Specific Bilingual German track − Distribution of the Topics of the Experiment 6 t iren5 m xpe E4 e h iftscop3 o f2 T o r e b m1 u N 00% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Mean Average Precision hagen Precision averages (%) for individual queries Topic 151 Topic 152 Topic 153 Topic 154 Topic 155 Topic 156 Topic 157 Topic 158 Topic 159 Topic 160 Topic 161 Topic 162 Topic 163 17.32 6.73 8.00 72.26 35.23 2.15 18.46 16.85 28.70 6.25 33.23 21.12 28.79
Topic 164 Topic 165 Topic 166 Topic 167 Topic 168 Topic 169 Topic 170 Topic 171 Topic 172 Topic 173 Topic 174 Topic 175 26.92 32.88 32.26 4.44 22.45 18.18 35.90 31.31 13.64 56.75 60.23 7.81 60% 0%5
Domain−Specific Bilingual German track − Box plot of the Topics of the Experiment 1 0.8 0.6 0.4 0.2
Domain−Specific Bilingual German track − Distribution of the Topics of the Experiment 00% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Exact R−Precision hagen 4 AUTOMATIC English title, description true Precision averages (%) for individual queries Topic 151 Topic 152 Topic 153 Topic 154 Topic 155 Topic 156 Topic 157 Topic 158 Topic 159 Topic 160 Topic 161 Topic 162 Topic 163 9.51 0.77 5.47 43.94 21.00 4.87 0.16 1.20 53.97 2.89 32.37 9.22 9.28
Topic 164 Topic 165 Topic 166 Topic 167 Topic 168 Topic 169 Topic 170 Topic 171 Topic 172 Topic 173 Topic 174 Topic 175 1.89 18.68 31.06 2.65 12.33 6.96 25.77 22.22 8.43 45.84 57.52 13.99 1 0.8 0.6 0.4 0.2 −0.2 −0.4 −0.6 −0.8
Domain−Specific Bilingual German track − Box plot of the Topics of the Experiment
Domain−Specific Bilingual German track − Distribution of the Topics of the Experiment 8 tn7 e m ire6 xp E itfscoep45 h fo3 T o r2 e b m u1 N 00% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Mean Average Precision hagen Precision averages (%) for individual queries Topic 151 Topic 152 Topic 153 Topic 154 Topic 155 Topic 156 Topic 157 Topic 158 Topic 159 Topic 160 Topic 161 Topic 162 Topic 163 22.83 2.88 8.00 54.01 25.91 10.75 0.00 5.62 52.78 5.56 38.92 17.24 21.40
Topic 164 Topic 165 Topic 166 Topic 167 Topic 168 Topic 169 Topic 170 Topic 171 Topic 172 Topic 173 Topic 174 Topic 175 0.00 27.40 36.56 4.44 20.41 9.09 35.90 30.81 18.18 50.40 63.64 18.75 60% 0%5
Domain−Specific Bilingual German track − Box plot of the Topics of the Experiment 1 0.8 0.6 0.4 0.2 −0.2 −0.4 −0.6 −0.8
4 ten3.5 m i rpe 3 x E iftsoeh2.25 c fopT1.5 o re 1 b uNm0.5
Domain−Specific Bilingual German track − Distribution of the Topics of the Experiment 00% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Exact R−Precision hagen 5 AUTOMATIC English title, description true 0%0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
Interpolated Recall Domain−Specific Bilingual German track − Box plot of the Topics of the Experiment Precision averages (%) for individual queries Topic 151 Topic 152 Topic 153 Topic 154 Topic 155 Topic 156 Topic 157 Topic 158 Topic 159 Topic 160 Topic 161 Topic 162 Topic 163 7.35 1.59 32.60 29.35 15.92 10.36 0.19 2.29 67.46 16.43 48.50 16.03 11.33
Topic 164 Topic 165 Topic 166 Topic 167 Topic 168 Topic 169 Topic 170 Topic 171 Topic 172 Topic 173 Topic 174 Topic 175 2.06 20.68 57.06 9.32 31.30 11.72 22.28 21.20 7.03 48.69 46.77 7.55 1 0.8 0.6 0.4 0.2 −0.2 −0.4 −0.6 −0.8
4 ten3.5 m i rpe 3 x E itfsoeh2.25 c fopT1.5 o re 1 b uNm0.5
Domain−Specific Bilingual German track − Distribution of the Topics of the Experiment 00% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Mean Average Precision Docs Cutoff Levels Precision at DCL (%) 5 docs 42.40 10 docs 40.00 15 docs 40.53 20 docs 39.00 30 docs 37.87 100 docs 31.76 200 docs 25.72 500 docs 15.29 1000 docs 9.69 R-Precision (precision after R document retrieved, where R = Relevant retrieved) 27.77 Precision averages (%) for individual queries Topic 151 Topic 152 Topic 153 Topic 154 Topic 155 Topic 156 Topic 157 Topic 158 Topic 159 Topic 160 Topic 161 Topic 162 Topic 163 16.54 10.10 36.00 39.78 19.69 15.05 1.54 7.87 60.19 28.47 49.37 32.33 21.79
Topic 164 Topic 165 Topic 166 Topic 167 Topic 168 Topic 169 Topic 170 Topic 171 Topic 172 Topic 173 Topic 174 Topic 175 0.00 31.51 54.84 17.78 44.22 9.09 25.64 35.35 13.64 52.38 53.98 17.19 1 0.8 0.6 0.4 0.2 −0.2 −0.4 −0.6 −0.8 hagen
Domain−Specific Bilingual German track − Retrieved documents vs Precision
FUHegpyydl500R 100% 90% 80% 70% 60% 0%5
Domain−Specific Bilingual German track − Box plot of the Topics of the Experiment
Domain−Specific Bilingual German track − Distribution of the Topics of the Experiment 5 t n e ir4 m xpe E fth3 e so c ip2 o T f o r e b1 m u N 00% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Exact R−Precision Overall statistics for 25 queries : Total number of documents over all queries Retrieved Relevant Relevant retrieved Priority
Query Construction 25,000 Source Language 913 Topic Fields 652 Pooled 2 AUTOMATIC German title, description true 0.0216 English to Russian using PROMT 0.1646 0%0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
Interpolated Recall Domain−Specific Bilingual Russian track − Box plot of the Topics of the Experiment Precision averages (%) for individual queries Topic 151 Topic 152 Topic 153 Topic 154 Topic 155 Topic 156 Topic 157 Topic 158 Topic 159 Topic 160 Topic 161 Topic 162 Topic 163 7.19 50.03 14.10 28.74 73.33 0.78 0.00 3.75 56.16 0.08 62.15 4.83 45.73
Topic 164 Topic 165 Topic 166 Topic 167 Topic 168 Topic 169 Topic 170 Topic 171 Topic 172 Topic 173 Topic 174 Topic 175 2.27 1.91 5.42 1.18 25.63 11.51 3.96 1.10 0.30 11.90 0.00 0.00 1 0.8 0.6 0.4 0.2 −0.2 −0.4 −0.6 −0.8 15 t n e m ir xpe E10 e h tf so cp i o fT 5 o r e b m u N 0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Mean Average Precision
Domain−Specific Bilingual Russian track − Distribution of the Topics of the Experiment 00% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Mean Average Precision −1 151 152 153 154 155 156 157 158 159 160 161 162 163
TopicIdentifie1r64 165 166 167 168 169 170 171 172 173 174 175 Docs Cutoff Levels Precision at DCL (%) 5 docs 23.20 10 docs 22.00 15 docs 21.07 20 docs 21.20 30 docs 19.73 100 docs 15.76 200 docs 10.44 500 docs 4.82 1000 docs 2.61 R-Precision (precision after R document retrieved, where R = Relevant retrieved) 17.38 Precision averages (%) for individual queries Topic 151 Topic 152 Topic 153 Topic 154 Topic 155 Topic 156 Topic 157 Topic 158 Topic 159 Topic 160 Topic 161 Topic 162 Topic 163 6.25 45.35 13.04 43.43 70.27 0.00 0.00 0.00 60.00 2.70 55.68 8.62 54.87
Topic 164 Topic 165 Topic 166 Topic 167 Topic 168 Topic 169 Topic 170 Topic 171 Topic 172 Topic 173 Topic 174 Topic 175 0.00 0.00 0.00 0.00 25.00 25.00 4.35 0.00 0.00 20.00 0.00 0.00 60% 30% 20% 10% 0%5
Domain−Specific Bilingual Russian track − Box plot of the Topics of the Experiment 1 0.8 0.6 0.4 0.2 −0.2 −0.4 −0.6 −0.8 15 t n e m ir xpe E10 e h tf so cp i o fT 5 o r e b m u N 0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Exact R−Precision
Domain−Specific Bilingual Russian track − Distribution of the Topics of the Experiment 00% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Exact R−Precision −1 151 152 153 154 155 156 157 158 159 160 161 162 163
TopicIdentifie1r64 165 166 167 168 169 170 171 172 173 174 175 Overall statistics for 25 queries : Total number of documents over all queries Retrieved Relevant Relevant retrieved Priority
Query Construction 25,000 Source Language 913 Topic Fields 571 Pooled 1 AUTOMATIC German title, description true 0.0086 English to Russian using Babelfish 0.1156 0%0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
Interpolated Recall Domain−Specific Bilingual Russian track − Box plot of the Topics of the Experiment Precision averages (%) for individual queries Topic 151 Topic 152 Topic 153 Topic 154 Topic 155 Topic 156 Topic 157 Topic 158 Topic 159 Topic 160 Topic 161 Topic 162 Topic 163 10.69 31.74 9.27 2.63 0.06 1.30 0.00 14.41 59.91 1.02 70.04 27.56 44.88
Topic 164 Topic 165 Topic 166 Topic 167 Topic 168 Topic 169 Topic 170 Topic 171 Topic 172 Topic 173 Topic 174 Topic 175 0.29 1.31 0.00 0.96 0.23 0.35 8.08 1.53 1.81 0.77 0.00 0.00 1 0.8 0.6 0.4 0.2 −0.2 −0.4 −0.6 −0.8 20 t n e m irpe15 x E e h iftso10 cp o T f o re 5 b m u N 0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Mean Average Precision
Domain−Specific Bilingual Russian track − Distribution of the Topics of the Experiment 00% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Mean Average Precision −1 151 152 153 154 155 156 157 158 159 160 161 162 163
TopicIdentifie1r64 165 166 167 168 169 170 171 172 173 174 175 Docs Cutoff Levels Precision at DCL (%) 5 docs 19.20 10 docs 16.00 15 docs 16.53 20 docs 15.80 30 docs 14.80 100 docs 11.72 200 docs 8.26 500 docs 4.14 1000 docs 2.28 R-Precision (precision after R document retrieved, where R = Relevant retrieved) 12.91 Precision averages (%) for individual queries Topic 151 Topic 152 Topic 153 Topic 154 Topic 155 Topic 156 Topic 157 Topic 158 Topic 159 Topic 160 Topic 161 Topic 162 Topic 163 12.50 40.70 13.04 16.16 1.35 0.00 0.00 14.29 60.00 7.21 65.91 24.14 53.10
Topic 164 Topic 165 Topic 166 Topic 167 Topic 168 Topic 169 Topic 170 Topic 171 Topic 172 Topic 173 Topic 174 Topic 175 0.00 5.56 0.00 0.00 0.00 0.00 8.70 0.00 0.00 0.00 0.00 0.00
Domain−Specific Bilingual Russian track − Retrieved documents vs Precision 60% 30% 20% 10% 0%5
Domain−Specific Bilingual Russian track − Box plot of the Topics of the Experiment 0.8 0.6 0.4 0.2 ce reen 0 if D −0.2 −0.4 −0.6 −0.8 15 t n e m ir xpe E10 e h ft so cp i o fT 5 o r e b m u N 0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Exact R−Precision
Domain−Specific Bilingual Russian track − Distribution of the Topics of the Experiment 00% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Exact R−Precision −1 151 152 153 154 155 156 157 158 159 160 161 162 163
TopicIdentifie1r64 165 166 167 168 169 170 171 172 173 174 175 Overall statistics for 25 queries : Total number of documents over all queries Retrieved Relevant Relevant retrieved Priority
Query Construction 25,000 Source Language 913 Topic Fields 456 Pooled 3 AUTOMATIC German title, description true 0.0077 German to Russian using PROMT 0.1034 0%0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
Interpolated Recall Domain−Specific Bilingual Russian track − Box plot of the Topics of the Experiment Precision averages (%) for individual queries Topic 151 Topic 152 Topic 153 Topic 154 Topic 155 Topic 156 Topic 157 Topic 158 Topic 159 Topic 160 Topic 161 Topic 162 Topic 163 13.94 0.06 14.29 7.52 0.00 2.68 50.29 0.27 67.26 0.00 4.14 33.44 14.68
Topic 164 Topic 165 Topic 166 Topic 167 Topic 168 Topic 169 Topic 170 Topic 171 Topic 172 Topic 173 Topic 174 Topic 175 8.13 14.05 3.66 1.41 29.52 3.58 1.26 0.33 0.14 0.00 0.00 0.00 1 0.8 0.6 0.4 0.2 ce reen 0 if D −0.2 −0.4 −0.6 −0.8 15 t n e m ir xep E10 e h ft so cp i o fT 5 o r e b m u N 0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Mean Average Precision
Domain−Specific Bilingual Russian track − Distribution of the Topics of the Experiment 00% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Mean Average Precision −1 151 152 153 154 155 156 157 158 159 160 161 162 163
TopicIdentifie1r64 165 166 167 168 169 170 171 172 173 174 175 Docs Cutoff Levels Precision at DCL (%) 5 docs 12.00 10 docs 10.80 15 docs 10.13 20 docs 10.20 30 docs 10.00 100 docs 8.32 200 docs 5.50 500 docs 3.01 1000 docs 1.82 R-Precision (precision after R document retrieved, where R = Relevant retrieved) 11.87 Precision averages (%) for individual queries Topic 151 Topic 152 Topic 153 Topic 154 Topic 155 Topic 156 Topic 157 Topic 158 Topic 159 Topic 160 Topic 161 Topic 162 Topic 163 25.00 2.33 13.04 23.23 0.00 0.00 50.00 0.00 71.43 0.00 9.09 37.93 20.35
Topic 164 Topic 165 Topic 166 Topic 167 Topic 168 Topic 169 Topic 170 Topic 171 Topic 172 Topic 173 Topic 174 Topic 175 0.00 19.44 0.00 0.00 25.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
Domain−Specific Bilingual Russian track − Retrieved documents vs Precision 60% 30% 20% 10% 0%5
Domain−Specific Bilingual Russian track − Box plot of the Topics of the Experiment 0.8 0.6 0.4 0.2 ce reen 0 if D −0.2 −0.4 −0.6 −0.8 15 t n e m ir xpe E10 e h ft so cp i o fT 5 o r e b m u N 0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Exact R−Precision
Domain−Specific Bilingual Russian track − Distribution of the Topics of the Experiment 00% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Exact R−Precision −1 151 152 153 154 155 156 157 158 159 160 161 162 163
TopicIdentifie1r64 165 166 167 168 169 170 171 172 173 174 175 Overall statistics for 25 queries : Total number of documents over all queries Retrieved Relevant Relevant retrieved Priority
Query Construction 25,000 Source Language 8,911 Topic Fields 1,578 Pooled 1 AUTOMATIC German title, description false 0.0143 German to Russian using PROMT, and German to 0.1606 English using Babelfish 100% 90% 80% 70% 30% 20% 10%
Domain−Specific Multilingual track − Interpolated Recall vs Average Precision 0%0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
Interpolated Recall Domain−Specific Multilingual track − Box plot of the Topics of the Experiment Precision averages (%) for individual queries Topic 151 Topic 152 Topic 153 Topic 154 Topic 155 Topic 156 Topic 157 Topic 158 Topic 159 Topic 160 Topic 161 Topic 162 Topic 163 1.12 0.01 9.16 3.42 0.72 6.43 5.12 0.62 2.67 0.17 9.61 0.30 2.52
Topic 164 Topic 165 Topic 166 Topic 167 Topic 168 Topic 169 Topic 170 Topic 171 Topic 172 Topic 173 Topic 174 Topic 175 5.47 2.40 0.28 1.96 4.21 0.03 19.52 33.52 4.68 2.35 0.39 0.17 1 0.8 0.6 0.4 0.2 ce reen 0 if D −0.2 −0.4 −0.6 −0.8 20 t n e m irpe15 x E e h iftso10 cp o T f o re 5 b m u N 0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Mean Average Precision
Domain−Specific Multilingual track − Distribution of the Topics of the Experiment 00% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Mean Average Precision −1 151 152 153 154 155 156 157 158 159 160 161 162 163
TopicIdentifie1r64 165 166 167 168 169 170 171 172 173 174 175 Docs Cutoff Levels Precision at DCL (%) 5 docs 12.00 10 docs 14.40 15 docs 15.73 20 docs 17.20 30 docs 18.27 100 docs 16.36 200 docs 13.24 500 docs 8.52 1000 docs 6.31 R-Precision (precision after R document retrieved, where R = Relevant retrieved) 11.15 100% 90% 80% 70% 60% 30% 20% 10% 0%5 Precision averages (%) for individual queries Topic 151 Topic 152 Topic 153 Topic 154 Topic 155 Topic 156 Topic 157 Topic 158 Topic 159 Topic 160 Topic 161 Topic 162 Topic 163 5.33 0.47 22.74 12.48 4.20 13.04 13.58 5.12 7.78 3.08 25.22 2.32 10.35
Topic 164 Topic 165 Topic 166 Topic 167 Topic 168 Topic 169 Topic 170 Topic 171 Topic 172 Topic 173 Topic 174 Topic 175 17.56 5.33 2.05 8.45 13.20 0.00 33.33 44.76 15.62 7.63 3.46 1.71 1 0.8 0.6 0.4 0.2 ce reen 0 if D −0.2 −0.4 −0.6 −0.8
Domain−Specific Multilingual track − Retrieved documents vs Precision
Domain−Specific Multilingual track − Box plot of the Topics of the Experiment 0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Exact R−Precision
Domain−Specific Multilingual track − Distribution of the Topics of the Experiment 8 tn7 e m ire6 xp E itfsceop54 h fo3 T o r2 e b m u1 N 00% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Exact R−Precision −1 151 152 153 154 155 156 157 158 159 160 161 162 163
TopicIdentifie1r64 165 166 167 168 169 170 171 172 173 174 175 Overall statistics for 25 queries : Total number of documents over all queries Retrieved Relevant Relevant retrieved Priority
Query Construction 25,000 Source Language 8,911 Topic Fields 2,422 Pooled 2 AUTOMATIC German title, description false 0.0298 English to German and Russian using Babelfish 0.2097 100% 90% 80% 70% 30% 20% 10%
Domain−Specific Multilingual track − Interpolated Recall vs Average Precision 0%0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
Interpolated Recall Domain−Specific Multilingual track − Box plot of the Topics of the Experiment Precision averages (%) for individual queries Topic 151 Topic 152 Topic 153 Topic 154 Topic 155 Topic 156 Topic 157 Topic 158 Topic 159 Topic 160 Topic 161 Topic 162 Topic 163 24.52 8.86 10.64 4.30 10.44 1.05 22.74 11.43 0.29 2.03 19.31 0.16 0.41
Topic 164 Topic 165 Topic 166 Topic 167 Topic 168 Topic 169 Topic 170 Topic 171 Topic 172 Topic 173 Topic 174 Topic 175 1.38 0.33 14.87 0.62 5.50 1.94 14.57 19.86 4.23 0.58 8.15 0.13 1 0.8 0.6 0.4 0.2 ce reen 0 if D −0.2 −0.4 −0.6 −0.8 15 t n e m ir xpe E10 e h tf so cp i o fT 5 o r e b m u N 0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Mean Average Precision
Domain−Specific Multilingual track − Distribution of the Topics of the Experiment 00% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Mean Average Precision BERK_MU_EN_T2FB_B_CMBZ −1 151 152 153 154 155 156 157 158 159 160 161 162 163
TopicIdentifie1r64 165 166 167 168 169 170 171 172 173 174 175 100% 90% 80% 70% 60% Domain−Specific Multilingual track − Box plot of the Topics of the Experiment Precision averages (%) for individual queries Topic 151 Topic 152 Topic 153 Topic 154 Topic 155 Topic 156 Topic 157 Topic 158 Topic 159 Topic 160 Topic 161 Topic 162 Topic 163