Appendix C
Results of the Domain Specific Track
Prepared by:
Giorgio Maria Di Nunzio and Nicola Ferro
{dinunzio, ferro}@dei.unipd.it
Department of Information Engineering
University of Padua
Italy
1
2
Introduction
3
4
Results for CLEF 2006 Domain Specific Tracks
The following pages contain the results and graphs for all the experiments that have been officially submitted to the CLEF 2006
campaign for the Domain Specific track.
This document is divided in three main parts:
1. List of submitted experiments
2. Track overview results and graphs
3. Individual experiment results and graphs
1. List of Submitted Experiments
This section gives a listing of all experiments and their characteristics:
Participant: the name of the participant who submitted the experiment.
Country: country of the participant.
Identifier: unique identifier for each experiment.
Task: track/task to which the experiment belongs.
Topic language: language of the topics used to create the experiment (ISO identifiers for language).
Topic fields: identifies the parts of the topics used to create the experiment (T = title, D = Description, N = Narrative).
Query constr.: identifies how the query has been constructed from topic fields (manual/automatic).
Pool: specifies if experiment was used for relevance assessment pooling.
2. Track Overview Results and Graphs
For each track/task graphs and tables are shown in order to compare the experiments.
The graphs and tables contain the following information:
- Mandatory experiments title + description (TD) of at most top five participants
- Interpolated recall vs precision averages plot
- Average precision comparison to median plot
- All experiments
- Average precision box plot
- Average precision Tukey t-test plot
- Mandatory experiments title + description (TD) of at most top five participants
- Document cutoff levels (DCL) vs precision at DCL plot
- R-Precision comparison to median plot
- All experiments
- R-Precision box plot
- R-Precision Tukey t-test plot
- A table with descriptive statistics of performance figures for each topic
3. Individual Experiment Results and Graphs
This section provides the individual results for each official experiment. For each experiment the following tables and graphs are
shown:
- Overall statistics and information
- Interpolated recall vs precision averages plot
- Average precision statistics and box plot
- Average precision comparison to median plot
- Document cutoff levels vs precision at DCL plot
- R-Precision statistics and box plot
- R-Precision comparison to median plot
5
6
List of Submitted Experiments
7
8
Participant Country Experiment ID Task Topic Topic Query Pool
Lang. Fields Construction
berkeley United States BERK_MO_EN_T2FB DS-MONO-EN-CLEF2006 en TD AUTOMATIC yes
tuchemniz Germany TUCMIgirten1 DS-MONO-EN-CLEF2006 en TD AUTOMATIC yes
tuchemniz Germany TUCMIgirten2 DS-MONO-EN-CLEF2006 en TD AUTOMATIC yes
tuchemniz Germany TUCMIgirten3 DS-MONO-EN-CLEF2006 en TD AUTOMATIC yes
tuchemniz Germany TUCMIgirten4 DS-MONO-EN-CLEF2006 en TD AUTOMATIC yes
unine Switzerland UniNEen1 DS-MONO-EN-CLEF2006 en TD AUTOMATIC yes
unine Switzerland UniNEen2 DS-MONO-EN-CLEF2006 en TD AUTOMATIC yes
unine Switzerland UniNEen3 DS-MONO-EN-CLEF2006 en TDN AUTOMATIC yes
berkeley United States BERK_MO_DE_T2FB DS-MONO-DE-CLEF2006 de TD AUTOMATIC yes
hagen Germany FUHggyynbfl500R DS-MONO-DE-CLEF2006 de TD AUTOMATIC yes
hagen Germany FUHggyynbfl500 DS-MONO-DE-CLEF2006 de TD AUTOMATIC yes
hagen Germany FUHggyydbfl500R DS-MONO-DE-CLEF2006 de TD AUTOMATIC yes
hagen Germany FUHggyydbfl500 DS-MONO-DE-CLEF2006 de TD AUTOMATIC yes
hagen Germany FUHggyydbfl102 DS-MONO-DE-CLEF2006 de TD AUTOMATIC yes
tuchemniz Germany TUCMIgirtde2 DS-MONO-DE-CLEF2006 de TD AUTOMATIC yes
tuchemniz Germany TUCMIgirtde3 DS-MONO-DE-CLEF2006 de TD AUTOMATIC yes
tuchemniz Germany TUCMIgirtde1 DS-MONO-DE-CLEF2006 de TD AUTOMATIC yes
tuchemniz Germany TUCMIgirtde4 DS-MONO-DE-CLEF2006 de TD AUTOMATIC yes
unine Switzerland UniNEde1 DS-MONO-DE-CLEF2006 de TD AUTOMATIC yes
unine Switzerland UniNEde2 DS-MONO-DE-CLEF2006 de TD AUTOMATIC yes
unine Switzerland UniNEde3 DS-MONO-DE-CLEF2006 de TDN AUTOMATIC yes
berkeley United States BERK_MO_RU_T2FB DS-MONO-RU-CLEF2006 ru TD AUTOMATIC yes
berkeley United States BERK_BI_RUEN_T2FB_B DS-BILI-X2EN-CLEF2006 de TD AUTOMATIC yes
berkeley United States BERK_BI_DEEN_T2FB_B DS-BILI-X2EN-CLEF2006 de TD AUTOMATIC yes
berkeley United States BERK_BI_DEEN_T2FB_P DS-BILI-X2EN-CLEF2006 de TD AUTOMATIC yes
berkeley United States BERK_BI_ENDE_T2FB_B DS-BILI-X2DE-CLEF2006 en TD AUTOMATIC yes
hagen Germany FUHegpyynl500 DS-BILI-X2DE-CLEF2006 en TD AUTOMATIC yes
hagen Germany FUHegpyydl500 DS-BILI-X2DE-CLEF2006 en TD AUTOMATIC yes
hagen Germany FUHegpyynl102 DS-BILI-X2DE-CLEF2006 en TD AUTOMATIC yes
hagen Germany FUHegpyydl102 DS-BILI-X2DE-CLEF2006 en TD AUTOMATIC yes
hagen Germany FUHegpyydl500R DS-BILI-X2DE-CLEF2006 en TD AUTOMATIC yes
berkeley United States BERK_BI_ENRU_T2FB_P DS-BILI-X2RU-CLEF2006 de TD AUTOMATIC yes
berkeley United States BERK_BI_ENRU_T2FB_B DS-BILI-X2RU-CLEF2006 de TD AUTOMATIC yes
berkeley United States BERK_BI_DERU_T2FB_P DS-BILI-X2RU-CLEF2006 de TD AUTOMATIC yes
berkeley United States BERK_MU_DE_T2FB_B_CMBZ DS-MULTI-CLEF2006 de TD AUTOMATIC no
berkeley United States BERK_MU_EN_T2FB_B_CMBZ DS-MULTI-CLEF2006 de TD AUTOMATIC no
9
10
Track Overview Results and Graphs
11
12
DS-MONO-CLEF2006 Track Overview Results and Graphs DS-MONO-EN-CLEF2006
Domain−Specific Monolingual English track Top 5 Participants − Interpolated Recall vs Average Precision
100%
unine [UniNEen2; MAP 43.03%; Pooled]
berkeley [BERK_MO_EN_T2FB; MAP 41.36%; Pooled]
90%
tuchemniz [TUCMIgirten2; MAP 35.53%; Pooled]
80%
70%
Average Precision
60%
50%
40%
30%
20%
10%
0%
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
Interpolated Recall
Domain−Specific Monolingual English track Top 5 Participants − Comparison to Median Mean Average Precision by Topic (Topics 5
11 to 175)
1
unine [UniNEen2; MAP 43.03%; Pooled]
berkeley [BERK_MO_EN_T2FB; MAP 41.36%; Pooled]
tuchemniz [TUCMIgirten2; MAP 35.53%; Pooled]
0.8
0.6
0.4
0.2
Difference
0
−0.2
−0.4
−0.6
−0.8
−1
151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175
Topic Identifier
13
DS-MONO-CLEF2006 Track Overview Results and Graphs DS-MONO-EN-CLEF2006
Domain−Specific Monolingual English track − Box Plot of the Topics
UniNEen3 [MAP 45.76%; Pooled]
UniNEen2 [MAP 43.03%; Pooled]
UniNEen1 [MAP 42.92%; Pooled]
BERK_MO_EN_T2FB [MAP 41.36%; Pooled]
Experiments
TUCMIgirten2 [MAP 35.53%; Pooled]
TUCMIgirten4 [MAP 35.38%; Pooled]
TUCMIgirten1 [MAP 35.10%; Pooled]
TUCMIgirten3 [MAP 34.50%; Pooled]
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
Mean Average Precision
14
DS-MONO-CLEF2006 Track Overview Results and Graphs DS-MONO-EN-CLEF2006
Domain−Specific Monolingual English track − Tukey T test with "top group" highlighted
UniNEen3
UniNEen2
UniNEen1
Experiments
BERK_MO_EN_T2FB
TUCMIgirten2
TUCMIgirten4
TUCMIgirten1
TUCMIgirten3
0.55 0.6 0.65 0.7 0.75 0.8
arcsin(sqrt(Mean average precision))
15
DS-MONO-CLEF2006 Track Overview Results and Graphs DS-MONO-EN-CLEF2006
Domain−Specific Monolingual English track Top 5 Participants − Retrieved documents vs Precision
100%
unine [UniNEen2; R−Prec 45.22%; Pooled]
berkeley [BERK_MO_EN_T2FB; R−Prec 44.19%; Pooled]
90%
tuchemniz [TUCMIgirten2; R−Prec 39.33%; Pooled]
80%
70%
60%
R−Precision
50%
40%
30%
20%
10%
0%
5 10 15 20 30 100 200 500 1000
Retrieved Documents (logarithmic scale)
Domain−Specific Monolingual English track Top 5 Participants − Comparison to Median R−Precision by Topic (Topics 151 to 175)
1
unine [UniNEen2; R−Prec 45.22%; Pooled]
berkeley [BERK_MO_EN_T2FB; R−Prec 44.19%; Pooled]
tuchemniz [TUCMIgirten2; R−Prec 39.33%; Pooled]
0.8
0.6
0.4
0.2
Difference
0
−0.2
−0.4
−0.6
−0.8
−1
151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175
Topic Identifier
16
DS-MONO-CLEF2006 Track Overview Results and Graphs DS-MONO-EN-CLEF2006
Domain−Specific Monolingual English track − Box Plot of the Topics
UniNEen3 [R−Prec 47.71%; Pooled]
UniNEen2 [R−Prec 45.22%; Pooled]
UniNEen1 [R−Prec 44.80%; Pooled]
BERK_MO_EN_T2FB [R−Prec 44.19%; Pooled]
Experiments
TUCMIgirten4 [R−Prec 39.34%; Pooled]
TUCMIgirten2 [R−Prec 39.33%; Pooled]
TUCMIgirten1 [R−Prec 38.96%; Pooled]
TUCMIgirten3 [R−Prec 38.52%; Pooled]
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
R−Precision
17
DS-MONO-CLEF2006 Track Overview Results and Graphs DS-MONO-EN-CLEF2006
Domain−Specific Monolingual English track − Tukey T test with "top group" highlighted
UniNEen3
UniNEen2
UniNEen1
Experiments
BERK_MO_EN_T2FB
TUCMIgirten4
TUCMIgirten2
TUCMIgirten1
TUCMIgirten3
0.62 0.64 0.66 0.68 0.7 0.72 0.74 0.76 0.78 0.8
arcsin(sqrt(R Precision))
18
DS-MONO-CLEF2006 Track Overview Results and Graphs DS-MONO-EN-CLEF2006
Average Precision R-Precision
Topic Minimum 1st Q. Median 3rd Q. Maximum Mean Std Minimum 1st Q. Median 3rd Q. Maximum Mean Std
151 0.5301 0.5385 0.6126 0.6749 0.7277 0.6137 0.0799 0.5255 0.5392 0.5706 0.6039 0.6784 0.5789 0.0509
152 0.4910 0.5540 0.5794 0.5933 0.5982 0.5678 0.0374 0.5308 0.5557 0.5718 0.5821 0.5953 0.5682 0.0205
153 0.0240 0.2221 0.4218 0.4419 0.4660 0.3327 0.1800 0.1246 0.2889 0.4360 0.4550 0.4844 0.3711 0.1398
154 0.7148 0.7193 0.7313 0.8031 0.8169 0.7548 0.0446 0.6059 0.6208 0.6716 0.7331 0.7458 0.6753 0.0581
155 0.4624 0.4852 0.4936 0.5475 0.6294 0.5180 0.0545 0.5033 0.5148 0.5214 0.5411 0.5757 0.5292 0.0232
156 0.4629 0.4982 0.5156 0.5251 0.5368 0.5097 0.0235 0.4811 0.4953 0.5000 0.5377 0.5943 0.5177 0.0372
157 0.1361 0.1995 0.3646 0.5440 0.5546 0.3634 0.1909 0.2211 0.2895 0.4053 0.5211 0.5368 0.3987 0.1358
158 0.1369 0.1851 0.2706 0.3241 0.3407 0.2547 0.0780 0.2232 0.2679 0.3438 0.4420 0.4643 0.3493 0.0975
159 0.7024 0.7750 0.7854 0.8179 0.8233 0.7853 0.0392 0.7090 0.7612 0.7724 0.7836 0.7910 0.7668 0.0264
160 0.1178 0.1495 0.1803 0.2441 0.2979 0.1954 0.0649 0.2264 0.2566 0.3170 0.3642 0.3887 0.3113 0.0609
161 0.3272 0.3741 0.5096 0.5139 0.5182 0.4551 0.0818 0.4042 0.4278 0.4869 0.4934 0.5354 0.4695 0.0448
162 0.2590 0.2810 0.2869 0.4999 0.5108 0.3632 0.1167 0.3514 0.3802 0.3978 0.4968 0.5144 0.4269 0.0646
163 0.1435 0.1522 0.2294 0.5686 0.5885 0.3291 0.2096 0.2535 0.2570 0.3275 0.5915 0.6268 0.4040 0.1719
164 0.0251 0.0268 0.0383 0.0565 0.1069 0.0469 0.0279 0.0526 0.0526 0.0921 0.1250 0.1447 0.0921 0.0379
165 0.5158 0.5372 0.6682 0.8141 0.8538 0.6761 0.1505 0.4833 0.5000 0.6083 0.7250 0.7667 0.6146 0.1242
166 0.4457 0.4667 0.5062 0.6424 0.6762 0.5441 0.0954 0.4805 0.4922 0.5137 0.6094 0.6328 0.5430 0.0639
167 0.0604 0.0674 0.0805 0.1137 0.2134 0.0996 0.0529 0.0870 0.1522 0.1957 0.2174 0.2174 0.1793 0.0490
168 0.0949 0.1140 0.1231 0.1344 0.1425 0.1226 0.0157 0.1515 0.1667 0.1919 0.1919 0.2626 0.1894 0.0345
169 0.0600 0.0849 0.1160 0.1641 0.1795 0.1212 0.0452 0.1190 0.1429 0.1429 0.1667 0.2381 0.1577 0.0359
170 0.2454 0.2501 0.2846 0.3406 0.3686 0.2956 0.0511 0.3182 0.3182 0.4091 0.5000 0.5000 0.4091 0.0876
171 0.4122 0.4224 0.4444 0.5485 0.5865 0.4787 0.0707 0.4670 0.4808 0.5000 0.5824 0.5989 0.5240 0.0546
172 0.0792 0.0873 0.0996 0.1222 0.1762 0.1092 0.0326 0.1000 0.1125 0.1375 0.1625 0.2000 0.1406 0.0352
173 0.3770 0.3792 0.5106 0.6084 0.6391 0.5015 0.1217 0.4256 0.4256 0.5248 0.5992 0.6033 0.5160 0.0869
174 0.5511 0.5624 0.5867 0.6135 0.6219 0.5873 0.0292 0.5731 0.5760 0.6053 0.6404 0.6667 0.6104 0.0381
175 0.0442 0.1100 0.1349 0.2742 0.3045 0.1734 0.0978 0.0755 0.1887 0.2170 0.2925 0.3019 0.2217 0.0766
ALL 0.3450 0.3524 0.3845 0.4297 0.4576 0.3920 0.0452 0.3852 0.3915 0.4177 0.4501 0.4771 0.4226 0.0360
19
20
DS-MONO-CLEF2006 Track Overview Results and Graphs DS-MONO-DE-CLEF2006
Domain−Specific Monolingual German track Top 5 Participants − Interpolated Recall vs Average Precision
100%
tuchemniz [TUCMIgirtde4; MAP 54.54%; Pooled]
unine [UniNEde2; MAP 50.51%; Pooled]
90% berkeley [BERK_MO_DE_T2FB; MAP 39.17%; Pooled]
hagen [FUHggyynbfl500R; MAP 35.39%; Pooled]
80%
70%
Average Precision
60%
50%
40%
30%
20%
10%
0%
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
Interpolated Recall
Domain−Specific Monolingual German track Top 5 Participants − Comparison to Median Mean Average Precision by Topic (Topics 15
1 to 175)
1
tuchemniz [TUCMIgirtde4; MAP 54.54%; Pooled]
unine [UniNEde2; MAP 50.51%; Pooled]
berkeley [BERK_MO_DE_T2FB; MAP 39.17%; Pooled]
hagen [FUHggyynbfl500R; MAP 35.39%; Pooled]
0.8
0.6
0.4
0.2
Difference
0
−0.2
−0.4
−0.6
−0.8
−1
151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175
Topic Identifier
21
DS-MONO-CLEF2006 Track Overview Results and Graphs DS-MONO-DE-CLEF2006
Domain−Specific Monolingual German track − Box Plot of the Topics
TUCMIgirtde4 [MAP 54.54%; Pooled]
TUCMIgirtde2 [MAP 52.02%; Pooled]
UniNEde3 [MAP 51.59%; Pooled]
TUCMIgirtde3 [MAP 51.36%; Pooled]
TUCMIgirtde1 [MAP 51.22%; Pooled]
UniNEde2 [MAP 50.51%; Pooled]
Experiments
UniNEde1 [MAP 50.15%; Pooled]
BERK_MO_DE_T2FB [MAP 39.17%; Pooled]
FUHggyynbfl500R [MAP 35.39%; Pooled]
FUHggyynbfl500 [MAP 35.25%; Pooled]
FUHggyydbfl500 [MAP 32.05%; Pooled]
FUHggyydbfl500R [MAP 31.79%; Pooled]
FUHggyydbfl102 [MAP 27.77%; Pooled]
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
Mean Average Precision
22
DS-MONO-CLEF2006 Track Overview Results and Graphs DS-MONO-DE-CLEF2006
Domain−Specific Monolingual German track − Tukey T test with "top group" highlighted
TUCMIgirtde4
UniNEde3
TUCMIgirtde2
TUCMIgirtde3
TUCMIgirtde1
UniNEde2
Experiments
UniNEde1
BERK_MO_DE_T2FB
FUHggyynbfl500R
FUHggyynbfl500
FUHggyydbfl500
FUHggyydbfl500R
FUHggyydbfl102
0.4 0.5 0.6 0.7 0.8 0.9 1
arcsin(sqrt(Mean average precision))
23
DS-MONO-CLEF2006 Track Overview Results and Graphs DS-MONO-DE-CLEF2006
Domain−Specific Monolingual German track Top 5 Participants − Retrieved documents vs Precision
100%
tuchemniz [TUCMIgirtde4; R−Prec 53.68%; Pooled]
unine [UniNEde2; R−Prec 49.89%; Pooled]
90% berkeley [BERK_MO_DE_T2FB; R−Prec 39.05%; Pooled]
hagen [FUHggyynbfl500R; R−Prec 39.88%; Pooled]
80%
70%
60%
R−Precision
50%
40%
30%
20%
10%
0%
5 10 15 20 30 100 200 500 1000
Retrieved Documents (logarithmic scale)
Domain−Specific Monolingual German track Top 5 Participants − Comparison to Median R−Precision by Topic (Topics 151 to 175)
1
tuchemniz [TUCMIgirtde4; R−Prec 53.68%; Pooled]
unine [UniNEde2; R−Prec 49.89%; Pooled]
berkeley [BERK_MO_DE_T2FB; R−Prec 39.05%; Pooled]
hagen [FUHggyynbfl500R; R−Prec 39.88%; Pooled]
0.8
0.6
0.4
0.2
Difference
0
−0.2
−0.4
−0.6
−0.8
−1
151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175
Topic Identifier
24
DS-MONO-CLEF2006 Track Overview Results and Graphs DS-MONO-DE-CLEF2006
Domain−Specific Monolingual German track − Box Plot of the Topics
TUCMIgirtde4 [R−Prec 53.68%; Pooled]
TUCMIgirtde2 [R−Prec 51.63%; Pooled]
TUCMIgirtde3 [R−Prec 51.18%; Pooled]
TUCMIgirtde1 [R−Prec 50.97%; Pooled]
UniNEde3 [R−Prec 50.21%; Pooled]
UniNEde1 [R−Prec 49.91%; Pooled]
Experiments
UniNEde2 [R−Prec 49.89%; Pooled]
FUHggyynbfl500R [R−Prec 39.88%; Pooled]
FUHggyynbfl500 [R−Prec 39.41%; Pooled]
BERK_MO_DE_T2FB [R−Prec 39.05%; Pooled]
FUHggyydbfl500 [R−Prec 36.31%; Pooled]
FUHggyydbfl500R [R−Prec 36.15%; Pooled]
FUHggyydbfl102 [R−Prec 33.33%; Pooled]
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
R−Precision
25
DS-MONO-CLEF2006 Track Overview Results and Graphs DS-MONO-DE-CLEF2006
Domain−Specific Monolingual German track − Tukey T test with "top group" highlighted
TUCMIgirtde4
TUCMIgirtde2
TUCMIgirtde3
TUCMIgirtde1
UniNEde3
UniNEde1
Experiments
UniNEde2
FUHggyynbfl500R
FUHggyynbfl500
BERK_MO_DE_T2FB
FUHggyydbfl500R
FUHggyydbfl500
FUHggyydbfl102
0.5 0.55 0.6 0.65 0.7 0.75 0.8 0.85 0.9
arcsin(sqrt(R Precision))
26
DS-MONO-CLEF2006 Track Overview Results and Graphs DS-MONO-DE-CLEF2006
Average Precision R-Precision
Topic Minimum 1st Q. Median 3rd Q. Maximum Mean Std Minimum 1st Q. Median 3rd Q. Maximum Mean Std
151 0.5696 0.5942 0.7077 0.7414 0.7793 0.6869 0.0789 0.5433 0.5738 0.6614 0.6860 0.7283 0.6405 0.0618
152 0.0000 0.0940 0.5175 0.7677 0.8219 0.4786 0.3148 0.0000 0.1514 0.5144 0.7212 0.7644 0.4704 0.2760
153 0.0547 0.3590 0.7169 0.7767 0.7991 0.5688 0.2496 0.0800 0.3933 0.7067 0.7333 0.7333 0.5569 0.2109
154 0.5048 0.6241 0.6928 0.8218 0.9124 0.7172 0.1323 0.5766 0.5922 0.6095 0.7245 0.8285 0.6628 0.0962
155 0.2172 0.2487 0.3405 0.4714 0.7024 0.3729 0.1409 0.2694 0.3264 0.4249 0.4832 0.6373 0.4205 0.1060
156 0.0148 0.1671 0.4835 0.6820 0.7088 0.4065 0.2504 0.0968 0.2204 0.4409 0.6183 0.6774 0.3970 0.2029
157 0.1087 0.1360 0.4364 0.5551 0.7954 0.3862 0.2357 0.1692 0.2231 0.4308 0.5615 0.6923 0.3964 0.1793
158 0.0472 0.1123 0.4022 0.4657 0.5731 0.3088 0.1972 0.1348 0.2163 0.4719 0.5225 0.5843 0.3760 0.1764
159 0.4679 0.5499 0.6237 0.6648 0.6916 0.6017 0.0793 0.4907 0.5625 0.5926 0.6134 0.6667 0.5833 0.0517
160 0.0000 0.5538 0.6059 0.6272 0.7348 0.5577 0.1755 0.0000 0.5608 0.5764 0.5920 0.6736 0.5342 0.1650
161 0.0667 0.3112 0.6549 0.7237 0.7678 0.5097 0.2677 0.1329 0.3884 0.6076 0.6748 0.6962 0.5131 0.2025
162 0.1412 0.2328 0.2476 0.5252 0.6260 0.3533 0.1670 0.2198 0.3233 0.4009 0.5453 0.6164 0.4294 0.1250
163 0.0816 0.1436 0.2428 0.2999 0.3423 0.2247 0.0856 0.1712 0.2646 0.3230 0.4047 0.4202 0.3161 0.0787
164 0.0028 0.0350 0.0551 0.1230 0.1648 0.0728 0.0549 0.0192 0.0625 0.0769 0.1298 0.1923 0.0962 0.0608
165 0.2164 0.3335 0.4068 0.8478 0.8938 0.5546 0.2677 0.2192 0.3938 0.4384 0.8116 0.8493 0.5616 0.2241
166 0.4046 0.5228 0.6009 0.6500 0.7680 0.5956 0.1030 0.4274 0.5343 0.5618 0.6257 0.7231 0.5769 0.0822
167 0.0538 0.0838 0.2680 0.4880 0.5154 0.2735 0.1879 0.0889 0.1333 0.3111 0.5000 0.5778 0.3162 0.1778
168 0.2078 0.3282 0.4517 0.4860 0.6418 0.4196 0.1129 0.3537 0.4320 0.4490 0.4745 0.5918 0.4589 0.0570
169 0.0661 0.1381 0.1874 0.2207 0.2568 0.1741 0.0609 0.0909 0.0909 0.2727 0.2727 0.3636 0.2238 0.1024
170 0.4124 0.5044 0.5461 0.5894 0.6154 0.5399 0.0578 0.4615 0.5128 0.5385 0.5641 0.5641 0.5325 0.0334
171 0.3133 0.3959 0.4405 0.5724 0.6041 0.4683 0.1001 0.3788 0.4268 0.4545 0.5480 0.5909 0.4821 0.0730
172 0.0701 0.0781 0.2230 0.4543 0.5697 0.2504 0.1930 0.0909 0.1364 0.2273 0.4545 0.5000 0.2657 0.1506
173 0.4795 0.5951 0.6339 0.6770 0.7176 0.6278 0.0637 0.5119 0.5982 0.6230 0.6369 0.6786 0.6187 0.0442
174 0.2557 0.4721 0.4820 0.5281 0.6214 0.4886 0.0865 0.3523 0.5270 0.5398 0.5639 0.6932 0.5424 0.0729
175 0.0708 0.1065 0.1810 0.2701 0.3141 0.1850 0.0833 0.1250 0.1641 0.2188 0.2539 0.3438 0.2127 0.0620
ALL 0.2777 0.3445 0.5015 0.5142 0.5454 0.4329 0.0976 0.3333 0.3836 0.4989 0.5102 0.5368 0.4474 0.0736
27
28
DS-MONO-CLEF2006 Track Overview Results and Graphs DS-MONO-RU-CLEF2006
Domain−Specific Monolingual Russian track Top 5 Participants − Interpolated Recall vs Average Precision
100%
berkeley [BERK_MO_RU_T2FB; MAP 25.42%; Pooled]
90%
80%
70%
Average Precision
60%
50%
40%
30%
20%
10%
0%
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
Interpolated Recall
Domain−Specific Monolingual Russian track Top 5 Participants − Comparison to Median Mean Average Precision by Topic (Topics 5
11 to 175)
1
berkeley [BERK_MO_RU_T2FB; MAP 25.42%; Pooled]
0.8
0.6
0.4
0.2
Difference
0
−0.2
−0.4
−0.6
−0.8
−1
151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175
Topic Identifier
29
DS-MONO-CLEF2006 Track Overview Results and Graphs DS-MONO-RU-CLEF2006
Domain−Specific Monolingual Russian track − Box Plot of the Topics
Experiments
BERK_MO_RU_T2FB [MAP 25.42%; Pooled]
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
Mean Average Precision
30
DS-MONO-CLEF2006 Track Overview Results and Graphs DS-MONO-RU-CLEF2006
31
DS-MONO-CLEF2006 Track Overview Results and Graphs DS-MONO-RU-CLEF2006
32
DS-MONO-CLEF2006 Track Overview Results and Graphs DS-MONO-RU-CLEF2006
Domain−Specific Monolingual Russian track Top 5 Participants − Retrieved documents vs Precision
100%
berkeley [BERK_MO_RU_T2FB; R−Prec 24.85%; Pooled]
90%
80%
70%
60%
R−Precision
50%
40%
30%
20%
10%
0%
5 10 15 20 30 100 200 500 1000
Retrieved Documents (logarithmic scale)
Domain−Specific Monolingual Russian track Top 5 Participants − Comparison to Median R−Precision by Topic (Topics 151 to 175)
1
berkeley [BERK_MO_RU_T2FB; R−Prec 24.85%; Pooled]
0.8
0.6
0.4
0.2
Difference
0
−0.2
−0.4
−0.6
−0.8
−1
151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175
Topic Identifier
33
DS-MONO-CLEF2006 Track Overview Results and Graphs DS-MONO-RU-CLEF2006
Domain−Specific Monolingual Russian track − Box Plot of the Topics
Experiments
BERK_MO_RU_T2FB [R−Prec 24.85%; Pooled]
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
R−Precision
34
DS-MONO-CLEF2006 Track Overview Results and Graphs DS-MONO-RU-CLEF2006
35
DS-MONO-CLEF2006 Track Overview Results and Graphs DS-MONO-RU-CLEF2006
36
DS-MONO-CLEF2006 Track Overview Results and Graphs DS-MONO-RU-CLEF2006
Average Precision R-Precision
Topic Minimum 1st Q. Median 3rd Q. Maximum Mean Std Minimum 1st Q. Median 3rd Q. Maximum Mean Std
151 0.1089 0.1089 0.1089 0.1089 0.1089 0.1089 0.0000 0.1875 0.1875 0.1875 0.1875 0.1875 0.1875 0.0000
152 0.5846 0.5846 0.5846 0.5846 0.5846 0.5846 0.0000 0.5581 0.5581 0.5581 0.5581 0.5581 0.5581 0.0000
153 0.1418 0.1418 0.1418 0.1418 0.1418 0.1418 0.0000 0.1739 0.1739 0.1739 0.1739 0.1739 0.1739 0.0000
154 0.0617 0.0617 0.0617 0.0617 0.0617 0.0617 0.0000 0.1212 0.1212 0.1212 0.1212 0.1212 0.1212 0.0000
155 0.2149 0.2149 0.2149 0.2149 0.2149 0.2149 0.0000 0.1351 0.1351 0.1351 0.1351 0.1351 0.1351 0.0000
156 0.2018 0.2018 0.2018 0.2018 0.2018 0.2018 0.0000 0.2500 0.2500 0.2500 0.2500 0.2500 0.2500 0.0000
157 0.5588 0.5588 0.5588 0.5588 0.5588 0.5588 0.0000 0.5000 0.5000 0.5000 0.5000 0.5000 0.5000 0.0000
158 0.1280 0.1280 0.1280 0.1280 0.1280 0.1280 0.0000 0.1429 0.1429 0.1429 0.1429 0.1429 0.1429 0.0000
159 0.5700 0.5700 0.5700 0.5700 0.5700 0.5700 0.0000 0.5238 0.5238 0.5238 0.5238 0.5238 0.5238 0.0000
160 0.8015 0.8015 0.8015 0.8015 0.8015 0.8015 0.0000 0.8198 0.8198 0.8198 0.8198 0.8198 0.8198 0.0000
161 0.3439 0.3439 0.3439 0.3439 0.3439 0.3439 0.0000 0.4091 0.4091 0.4091 0.4091 0.4091 0.4091 0.0000
162 0.3987 0.3987 0.3987 0.3987 0.3987 0.3987 0.0000 0.3276 0.3276 0.3276 0.3276 0.3276 0.3276 0.0000
163 0.8590 0.8590 0.8590 0.8590 0.8590 0.8590 0.0000 0.8584 0.8584 0.8584 0.8584 0.8584 0.8584 0.0000
164 0.0528 0.0528 0.0528 0.0528 0.0528 0.0528 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000
165 0.1075 0.1075 0.1075 0.1075 0.1075 0.1075 0.0000 0.1111 0.1111 0.1111 0.1111 0.1111 0.1111 0.0000
166 0.0594 0.0594 0.0594 0.0594 0.0594 0.0594 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000
167 0.0910 0.0910 0.0910 0.0910 0.0910 0.0910 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000
168 0.2619 0.2619 0.2619 0.2619 0.2619 0.2619 0.0000 0.2500 0.2500 0.2500 0.2500 0.2500 0.2500 0.0000
169 0.0200 0.0200 0.0200 0.0200 0.0200 0.0200 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000
170 0.0747 0.0747 0.0747 0.0747 0.0747 0.0747 0.0000 0.0435 0.0435 0.0435 0.0435 0.0435 0.0435 0.0000
171 0.5031 0.5031 0.5031 0.5031 0.5031 0.5031 0.0000 0.5000 0.5000 0.5000 0.5000 0.5000 0.5000 0.0000
172 0.0030 0.0030 0.0030 0.0030 0.0030 0.0030 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000
173 0.2085 0.2085 0.2085 0.2085 0.2085 0.2085 0.0000 0.3000 0.3000 0.3000 0.3000 0.3000 0.3000 0.0000
174 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000
175 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000
ALL 0.2542 0.2542 0.2542 0.2542 0.2542 0.2542 0.0000 0.2485 0.2485 0.2485 0.2485 0.2485 0.2485 0.0000
37
38
DS-BILI-CLEF2006 Track Overview Results and Graphs DS-BILI-X2EN-CLEF2006
Domain−Specific Bilingual English track Top 5 Participants − Interpolated Recall vs Average Precision
100%
berkeley [BERK_BI_DEEN_T2FB_B; MAP 33.01%; Pooled]
90%
80%
70%
Average Precision
60%
50%
40%
30%
20%
10%
0%
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
Interpolated Recall
Domain−Specific Bilingual English track Top 5 Participants − Comparison to Median Mean Average Precision by Topic (Topics 151to 175)
1
berkeley [BERK_BI_DEEN_T2FB_B; MAP 33.01%; Pooled]
0.8
0.6
0.4
0.2
Difference
0
−0.2
−0.4
−0.6
−0.8
−1
151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175
Topic Identifier
39
DS-BILI-CLEF2006 Track Overview Results and Graphs DS-BILI-X2EN-CLEF2006
Domain−Specific Bilingual English track − Box Plot of the Topics
BERK_BI_DEEN_T2FB_B [MAP 33.01%; Pooled]
Experiments
BERK_BI_RUEN_T2FB_B [MAP 32.28%; Pooled]
BERK_BI_DEEN_T2FB_P [MAP 31.76%; Pooled]
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
Mean Average Precision
40
DS-BILI-CLEF2006 Track Overview Results and Graphs DS-BILI-X2EN-CLEF2006
Domain−Specific Bilingual English track − Tukey T test with "top group" highlighted
BERK_BI_DEEN_T2FB_B
Experiments
BERK_BI_RUEN_T2FB_B
BERK_BI_DEEN_T2FB_P
0.24 0.26 0.28 0.3 0.32 0.34 0.36 0.38 0.4 0.42
Mean average precision
41
DS-BILI-CLEF2006 Track Overview Results and Graphs DS-BILI-X2EN-CLEF2006
Domain−Specific Bilingual English track Top 5 Participants − Retrieved documents vs Precision
100%
berkeley [BERK_BI_DEEN_T2FB_B; R−Prec 37.46%; Pooled]
90%
80%
70%
60%
R−Precision
50%
40%
30%
20%
10%
0%
5 10 15 20 30 100 200 500 1000
Retrieved Documents (logarithmic scale)
Domain−Specific Bilingual English track Top 5 Participants − Comparison to Median R−Precision by Topic (Topics 151 to 175)
1
berkeley [BERK_BI_DEEN_T2FB_B; R−Prec 37.46%; Pooled]
0.8
0.6
0.4
0.2
Difference
0
−0.2
−0.4
−0.6
−0.8
−1
151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175
Topic Identifier
42
DS-BILI-CLEF2006 Track Overview Results and Graphs DS-BILI-X2EN-CLEF2006
Domain−Specific Bilingual English track − Box Plot of the Topics
BERK_BI_RUEN_T2FB_B [R−Prec 37.63%; Pooled]
Experiments
BERK_BI_DEEN_T2FB_B [R−Prec 37.46%; Pooled]
BERK_BI_DEEN_T2FB_P [R−Prec 36.19%; Pooled]
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
R−Precision
43
DS-BILI-CLEF2006 Track Overview Results and Graphs DS-BILI-X2EN-CLEF2006
Domain−Specific Bilingual English track − Tukey T test with "top group" highlighted
BERK_BI_RUEN_T2FB_B
Experiments
BERK_BI_DEEN_T2FB_B
BERK_BI_DEEN_T2FB_P
0.3 0.32 0.34 0.36 0.38 0.4 0.42 0.44
R Precision
44
DS-BILI-CLEF2006 Track Overview Results and Graphs DS-BILI-X2EN-CLEF2006
Average Precision R-Precision
Topic Minimum 1st Q. Median 3rd Q. Maximum Mean Std Minimum 1st Q. Median 3rd Q. Maximum Mean Std
151 0.6477 0.6508 0.6601 0.7031 0.7174 0.6751 0.0372 0.6353 0.6353 0.6353 0.6618 0.6706 0.6471 0.0204
152 0.1687 0.1993 0.2911 0.5041 0.5751 0.3450 0.2085 0.2962 0.3218 0.3988 0.5242 0.5660 0.4203 0.1362
153 0.0241 0.1323 0.4568 0.4568 0.4568 0.3126 0.2498 0.1003 0.1920 0.4671 0.4671 0.4671 0.3449 0.2118
154 0.5225 0.5338 0.5677 0.6876 0.7275 0.6059 0.1077 0.5254 0.5328 0.5551 0.6568 0.6907 0.5904 0.0881
155 0.0006 0.0063 0.0235 0.2904 0.3793 0.1345 0.2123 0.0164 0.0321 0.0789 0.3405 0.4276 0.1743 0.2216
156 0.0489 0.0678 0.1244 0.3168 0.3809 0.1847 0.1741 0.1604 0.1769 0.2264 0.4033 0.4623 0.2830 0.1587
157 0.0241 0.1219 0.4154 0.5198 0.5546 0.3314 0.2751 0.0526 0.1526 0.4526 0.5079 0.5263 0.3439 0.2549
158 0.0126 0.0193 0.0395 0.0699 0.0800 0.0440 0.0340 0.0357 0.0558 0.1161 0.1830 0.2054 0.1190 0.0849
159 0.6929 0.6929 0.6929 0.6986 0.7005 0.6954 0.0044 0.7015 0.7015 0.7015 0.7071 0.7090 0.7040 0.0043
160 0.1996 0.2538 0.4164 0.4557 0.4689 0.3616 0.1428 0.3208 0.3538 0.4528 0.4726 0.4792 0.4176 0.0849
161 0.2842 0.2973 0.3366 0.4395 0.4739 0.3649 0.0980 0.3596 0.3812 0.4462 0.4856 0.4987 0.4348 0.0702
162 0.0457 0.0555 0.0848 0.3554 0.4456 0.1921 0.2205 0.1086 0.1230 0.1661 0.3986 0.4760 0.2503 0.1976
163 0.1332 0.1358 0.1436 0.1545 0.1581 0.1450 0.0125 0.2254 0.2306 0.2465 0.2465 0.2465 0.2394 0.0122
164 0.0608 0.1175 0.2878 0.2878 0.2878 0.2121 0.1311 0.1711 0.2237 0.3816 0.3816 0.3816 0.3114 0.1215
165 0.3054 0.3763 0.5888 0.7585 0.8151 0.5698 0.2554 0.3667 0.4333 0.6333 0.7083 0.7333 0.5778 0.1895
166 0.2056 0.2749 0.4829 0.5212 0.5340 0.4075 0.1767 0.3047 0.3564 0.5117 0.5234 0.5273 0.4479 0.1243
167 0.1364 0.1523 0.2000 0.2302 0.2402 0.1922 0.0524 0.2174 0.2174 0.2174 0.3152 0.3478 0.2609 0.0753
168 0.0068 0.0633 0.2328 0.4731 0.5532 0.2643 0.2746 0.0101 0.0934 0.3434 0.5404 0.6061 0.3199 0.2987
169 0.1620 0.1636 0.1683 0.1884 0.1951 0.1751 0.0176 0.1905 0.1905 0.1905 0.2262 0.2381 0.2063 0.0275
170 0.2490 0.2730 0.3449 0.3869 0.4009 0.3316 0.0768 0.4091 0.4091 0.4091 0.4773 0.5000 0.4394 0.0525
171 0.1119 0.2069 0.4919 0.4919 0.4919 0.3652 0.2194 0.1813 0.2734 0.5495 0.5495 0.5495 0.4267 0.2125
172 0.1762 0.1874 0.2209 0.2297 0.2326 0.2099 0.0298 0.1750 0.1875 0.2250 0.3000 0.3250 0.2417 0.0764
173 0.0025 0.1061 0.4167 0.4775 0.4977 0.3056 0.2656 0.0289 0.1467 0.5000 0.5248 0.5331 0.3540 0.2820
174 0.3745 0.4169 0.5439 0.6049 0.6253 0.5146 0.1279 0.4211 0.4591 0.5731 0.6170 0.6316 0.5419 0.1087
175 0.1138 0.1138 0.1138 0.1910 0.2167 0.1481 0.0594 0.1132 0.1132 0.1132 0.2547 0.3019 0.1761 0.1089
ALL 0.3176 0.3189 0.3228 0.3283 0.3301 0.3235 0.0063 0.3619 0.3650 0.3746 0.3759 0.3763 0.3709 0.0079
45
46
DS-BILI-CLEF2006 Track Overview Results and Graphs DS-BILI-X2DE-CLEF2006
Domain−Specific Bilingual German track Top 5 Participants − Interpolated Recall vs Average Precision
100%
hagen [FUHegpyynl500; MAP 24.48%; Pooled]
berkeley [BERK_BI_ENDE_T2FB_B; MAP 23.66%; Pooled]
90%
80%
70%
Average Precision
60%
50%
40%
30%
20%
10%
0%
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
Interpolated Recall
Domain−Specific Bilingual German track Top 5 Participants − Comparison to Median Mean Average Precision by Topic (Topics 151to 175)
1
hagen [FUHegpyynl500; MAP 24.48%; Pooled]
berkeley [BERK_BI_ENDE_T2FB_B; MAP 23.66%; Pooled]
0.8
0.6
0.4
0.2
Difference
0
−0.2
−0.4
−0.6
−0.8
−1
151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175
Topic Identifier
47
DS-BILI-CLEF2006 Track Overview Results and Graphs DS-BILI-X2DE-CLEF2006
Domain−Specific Bilingual German track − Box Plot of the Topics
FUHegpyynl500 [MAP 24.48%; Pooled]
BERK_BI_ENDE_T2FB_B [MAP 23.66%; Pooled]
FUHegpyydl500 [MAP 21.90%; Pooled]
Experiments
FUHegpyydl500R [MAP 21.80%; Pooled]
FUHegpyynl102 [MAP 19.80%; Pooled]
FUHegpyydl102 [MAP 17.68%; Pooled]
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
Mean Average Precision
48
DS-BILI-CLEF2006 Track Overview Results and Graphs DS-BILI-X2DE-CLEF2006
Domain−Specific Bilingual German track − Tukey T test with "top group" highlighted
FUHegpyynl500
FUHegpyydl500R
FUHegpyydl500
Experiments
BERK_BI_ENDE_T2FB_B
FUHegpyynl102
FUHegpyydl102
0.35 0.4 0.45 0.5 0.55 0.6 0.65 0.7
arcsin(sqrt(Mean average precision))
49
DS-BILI-CLEF2006 Track Overview Results and Graphs DS-BILI-X2DE-CLEF2006
Domain−Specific Bilingual German track Top 5 Participants − Retrieved documents vs Precision
100%
hagen [FUHegpyynl500; R−Prec 30.38%; Pooled]
berkeley [BERK_BI_ENDE_T2FB_B; R−Prec 26.63%; Pooled]
90%
80%
70%
60%
R−Precision
50%
40%
30%
20%
10%
0%
5 10 15 20 30 100 200 500 1000
Retrieved Documents (logarithmic scale)
Domain−Specific Bilingual German track Top 5 Participants − Comparison to Median R−Precision by Topic (Topics 151 to 175)
1
hagen [FUHegpyynl500; R−Prec 30.38%; Pooled]
berkeley [BERK_BI_ENDE_T2FB_B; R−Prec 26.63%; Pooled]
0.8
0.6
0.4
0.2
Difference
0
−0.2
−0.4
−0.6
−0.8
−1
151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175
Topic Identifier
50
DS-BILI-CLEF2006 Track Overview Results and Graphs DS-BILI-X2DE-CLEF2006
Domain−Specific Bilingual German track − Box Plot of the Topics
FUHegpyynl500 [R−Prec 30.38%; Pooled]
FUHegpyydl500 [R−Prec 27.81%; Pooled]
FUHegpyydl500R [R−Prec 27.77%; Pooled]
Experiments
BERK_BI_ENDE_T2FB_B [R−Prec 26.63%; Pooled]
FUHegpyynl102 [R−Prec 25.52%; Pooled]
FUHegpyydl102 [R−Prec 23.26%; Pooled]
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
R−Precision
51
DS-BILI-CLEF2006 Track Overview Results and Graphs DS-BILI-X2DE-CLEF2006
Domain−Specific Bilingual German track − Tukey T test with "top group" highlighted
FUHegpyynl500
FUHegpyydl500R
FUHegpyydl500
Experiments
FUHegpyynl102
BERK_BI_ENDE_T2FB_B
FUHegpyydl102
0.35 0.4 0.45 0.5 0.55 0.6 0.65 0.7
arcsin(sqrt(R Precision))
52
DS-BILI-CLEF2006 Track Overview Results and Graphs DS-BILI-X2DE-CLEF2006
Average Precision R-Precision
Topic Minimum 1st Q. Median 3rd Q. Maximum Mean Std Minimum 1st Q. Median 3rd Q. Maximum Mean Std
151 0.0467 0.0567 0.0785 0.0951 0.7841 0.1899 0.2916 0.1378 0.1654 0.1732 0.2283 0.6969 0.2625 0.2148
152 0.0010 0.0077 0.0135 0.0189 0.0813 0.0226 0.0294 0.0192 0.0288 0.0697 0.1010 0.1346 0.0705 0.0434
153 0.0397 0.0547 0.3480 0.4089 0.5316 0.2885 0.1991 0.0800 0.0800 0.3800 0.4533 0.5067 0.3133 0.1874
154 0.0767 0.2812 0.3664 0.6908 0.8019 0.4306 0.2728 0.1861 0.3905 0.4690 0.6350 0.7226 0.4787 0.1937
155 0.0018 0.1565 0.1846 0.2551 0.2694 0.1753 0.0971 0.0104 0.1917 0.2280 0.3264 0.3523 0.2228 0.1230
156 0.0000 0.0067 0.0669 0.0997 0.1036 0.0573 0.0461 0.0000 0.0215 0.1237 0.1505 0.1505 0.0950 0.0675
157 0.0016 0.0019 0.0313 0.1087 0.7927 0.1612 0.3124 0.0000 0.0154 0.1000 0.2000 0.6923 0.1846 0.2643
158 0.0120 0.0224 0.0234 0.0783 0.0891 0.0414 0.0332 0.0562 0.0787 0.0955 0.1348 0.1685 0.1049 0.0418
159 0.2572 0.4319 0.6030 0.6746 0.6919 0.5436 0.1724 0.2870 0.4907 0.5648 0.6111 0.6204 0.5231 0.1267
160 0.0004 0.0201 0.0492 0.1605 0.1643 0.0739 0.0721 0.0000 0.0556 0.1111 0.2778 0.2847 0.1400 0.1209
161 0.2536 0.3237 0.4269 0.4850 0.6059 0.4203 0.1285 0.3323 0.3892 0.4684 0.4937 0.5854 0.4562 0.0889
162 0.0922 0.0966 0.1330 0.1603 0.2902 0.1509 0.0731 0.1724 0.2112 0.2694 0.3233 0.4052 0.2751 0.0864
163 0.0928 0.1133 0.1516 0.1872 0.2026 0.1499 0.0451 0.2140 0.2179 0.2568 0.2879 0.3191 0.2588 0.0416
164 0.0189 0.0195 0.0341 0.1320 0.2338 0.0787 0.0875 0.0000 0.0000 0.0385 0.1731 0.2692 0.0865 0.1126
165 0.0197 0.1868 0.2068 0.2553 0.2594 0.1891 0.0879 0.0411 0.2740 0.3151 0.3151 0.3288 0.2648 0.1112
166 0.2525 0.3106 0.4455 0.5706 0.6051 0.4383 0.1489 0.3226 0.3656 0.4906 0.5699 0.6075 0.4745 0.1174
167 0.0241 0.0243 0.0354 0.0657 0.0932 0.0463 0.0281 0.0444 0.0444 0.0667 0.1556 0.1778 0.0926 0.0587
168 0.1233 0.1461 0.3208 0.3346 0.4099 0.2759 0.1146 0.2041 0.2245 0.4218 0.4422 0.4422 0.3594 0.1133
169 0.0455 0.0696 0.1172 0.2417 0.2447 0.1393 0.0851 0.0909 0.0909 0.0909 0.1818 0.2727 0.1364 0.0761
170 0.1617 0.2228 0.2593 0.2812 0.5663 0.2918 0.1409 0.2308 0.2564 0.3590 0.3590 0.5897 0.3590 0.1267
171 0.1790 0.1905 0.1999 0.2120 0.2222 0.2006 0.0163 0.2727 0.3081 0.3106 0.3485 0.3535 0.3173 0.0299
172 0.0221 0.0701 0.0742 0.0843 0.0986 0.0706 0.0260 0.0455 0.0909 0.1364 0.1364 0.1818 0.1212 0.0469
173 0.0027 0.4584 0.4868 0.5205 0.6057 0.4268 0.2139 0.0159 0.5040 0.5238 0.5675 0.5992 0.4557 0.2182
174 0.0564 0.4581 0.4689 0.5190 0.5752 0.4244 0.1856 0.1250 0.5284 0.5341 0.6023 0.6364 0.4934 0.1858
175 0.0476 0.0745 0.0751 0.1399 0.1923 0.1008 0.0543 0.0781 0.1562 0.1641 0.1875 0.3125 0.1771 0.0763
ALL 0.1768 0.1980 0.2185 0.2366 0.2448 0.2155 0.0250 0.2326 0.2552 0.2720 0.2781 0.3038 0.2689 0.0241
53
54
DS-BILI-CLEF2006 Track Overview Results and Graphs DS-BILI-X2RU-CLEF2006
Domain−Specific Bilingual Russian track Top 5 Participants − Interpolated Recall vs Average Precision
100%
berkeley [BERK_BI_ENRU_T2FB_P; MAP 16.48%; Pooled]
90%
80%
70%
Average Precision
60%
50%
40%
30%
20%
10%
0%
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
Interpolated Recall
Domain−Specific Bilingual Russian track Top 5 Participants − Comparison to Median Mean Average Precision by Topic (Topics 151to 175)
1
berkeley [BERK_BI_ENRU_T2FB_P; MAP 16.48%; Pooled]
0.8
0.6
0.4
0.2
Difference
0
−0.2
−0.4
−0.6
−0.8
−1
151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175
Topic Identifier
55
DS-BILI-CLEF2006 Track Overview Results and Graphs DS-BILI-X2RU-CLEF2006
Domain−Specific Bilingual Russian track − Box Plot of the Topics
BERK_BI_ENRU_T2FB_P [MAP 16.48%; Pooled]
Experiments
BERK_BI_ENRU_T2FB_B [MAP 11.55%; Pooled]
BERK_BI_DERU_T2FB_P [MAP 10.83%; Pooled]
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
Mean Average Precision
56
DS-BILI-CLEF2006 Track Overview Results and Graphs DS-BILI-X2RU-CLEF2006
Domain−Specific Bilingual Russian track − Tukey T test with "top group" highlighted
BERK_BI_ENRU_T2FB_P
Experiments
BERK_BI_DERU_T2FB_P
BERK_BI_ENRU_T2FB_B
0.2 0.25 0.3 0.35 0.4 0.45 0.5
arcsin(sqrt(Mean average precision))
57
DS-BILI-CLEF2006 Track Overview Results and Graphs DS-BILI-X2RU-CLEF2006
Domain−Specific Bilingual Russian track Top 5 Participants − Retrieved documents vs Precision
100%
berkeley [BERK_BI_ENRU_T2FB_P; R−Prec 17.38%; Pooled]
90%
80%
70%
60%
R−Precision
50%
40%
30%
20%
10%
0%
5 10 15 20 30 100 200 500 1000
Retrieved Documents (logarithmic scale)
Domain−Specific Bilingual Russian track Top 5 Participants − Comparison to Median R−Precision by Topic (Topics 151 to 175)
1
berkeley [BERK_BI_ENRU_T2FB_P; R−Prec 17.38%; Pooled]
0.8
0.6
0.4
0.2
Difference
0
−0.2
−0.4
−0.6
−0.8
−1
151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175
Topic Identifier
58
DS-BILI-CLEF2006 Track Overview Results and Graphs DS-BILI-X2RU-CLEF2006
Domain−Specific Bilingual Russian track − Box Plot of the Topics
BERK_BI_ENRU_T2FB_P [R−Prec 17.38%; Pooled]
Experiments
BERK_BI_ENRU_T2FB_B [R−Prec 12.91%; Pooled]
BERK_BI_DERU_T2FB_P [R−Prec 11.87%; Pooled]
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
R−Precision
59
DS-BILI-CLEF2006 Track Overview Results and Graphs DS-BILI-X2RU-CLEF2006
Domain−Specific Bilingual Russian track − Tukey T test with "top group" highlighted
BERK_BI_ENRU_T2FB_P
Experiments
BERK_BI_ENRU_T2FB_B
BERK_BI_DERU_T2FB_P
0.2 0.25 0.3 0.35 0.4
arcsin(sqrt(R Precision))
60
DS-BILI-CLEF2006 Track Overview Results and Graphs DS-BILI-X2RU-CLEF2006
Average Precision R-Precision
Topic Minimum 1st Q. Median 3rd Q. Maximum Mean Std Minimum 1st Q. Median 3rd Q. Maximum Mean Std
151 0.0719 0.0806 0.1069 0.1312 0.1394 0.1061 0.0337 0.0625 0.0781 0.1250 0.2188 0.2500 0.1458 0.0955
152 0.0006 0.0798 0.3174 0.4546 0.5003 0.2728 0.2528 0.0233 0.1192 0.4070 0.4419 0.4535 0.2946 0.2361
153 0.0927 0.1048 0.1410 0.1424 0.1429 0.1255 0.0284 0.1304 0.1304 0.1304 0.1304 0.1304 0.1304 0.0000
154 0.0263 0.0386 0.0752 0.2344 0.2874 0.1297 0.1388 0.1616 0.1793 0.2323 0.3838 0.4343 0.2761 0.1415
155 0.0000 0.0001 0.0006 0.5501 0.7333 0.2446 0.4232 0.0000 0.0034 0.0135 0.5304 0.7027 0.2387 0.4019
156 0.0078 0.0091 0.0130 0.0233 0.0268 0.0159 0.0098 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000
157 0.0000 0.0000 0.0000 0.3772 0.5029 0.1676 0.2903 0.0000 0.0000 0.0000 0.3750 0.5000 0.1667 0.2887
158 0.0027 0.0114 0.0375 0.1175 0.1441 0.0614 0.0737 0.0000 0.0000 0.0000 0.1071 0.1429 0.0476 0.0825
159 0.5616 0.5710 0.5991 0.6542 0.6726 0.6111 0.0564 0.6000 0.6000 0.6000 0.6857 0.7143 0.6381 0.0660
160 0.0000 0.0002 0.0008 0.0079 0.0102 0.0037 0.0057 0.0000 0.0068 0.0270 0.0608 0.0721 0.0330 0.0364
161 0.0414 0.1864 0.6215 0.6807 0.7004 0.4544 0.3599 0.0909 0.2074 0.5568 0.6335 0.6591 0.4356 0.3029
162 0.0483 0.1051 0.2756 0.3197 0.3344 0.2194 0.1511 0.0862 0.1250 0.2414 0.3448 0.3793 0.2356 0.1466
163 0.1468 0.2223 0.4488 0.4552 0.4573 0.3510 0.1769 0.2035 0.2854 0.5310 0.5442 0.5487 0.4277 0.1944
164 0.0029 0.0079 0.0227 0.0667 0.0813 0.0356 0.0408 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000
165 0.0131 0.0146 0.0191 0.1101 0.1405 0.0576 0.0719 0.0000 0.0139 0.0556 0.1597 0.1944 0.0833 0.1002
166 0.0000 0.0091 0.0366 0.0498 0.0542 0.0303 0.0277 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000
167 0.0096 0.0102 0.0118 0.0135 0.0141 0.0118 0.0022 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000
168 0.0023 0.0658 0.2563 0.2855 0.2952 0.1846 0.1591 0.0000 0.0625 0.2500 0.2500 0.2500 0.1667 0.1443
169 0.0035 0.0116 0.0358 0.0953 0.1151 0.0515 0.0574 0.0000 0.0000 0.0000 0.1875 0.2500 0.0833 0.1443
170 0.0126 0.0194 0.0396 0.0705 0.0808 0.0443 0.0343 0.0000 0.0109 0.0435 0.0761 0.0870 0.0435 0.0435
171 0.0033 0.0052 0.0110 0.0142 0.0153 0.0099 0.0061 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000
172 0.0014 0.0018 0.0030 0.0143 0.0181 0.0075 0.0092 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000
173 0.0000 0.0019 0.0077 0.0912 0.1190 0.0422 0.0666 0.0000 0.0000 0.0000 0.1500 0.2000 0.0667 0.1155
174 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000
175 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000
ALL 0.1083 0.1101 0.1155 0.1525 0.1648 0.1295 0.0308 0.1187 0.1213 0.1291 0.1626 0.1738 0.1405 0.0293
61
62
DS-MULTI-CLEF2006 Track Overview Results and Graphs DS-MULTI-CLEF2006
Domain−Specific Multilingual track Top 5 Participants − Interpolated Recall vs Average Precision
100%
berkeley [BERK_MU_EN_T2FB_B_CMBZ; MAP 7.53%; Not Pooled]
90%
80%
70%
Average Precision
60%
50%
40%
30%
20%
10%
0%
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
Interpolated Recall
Domain−Specific Multilingual track Top 5 Participants − Comparison to Median Mean Average Precision by Topic (Topics 151 to 7
15)
1
berkeley [BERK_MU_EN_T2FB_B_CMBZ; MAP 7.53%; Not Pooled]
0.8
0.6
0.4
0.2
Difference
0
−0.2
−0.4
−0.6
−0.8
−1
151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175
Topic Identifier
63
DS-MULTI-CLEF2006 Track Overview Results and Graphs DS-MULTI-CLEF2006
Domain−Specific Multilingual track − Box Plot of the Topics
BERK_MU_EN_T2FB_B_CMBZ [MAP 7.53%; Not Pooled]
Experiments
BERK_MU_DE_T2FB_B_CMBZ [MAP 4.67%; Not Pooled]
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
Mean Average Precision
64
DS-MULTI-CLEF2006 Track Overview Results and Graphs DS-MULTI-CLEF2006
Domain−Specific Multilingual track − Tukey T test with "top group" highlighted
BERK_MU_EN_T2FB_B_CMBZ
Experiments
BERK_MU_DE_T2FB_B_CMBZ
0.12 0.14 0.16 0.18 0.2 0.22 0.24 0.26 0.28 0.3
arcsin(sqrt(Mean average precision))
65
DS-MULTI-CLEF2006 Track Overview Results and Graphs DS-MULTI-CLEF2006
Domain−Specific Multilingual track Top 5 Participants − Retrieved documents vs Precision
100%
berkeley [BERK_MU_EN_T2FB_B_CMBZ; R−Prec 15.97%; Not Pooled]
90%
80%
70%
60%
R−Precision
50%
40%
30%
20%
10%
0%
5 10 15 20 30 100 200 500 1000
Retrieved Documents (logarithmic scale)
Domain−Specific Multilingual track Top 5 Participants − Comparison to Median R−Precision by Topic (Topics 151 to 175)
1
berkeley [BERK_MU_EN_T2FB_B_CMBZ; R−Prec 15.97%; Not Pooled]
0.8
0.6
0.4
0.2
Difference
0
−0.2
−0.4
−0.6
−0.8
−1
151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175
Topic Identifier
66
DS-MULTI-CLEF2006 Track Overview Results and Graphs DS-MULTI-CLEF2006
Domain−Specific Multilingual track − Box Plot of the Topics
BERK_MU_EN_T2FB_B_CMBZ [R−Prec 15.97%; Not Pooled]
Experiments
BERK_MU_DE_T2FB_B_CMBZ [R−Prec 11.15%; Not Pooled]
0% 10% 20% 30% 40% 50% 60% 70% 80% 90%100%
R−Precision
67
DS-MULTI-CLEF2006 Track Overview Results and Graphs DS-MULTI-CLEF2006
Domain−Specific Multilingual track − Tukey T test with "top group" highlighted
BERK_MU_EN_T2FB_B_CMBZ
Experiments
BERK_MU_DE_T2FB_B_CMBZ
0.26 0.28 0.3 0.32 0.34 0.36 0.38 0.4 0.42 0.44
arcsin(sqrt(R Precision))
68
DS-MULTI-CLEF2006 Track Overview Results and Graphs DS-MULTI-CLEF2006
Average Precision R-Precision
Topic Minimum 1st Q. Median 3rd Q. Maximum Mean Std Minimum 1st Q. Median 3rd Q. Maximum Mean Std
151 0.0112 0.0112 0.1282 0.2452 0.2452 0.1282 0.1654 0.0533 0.0533 0.2190 0.3848 0.3848 0.2190 0.2344
152 0.0001 0.0001 0.0444 0.0886 0.0886 0.0444 0.0626 0.0047 0.0047 0.1228 0.2409 0.2409 0.1228 0.1670
153 0.0916 0.0916 0.0990 0.1064 0.1064 0.0990 0.0105 0.2274 0.2274 0.2390 0.2506 0.2506 0.2390 0.0164
154 0.0342 0.0342 0.0386 0.0430 0.0430 0.0386 0.0062 0.1248 0.1248 0.1314 0.1379 0.1379 0.1314 0.0093
155 0.0072 0.0072 0.0558 0.1044 0.1044 0.0558 0.0687 0.0420 0.0420 0.1436 0.2452 0.2452 0.1436 0.1437
156 0.0105 0.0105 0.0374 0.0643 0.0643 0.0374 0.0380 0.0242 0.0242 0.0773 0.1304 0.1304 0.0773 0.0752
157 0.0512 0.0512 0.1393 0.2274 0.2274 0.1393 0.1246 0.1358 0.1358 0.2531 0.3704 0.3704 0.2531 0.1659
158 0.0062 0.0062 0.0602 0.1143 0.1143 0.0602 0.0764 0.0512 0.0512 0.1395 0.2279 0.2279 0.1395 0.1250
159 0.0029 0.0029 0.0148 0.0267 0.0267 0.0148 0.0169 0.0317 0.0317 0.0548 0.0778 0.0778 0.0548 0.0326
160 0.0017 0.0017 0.0110 0.0203 0.0203 0.0110 0.0132 0.0308 0.0308 0.0625 0.0942 0.0942 0.0625 0.0449
161 0.0961 0.0961 0.1446 0.1931 0.1931 0.1446 0.0686 0.2522 0.2522 0.2949 0.3376 0.3376 0.2949 0.0604
162 0.0016 0.0016 0.0023 0.0030 0.0030 0.0023 0.0010 0.0216 0.0216 0.0224 0.0232 0.0232 0.0224 0.0012
163 0.0041 0.0041 0.0147 0.0252 0.0252 0.0147 0.0149 0.0234 0.0234 0.0635 0.1035 0.1035 0.0635 0.0566
164 0.0138 0.0138 0.0342 0.0547 0.0547 0.0342 0.0289 0.0534 0.0534 0.1145 0.1756 0.1756 0.1145 0.0864
165 0.0033 0.0033 0.0137 0.0240 0.0240 0.0137 0.0146 0.0178 0.0178 0.0355 0.0533 0.0533 0.0355 0.0251
166 0.0028 0.0028 0.0757 0.1487 0.1487 0.0757 0.1031 0.0205 0.0205 0.1596 0.2986 0.2986 0.1596 0.1966
167 0.0062 0.0062 0.0129 0.0196 0.0196 0.0129 0.0095 0.0141 0.0141 0.0493 0.0845 0.0845 0.0493 0.0498
168 0.0421 0.0421 0.0485 0.0550 0.0550 0.0485 0.0091 0.1320 0.1320 0.1580 0.1840 0.1840 0.1580 0.0368
169 0.0003 0.0003 0.0098 0.0194 0.0194 0.0098 0.0135 0.0000 0.0000 0.0246 0.0492 0.0492 0.0246 0.0348
170 0.1457 0.1457 0.1705 0.1952 0.1952 0.1705 0.0350 0.2738 0.2738 0.3036 0.3333 0.3333 0.3036 0.0421
171 0.1986 0.1986 0.2669 0.3352 0.3352 0.2669 0.0966 0.3560 0.3560 0.4018 0.4476 0.4476 0.4018 0.0648
172 0.0423 0.0423 0.0445 0.0468 0.0468 0.0445 0.0031 0.1250 0.1250 0.1406 0.1562 0.1562 0.1406 0.0221
173 0.0058 0.0058 0.0146 0.0235 0.0235 0.0146 0.0125 0.0439 0.0439 0.0601 0.0763 0.0763 0.0601 0.0229
174 0.0039 0.0039 0.0427 0.0815 0.0815 0.0427 0.0549 0.0346 0.0346 0.1066 0.1787 0.1787 0.1066 0.1019
175 0.0013 0.0013 0.0015 0.0017 0.0017 0.0015 0.0003 0.0085 0.0085 0.0128 0.0171 0.0171 0.0128 0.0060
ALL 0.0467 0.0467 0.0610 0.0753 0.0753 0.0610 0.0202 0.1115 0.1115 0.1356 0.1597 0.1597 0.1356 0.0341
69
70
Individual Experiment Results and Graphs
71
72
berkeley BERK_MO_EN_T2FB DS-MONO-EN-CLEF2006
Overall statistics for 25 queries : Priority 1
Total number of documents over all queries Query Construction AUTOMATIC
Retrieved 25,000 Source Language English
Relevant 4,239 Topic Fields title, description
Relevant retrieved 3,359 Pooled true
Geometric Mean Average Precision 0.3399 Berkeley Trec2 algorithm with blind feedback
Binary Preference (BPREF) 0.4599
Interploated Recall (%) Precision Averages (%)
Domain−Specific Monolingual English track − Interpolated Recall vs Average Precision
100%
0 87.29
BERK_MO_EN_T2FB
10 68.16 90%
20 64.06
80%
30 59.01
40 52.10 70%
50 44.34
Average Precision
60%
60 35.68
70 27.98 50%
80 17.04 40%
90 10.92
30%
100 0.97
Average precision (non-interpolated) for all 20%
relevant documents (averaged over queries)
41.36 10%
0%
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
Interpolated Recall
Mean Average Precision Domain−Specific Monolingual English track − Box plot of the Topics of the Experiment
Maximum 0.7779
Minimum 0.0442
First Quartile 0.1927
Second Quartile 0.4573
Third Quartile 0.5680
Interquartile range 0.3754
Mean 0.4136
Standard Deviation 0.2195
Lower Outlier Threshold 0.0442
Upper Outlier Threshold 0.7779 0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Mean Average Precision
Mean With No Outliers 0.4136
Std With No Outliers 0.2195
Domain−Specific Monolingual English track − Distribution of the Topics of the Experiment
4
Number of Topics of the Experiment
3.5
3
2.5
2
1.5
1
0.5
0
0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Mean Average Precision
Precision averages (%) for individual 1
Domain−Specific Monolingual English track − Comparison to Median Mean Average Precision by Topic (Topics 151 to 175)
queries BERK_MO_EN_T2FB
Topic 151 72.77 Topic 164 10.69 0.8
Topic 152 53.19 Topic 165 77.79
Topic 153 45.68 Topic 166 53.67
0.6
Topic 154 73.77 Topic 167 14.40
Topic 155 48.75 Topic 168 13.94
Topic 156 46.29 Topic 169 17.95 0.4
Topic 157 55.46 Topic 170 31.46
Topic 158 19.70 Topic 171 45.73 0.2
Topic 159 70.24 Topic 172 17.62
Difference
Topic 160 29.79 Topic 173 60.84 0
Topic 161 50.90 Topic 174 60.97
Topic 162 28.36 Topic 175 4.42
−0.2
Topic 163 29.54
−0.4
−0.6
−0.8
−1
151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175
Topic Identifier
73
berkeley BERK_MO_EN_T2FB DS-MONO-EN-CLEF2006
Docs Cutoff Levels Precision at DCL (%)
Domain−Specific Monolingual English track − Retrieved documents vs Precision
100%
5 docs 69.60
BERK_MO_EN_T2FB
10 docs 69.20 90%
15 docs 68.00
80%
20 docs 65.40
30 docs 62.67 70%
100 docs 54.56
60%
200 docs 41.26
R−Precision
500 docs 22.92 50%
1000 docs 13.44 40%
R-Precision (precision after R document retrieved,
where R = Relevant retrieved) 30%
44.19
20%
10%
0%
5 10 15 20 30 100 200 500 1000
Retrieved Documents (logarithmic scale)
Exact R-Precision Domain−Specific Monolingual English track − Box plot of the Topics of the Experiment
Maximum 0.7090
Minimum 0.0755
First Quartile 0.2732
Second Quartile 0.4906
Third Quartile 0.5863
Interquartile range 0.3131
Mean 0.4419
Standard Deviation 0.1931
Lower Outlier Threshold 0.0755
Upper Outlier Threshold 0.7090 0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Exact R−Precision
Mean With No Outliers 0.4419
Std With No Outliers 0.1931
Domain−Specific Monolingual English track − Distribution of the Topics of the Experiment
5
Number of Topics of the Experiment
4
3
2
1
0
0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Exact R−Precision
Precision averages (%) for individual 1
Domain−Specific Monolingual English track − Comparison to Median Mean Exact R−Precision by Topic (Topics 151 to 175)
queries BERK_MO_EN_T2FB
Topic 151 67.84 Topic 164 14.47 0.8
Topic 152 58.06 Topic 165 70.00
Topic 153 46.71 Topic 166 53.12
0.6
Topic 154 69.49 Topic 167 17.39
Topic 155 51.32 Topic 168 26.26
Topic 156 49.06 Topic 169 16.67 0.4
Topic 157 52.63 Topic 170 45.45
Topic 158 27.68 Topic 171 51.10 0.2
Topic 159 70.90 Topic 172 17.50
Difference
Topic 160 38.87 Topic 173 60.33 0
Topic 161 53.54 Topic 174 64.33
Topic 162 35.14 Topic 175 7.55
−0.2
Topic 163 39.44
−0.4
−0.6
−0.8
−1
151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175
Topic Identifier
74
tuchemniz TUCMIgirten1 DS-MONO-EN-CLEF2006
Overall statistics for 25 queries : Priority 1
Total number of documents over all queries Query Construction AUTOMATIC
Retrieved 25,000 Source Language English
Relevant 4,239 Topic Fields title, description
Relevant retrieved 3,182 Pooled true
Geometric Mean Average Precision 0.2667 Snowball-Porter-QE-BF2
Binary Preference (BPREF) 0.4050
Interploated Recall (%) Precision Averages (%)
Domain−Specific Monolingual English track − Interpolated Recall vs Average Precision
100%
0 84.59
TUCMIgirten1
10 64.20 90%
20 57.99
80%
30 52.31
40 44.17 70%
50 34.75
Average Precision
60%
60 25.08
70 20.57 50%
80 14.67 40%
90 6.07
30%
100 0.00
Average precision (non-interpolated) for all 20%
relevant documents (averaged over queries)
35.10 10%
0%
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
Interpolated Recall
Mean Average Precision Domain−Specific Monolingual English track − Box plot of the Topics of the Experiment
Maximum 0.7857
Minimum 0.0273
First Quartile 0.1420
Second Quartile 0.3773
Third Quartile 0.5269
Interquartile range 0.3849
Mean 0.3510
Standard Deviation 0.2185
Lower Outlier Threshold 0.0273
Upper Outlier Threshold 0.7857 0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Mean Average Precision
Mean With No Outliers 0.3510
Std With No Outliers 0.2185
Domain−Specific Monolingual English track − Distribution of the Topics of the Experiment
4
Number of Topics of the Experiment
3.5
3
2.5
2
1.5
1
0.5
0
0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Mean Average Precision
Precision averages (%) for individual 1
Domain−Specific Monolingual English track − Comparison to Median Mean Average Precision by Topic (Topics 151 to 175)
queries TUCMIgirten1
Topic 151 53.31 Topic 164 2.73 0.8
Topic 152 58.17 Topic 165 54.31
Topic 153 42.69 Topic 166 46.30
0.6
Topic 154 72.48 Topic 167 6.34
Topic 155 49.36 Topic 168 12.24
Topic 156 52.48 Topic 169 11.62 0.4
Topic 157 19.52 Topic 170 25.47
Topic 158 24.68 Topic 171 41.22 0.2
Topic 159 78.57 Topic 172 7.92
Difference
Topic 160 16.26 Topic 173 37.73 0
Topic 161 51.46 Topic 174 56.68
Topic 162 29.02 Topic 175 11.99
−0.2
Topic 163 14.85
−0.4
−0.6
−0.8
−1
151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175
Topic Identifier
75
tuchemniz TUCMIgirten1 DS-MONO-EN-CLEF2006
Docs Cutoff Levels Precision at DCL (%)
Domain−Specific Monolingual English track − Retrieved documents vs Precision
100%
5 docs 66.40
TUCMIgirten1
10 docs 64.80 90%
15 docs 63.73
80%
20 docs 62.40
30 docs 58.80 70%
100 docs 48.96
60%
200 docs 36.80
R−Precision
500 docs 21.20 50%
1000 docs 12.73 40%
R-Precision (precision after R document retrieved,
where R = Relevant retrieved) 30%
38.96
20%
10%
0%
5 10 15 20 30 100 200 500 1000
Retrieved Documents (logarithmic scale)
Exact R-Precision Domain−Specific Monolingual English track − Box plot of the Topics of the Experiment
Maximum 0.7910
Minimum 0.0526
First Quartile 0.2498
Second Quartile 0.4256
Third Quartile 0.5041
Interquartile range 0.2543
Mean 0.3896
Standard Deviation 0.1769
Lower Outlier Threshold 0.0526
Upper Outlier Threshold 0.7910 0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Exact R−Precision
Mean With No Outliers 0.3896
Std With No Outliers 0.1769
Domain−Specific Monolingual English track − Distribution of the Topics of the Experiment
4
Number of Topics of the Experiment
3.5
3
2.5
2
1.5
1
0.5
0
0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Exact R−Precision
Precision averages (%) for individual 1
Domain−Specific Monolingual English track − Comparison to Median Mean Exact R−Precision by Topic (Topics 151 to 175)
queries TUCMIgirten1
Topic 151 52.55 Topic 164 5.26 0.8
Topic 152 56.60 Topic 165 50.00
Topic 153 43.60 Topic 166 49.22
0.6
Topic 154 61.86 Topic 167 21.74
Topic 155 51.64 Topic 168 18.18
Topic 156 50.00 Topic 169 16.67 0.4
Topic 157 28.42 Topic 170 31.82
Topic 158 28.57 Topic 171 47.80 0.2
Topic 159 79.10 Topic 172 15.00
Difference
Topic 160 29.81 Topic 173 42.56 0
Topic 161 49.34 Topic 174 57.89
Topic 162 39.62 Topic 175 20.75
−0.2
Topic 163 26.06
−0.4
−0.6
−0.8
−1
151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175
Topic Identifier
76
tuchemniz TUCMIgirten2 DS-MONO-EN-CLEF2006
Overall statistics for 25 queries : Priority 2
Total number of documents over all queries Query Construction AUTOMATIC
Retrieved 25,000 Source Language English
Relevant 4,239 Topic Fields title, description
Relevant retrieved 3,197 Pooled true
Geometric Mean Average Precision 0.2716 Snowball-Porter-QE-BF2-DC
Binary Preference (BPREF) 0.4065
Interploated Recall (%) Precision Averages (%)
Domain−Specific Monolingual English track − Interpolated Recall vs Average Precision
100%
0 84.71
TUCMIgirten2
10 64.96 90%
20 59.01
80%
30 53.49
40 44.82 70%
50 35.45
Average Precision
60%
60 25.64
70 20.44 50%
80 13.58 40%
90 5.79
30%
100 0.00
Average precision (non-interpolated) for all 20%
relevant documents (averaged over queries)
35.53 10%
0%
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
Interpolated Recall
Mean Average Precision Domain−Specific Monolingual English track − Box plot of the Topics of the Experiment
Maximum 0.7675
Minimum 0.0251
First Quartile 0.1601
Second Quartile 0.4202
Third Quartile 0.5327
Interquartile range 0.3726
Mean 0.3553
Standard Deviation 0.2160
Lower Outlier Threshold 0.0251
Upper Outlier Threshold 0.7675 0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Mean Average Precision
Mean With No Outliers 0.3553
Std With No Outliers 0.2160
Domain−Specific Monolingual English track − Distribution of the Topics of the Experiment
4
Number of Topics of the Experiment
3.5
3
2.5
2
1.5
1
0.5
0
0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Mean Average Precision
Precision averages (%) for individual 1
Domain−Specific Monolingual English track − Comparison to Median Mean Average Precision by Topic (Topics 151 to 175)
queries TUCMIgirten2
Topic 151 54.38 Topic 164 2.51 0.8
Topic 152 59.19 Topic 165 53.13
Topic 153 46.60 Topic 166 47.57
0.6
Topic 154 71.48 Topic 167 6.04
Topic 155 48.28 Topic 168 9.49
Topic 156 53.68 Topic 169 17.19 0.4
Topic 157 13.61 Topic 170 24.55
Topic 158 17.31 Topic 171 43.09 0.2
Topic 159 76.75 Topic 172 10.02
Difference
Topic 160 28.07 Topic 173 42.02 0
Topic 161 51.02 Topic 174 55.11
Topic 162 25.90 Topic 175 14.99
−0.2
Topic 163 16.34
−0.4
−0.6
−0.8
−1
151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175
Topic Identifier
77
tuchemniz TUCMIgirten2 DS-MONO-EN-CLEF2006
Docs Cutoff Levels Precision at DCL (%)
Domain−Specific Monolingual English track − Retrieved documents vs Precision
100%
5 docs 66.40
TUCMIgirten2
10 docs 64.80 90%
15 docs 65.07
80%
20 docs 63.60
30 docs 59.20 70%
100 docs 49.56
60%
200 docs 37.36
R−Precision
500 docs 21.25 50%
1000 docs 12.79 40%
R-Precision (precision after R document retrieved,
where R = Relevant retrieved) 30%
39.33
20%
10%
0%
5 10 15 20 30 100 200 500 1000
Retrieved Documents (logarithmic scale)
Exact R-Precision Domain−Specific Monolingual English track − Box plot of the Topics of the Experiment
Maximum 0.7687
Minimum 0.0526
First Quartile 0.2251
Second Quartile 0.4669
Third Quartile 0.5090
Interquartile range 0.2840
Mean 0.3933
Standard Deviation 0.1810
Lower Outlier Threshold 0.0526
Upper Outlier Threshold 0.7687 0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Exact R−Precision
Mean With No Outliers 0.3933
Std With No Outliers 0.1810
Domain−Specific Monolingual English track − Distribution of the Topics of the Experiment
6
Number of Topics of the Experiment
5
4
3
2
1
0
0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Exact R−Precision
Precision averages (%) for individual 1
Domain−Specific Monolingual English track − Comparison to Median Mean Exact R−Precision by Topic (Topics 151 to 175)
queries TUCMIgirten2
Topic 151 55.29 Topic 164 5.26 0.8
Topic 152 58.36 Topic 165 48.33
Topic 153 48.44 Topic 166 49.22
0.6
Topic 154 62.29 Topic 167 17.39
Topic 155 53.62 Topic 168 15.15
Topic 156 50.00 Topic 169 14.29 0.4
Topic 157 22.11 Topic 170 36.36
Topic 158 25.89 Topic 171 48.90 0.2
Topic 159 76.87 Topic 172 20.00
Difference
Topic 160 36.60 Topic 173 46.69 0
Topic 161 48.56 Topic 174 57.31
Topic 162 37.70 Topic 175 22.64
−0.2
Topic 163 26.06
−0.4
−0.6
−0.8
−1
151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175
Topic Identifier
78
tuchemniz TUCMIgirten3 DS-MONO-EN-CLEF2006
Overall statistics for 25 queries : Priority 3
Total number of documents over all queries Query Construction AUTOMATIC
Retrieved 25,000 Source Language English
Relevant 4,239 Topic Fields title, description
Relevant retrieved 3,171 Pooled true
Geometric Mean Average Precision 0.2611 Snowball-Porter-QE-BF2-TC
Binary Preference (BPREF) 0.4002
Interploated Recall (%) Precision Averages (%)
Domain−Specific Monolingual English track − Interpolated Recall vs Average Precision
100%
0 83.53
TUCMIgirten3
10 63.29 90%
20 57.46
80%
30 53.20
40 43.30 70%
50 33.96
Average Precision
60%
60 24.24
70 19.37 50%
80 13.00 40%
90 5.87
30%
100 0.00
Average precision (non-interpolated) for all 20%
relevant documents (averaged over queries)
34.50 10%
0%
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
Interpolated Recall
Mean Average Precision Domain−Specific Monolingual English track − Box plot of the Topics of the Experiment
Maximum 0.7851
Minimum 0.0262
First Quartile 0.1296
Second Quartile 0.3810
Third Quartile 0.5182
Interquartile range 0.3887
Mean 0.3450
Standard Deviation 0.2184
Lower Outlier Threshold 0.0262
Upper Outlier Threshold 0.7851 0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Mean Average Precision
Mean With No Outliers 0.3450
Std With No Outliers 0.2184
Domain−Specific Monolingual English track − Distribution of the Topics of the Experiment
4
Number of Topics of the Experiment
3.5
3
2.5
2
1.5
1
0.5
0
0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Mean Average Precision
Precision averages (%) for individual 1
Domain−Specific Monolingual English track − Comparison to Median Mean Average Precision by Topic (Topics 151 to 175)
queries TUCMIgirten3
Topic 151 53.01 Topic 164 2.62 0.8
Topic 152 59.48 Topic 165 51.58
Topic 153 42.69 Topic 166 44.57
0.6
Topic 154 71.48 Topic 167 7.92
Topic 155 46.24 Topic 168 10.75
Topic 156 52.54 Topic 169 10.33 0.4
Topic 157 20.38 Topic 170 25.47
Topic 158 13.69 Topic 171 41.40 0.2
Topic 159 78.51 Topic 172 9.11
Difference
Topic 160 20.76 Topic 173 38.10 0
Topic 161 51.32 Topic 174 56.68
Topic 162 28.16 Topic 175 10.03
−0.2
Topic 163 15.59
−0.4
−0.6
−0.8
−1
151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175
Topic Identifier
79
tuchemniz TUCMIgirten3 DS-MONO-EN-CLEF2006
Docs Cutoff Levels Precision at DCL (%)
Domain−Specific Monolingual English track − Retrieved documents vs Precision
100%
5 docs 65.60
TUCMIgirten3
10 docs 63.20 90%
15 docs 62.13
80%
20 docs 61.60
30 docs 58.00 70%
100 docs 48.88
60%
200 docs 36.54
R−Precision
500 docs 20.95 50%
1000 docs 12.68 40%
R-Precision (precision after R document retrieved,
where R = Relevant retrieved) 30%
38.52
20%
10%
0%
5 10 15 20 30 100 200 500 1000
Retrieved Documents (logarithmic scale)
Exact R-Precision Domain−Specific Monolingual English track − Box plot of the Topics of the Experiment
Maximum 0.7910
Minimum 0.0789
First Quartile 0.2218
Second Quartile 0.4256
Third Quartile 0.5008
Interquartile range 0.2791
Mean 0.3852
Standard Deviation 0.1795
Lower Outlier Threshold 0.0789
Upper Outlier Threshold 0.7910 0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Exact R−Precision
Mean With No Outliers 0.3852
Std With No Outliers 0.1795
Domain−Specific Monolingual English track − Distribution of the Topics of the Experiment
4
Number of Topics of the Experiment
3.5
3
2.5
2
1.5
1
0.5
0
0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Exact R−Precision
Precision averages (%) for individual 1
Domain−Specific Monolingual English track − Comparison to Median Mean Exact R−Precision by Topic (Topics 151 to 175)
queries TUCMIgirten3
Topic 151 53.33 Topic 164 7.89 0.8
Topic 152 57.77 Topic 165 50.00
Topic 153 43.60 Topic 166 48.05
0.6
Topic 154 60.59 Topic 167 21.74
Topic 155 50.33 Topic 168 15.15
Topic 156 48.11 Topic 169 14.29 0.4
Topic 157 29.47 Topic 170 31.82
Topic 158 22.32 Topic 171 48.35 0.2
Topic 159 79.10 Topic 172 15.00
Difference
Topic 160 36.23 Topic 173 42.56 0
Topic 161 48.82 Topic 174 57.89
Topic 162 38.34 Topic 175 16.98
−0.2
Topic 163 25.35
−0.4
−0.6
−0.8
−1
151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175
Topic Identifier
80
tuchemniz TUCMIgirten4 DS-MONO-EN-CLEF2006
Overall statistics for 25 queries : Priority 4
Total number of documents over all queries Query Construction AUTOMATIC
Retrieved 25,000 Source Language English
Relevant 4,239 Topic Fields title, description
Relevant retrieved 3,172 Pooled true
Geometric Mean Average Precision 0.2695 Snowball-Porter-QE-BF
Binary Preference (BPREF) 0.4099
Interploated Recall (%) Precision Averages (%)
Domain−Specific Monolingual English track − Interpolated Recall vs Average Precision
100%
0 82.51
TUCMIgirten4
10 64.65 90%
20 58.86
80%
30 53.11
40 44.57 70%
50 34.86
Average Precision
60%
60 25.77
70 20.39 50%
80 13.89 40%
90 6.43
30%
100 0.00
Average precision (non-interpolated) for all 20%
relevant documents (averaged over queries)
35.38 10%
0%
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
Interpolated Recall
Mean Average Precision Domain−Specific Monolingual English track − Box plot of the Topics of the Experiment
Maximum 0.7824
Minimum 0.0276
First Quartile 0.1356
Second Quartile 0.3770
Third Quartile 0.5323
Interquartile range 0.3967
Mean 0.3538
Standard Deviation 0.2189
Lower Outlier Threshold 0.0276
Upper Outlier Threshold 0.7824 0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Mean Average Precision
Mean With No Outliers 0.3538
Std With No Outliers 0.2189
Domain−Specific Monolingual English track − Distribution of the Topics of the Experiment
5
Number of Topics of the Experiment
4
3
2
1
0
0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Mean Average Precision
Precision averages (%) for individual 1
Domain−Specific Monolingual English track − Comparison to Median Mean Average Precision by Topic (Topics 151 to 175)
queries TUCMIgirten4
Topic 151 55.92 Topic 164 2.76 0.8
Topic 152 57.60 Topic 165 55.85
Topic 153 41.66 Topic 166 47.03
0.6
Topic 154 72.37 Topic 167 7.14
Topic 155 49.36 Topic 168 12.38
Topic 156 52.37 Topic 169 11.59 0.4
Topic 157 21.00 Topic 170 24.54
Topic 158 29.44 Topic 171 43.14 0.2
Topic 159 78.24 Topic 172 8.35
Difference
Topic 160 13.95 Topic 173 37.70 0
Topic 161 51.82 Topic 174 55.80
Topic 162 28.05 Topic 175 11.97
−0.2
Topic 163 14.35
−0.4
−0.6
−0.8
−1
151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175
Topic Identifier
81
tuchemniz TUCMIgirten4 DS-MONO-EN-CLEF2006
Docs Cutoff Levels Precision at DCL (%)
Domain−Specific Monolingual English track − Retrieved documents vs Precision
100%
5 docs 67.20
TUCMIgirten4
10 docs 65.60 90%
15 docs 63.73
80%
20 docs 63.00
30 docs 58.80 70%
100 docs 49.36
60%
200 docs 36.96
R−Precision
500 docs 21.17 50%
1000 docs 12.69 40%
R-Precision (precision after R document retrieved,
where R = Relevant retrieved) 30%
39.34
20%
10%
0%
5 10 15 20 30 100 200 500 1000
Retrieved Documents (logarithmic scale)
Exact R-Precision Domain−Specific Monolingual English track − Box plot of the Topics of the Experiment
Maximum 0.7761
Minimum 0.0526
First Quartile 0.2445
Second Quartile 0.4256
Third Quartile 0.5165
Interquartile range 0.2720
Mean 0.3934
Standard Deviation 0.1793
Lower Outlier Threshold 0.0526
Upper Outlier Threshold 0.7761 0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Exact R−Precision
Mean With No Outliers 0.3934
Std With No Outliers 0.1793
Domain−Specific Monolingual English track − Distribution of the Topics of the Experiment
4
Number of Topics of the Experiment
3.5
3
2.5
2
1.5
1
0.5
0
0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Exact R−Precision
Precision averages (%) for individual 1
Domain−Specific Monolingual English track − Comparison to Median Mean Exact R−Precision by Topic (Topics 151 to 175)
queries TUCMIgirten4
Topic 151 54.51 Topic 164 5.26 0.8
Topic 152 55.72 Topic 165 51.67
Topic 153 44.29 Topic 166 49.61
0.6
Topic 154 64.83 Topic 167 21.74
Topic 155 51.64 Topic 168 19.19
Topic 156 50.00 Topic 169 14.29 0.4
Topic 157 29.47 Topic 170 31.82
Topic 158 40.18 Topic 171 46.70 0.2
Topic 159 77.61 Topic 172 12.50
Difference
Topic 160 27.17 Topic 173 42.56 0
Topic 161 49.34 Topic 174 57.31
Topic 162 39.94 Topic 175 20.75
−0.2
Topic 163 25.35
−0.4
−0.6
−0.8
−1
151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175
Topic Identifier
82
unine UniNEen1 DS-MONO-EN-CLEF2006
Overall statistics for 25 queries : Priority 2
Total number of documents over all queries Query Construction AUTOMATIC
Retrieved 25,000 Source Language English
Relevant 4,239 Topic Fields title, description
Relevant retrieved 3,333 Pooled true
Geometric Mean Average Precision 0.3077 data fusion of DFR & LM
Binary Preference (BPREF) 0.4797
Interploated Recall (%) Precision Averages (%)
Domain−Specific Monolingual English track − Interpolated Recall vs Average Precision
100%
0 79.65
UniNEen1
10 66.93 90%
20 61.74
80%
30 58.47
40 54.56 70%
50 48.30
Average Precision
60%
60 39.77
70 31.15 50%
80 23.49 40%
90 12.86
30%
100 0.00
Average precision (non-interpolated) for all 20%
relevant documents (averaged over queries)
42.92 10%
0%
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
Interpolated Recall
Mean Average Precision Domain−Specific Monolingual English track − Box plot of the Topics of the Experiment
Maximum 0.8233
Minimum 0.0240
First Quartile 0.1363
Second Quartile 0.5108
Third Quartile 0.6070
Interquartile range 0.4707
Mean 0.4292
Standard Deviation 0.2591
Lower Outlier Threshold 0.0240
Upper Outlier Threshold 0.8233 0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Mean Average Precision
Mean With No Outliers 0.4292
Std With No Outliers 0.2591
Domain−Specific Monolingual English track − Distribution of the Topics of the Experiment
5
Number of Topics of the Experiment
4
3
2
1
0
0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Mean Average Precision
Precision averages (%) for individual 1
Domain−Specific Monolingual English track − Comparison to Median Mean Average Precision by Topic (Topics 151 to 175)
queries UniNEen1
Topic 151 66.78 Topic 164 5.39 0.8
Topic 152 57.71 Topic 165 79.89
Topic 153 2.40 Topic 166 65.87
0.6
Topic 154 80.72 Topic 167 8.17
Topic 155 55.72 Topic 168 14.25
Topic 156 49.80 Topic 169 6.00 0.4
Topic 157 55.27 Topic 170 36.86
Topic 158 33.06 Topic 171 55.46 0.2
Topic 159 82.33 Topic 172 9.90
Difference
Topic 160 11.78 Topic 173 60.84 0
Topic 161 38.95 Topic 174 60.66
Topic 162 51.08 Topic 175 25.15
−0.2
Topic 163 58.85
−0.4
−0.6
−0.8
−1
151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175
Topic Identifier
83
unine UniNEen1 DS-MONO-EN-CLEF2006
Docs Cutoff Levels Precision at DCL (%)
Domain−Specific Monolingual English track − Retrieved documents vs Precision
100%
5 docs 68.80
UniNEen1
10 docs 64.40 90%
15 docs 65.33
80%
20 docs 64.00
30 docs 61.07 70%
100 docs 53.48
60%
200 docs 40.70
R−Precision
500 docs 22.56 50%
1000 docs 13.33 40%
R-Precision (precision after R document retrieved,
where R = Relevant retrieved) 30%
44.80
20%
10%
0%
5 10 15 20 30 100 200 500 1000
Retrieved Documents (logarithmic scale)
Exact R-Precision Domain−Specific Monolingual English track − Box plot of the Topics of the Experiment
Maximum 0.7687
Minimum 0.0870
First Quartile 0.2178
Second Quartile 0.5368
Third Quartile 0.5965
Interquartile range 0.3787
Mean 0.4480
Standard Deviation 0.2199
Lower Outlier Threshold 0.0870
Upper Outlier Threshold 0.7687 0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Exact R−Precision
Mean With No Outliers 0.4480
Std With No Outliers 0.2199
Domain−Specific Monolingual English track − Distribution of the Topics of the Experiment
5
Number of Topics of the Experiment
4
3
2
1
0
0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Exact R−Precision
Precision averages (%) for individual 1
Domain−Specific Monolingual English track − Comparison to Median Mean Exact R−Precision by Topic (Topics 151 to 175)
queries UniNEen1
Topic 151 59.61 Topic 164 13.16 0.8
Topic 152 55.43 Topic 165 71.67
Topic 153 12.46 Topic 166 59.77
0.6
Topic 154 74.15 Topic 167 8.70
Topic 155 54.61 Topic 168 19.19
Topic 156 53.77 Topic 169 11.90 0.4
Topic 157 53.68 Topic 170 50.00
Topic 158 46.43 Topic 171 57.69 0.2
Topic 159 76.87 Topic 172 10.00
Difference
Topic 160 22.64 Topic 173 59.50 0
Topic 161 43.83 Topic 174 63.74
Topic 162 50.16 Topic 175 28.30
−0.2
Topic 163 62.68
−0.4
−0.6
−0.8
−1
151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175
Topic Identifier
84
unine UniNEen2 DS-MONO-EN-CLEF2006
Overall statistics for 25 queries : Priority 3
Total number of documents over all queries Query Construction AUTOMATIC
Retrieved 25,000 Source Language English
Relevant 4,239 Topic Fields title, description
Relevant retrieved 3,325 Pooled true
Geometric Mean Average Precision 0.3248 data fusion of DFR, LM & Okapi
Binary Preference (BPREF) 0.4849
Interploated Recall (%) Precision Averages (%)
Domain−Specific Monolingual English track − Interpolated Recall vs Average Precision
100%
0 85.08
UniNEen2
10 70.48 90%
20 62.81
80%
30 57.92
40 54.23 70%
50 48.63
Average Precision
60%
60 40.16
70 31.64 50%
80 23.42 40%
90 12.32
30%
100 0.00
Average precision (non-interpolated) for all 20%
relevant documents (averaged over queries)
43.03 10%
0%
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
Interpolated Recall
Mean Average Precision Domain−Specific Monolingual English track − Box plot of the Topics of the Experiment
Maximum 0.8293
Minimum 0.0591
First Quartile 0.1520
Second Quartile 0.5107
Third Quartile 0.6062
Interquartile range 0.4542
Mean 0.4303
Standard Deviation 0.2550
Lower Outlier Threshold 0.0591
Upper Outlier Threshold 0.8293 0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Mean Average Precision
Mean With No Outliers 0.4303
Std With No Outliers 0.2550
Domain−Specific Monolingual English track − Distribution of the Topics of the Experiment
5
Number of Topics of the Experiment
4
3
2
1
0
0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Mean Average Precision
Precision averages (%) for individual 1
Domain−Specific Monolingual English track − Comparison to Median Mean Average Precision by Topic (Topics 151 to 175)
queries UniNEen2
Topic 151 66.61 Topic 164 5.91 0.8
Topic 152 59.82 Topic 165 82.93
Topic 153 6.53 Topic 166 67.62
0.6
Topic 154 79.89 Topic 167 8.34
Topic 155 53.78 Topic 168 12.95
Topic 156 50.74 Topic 169 6.64 0.4
Topic 157 51.92 Topic 170 35.04
Topic 158 31.76 Topic 171 54.23 0.2
Topic 159 81.95 Topic 172 10.61
Difference
Topic 160 15.94 Topic 173 60.10 0
Topic 161 32.72 Topic 174 62.19
Topic 162 51.07 Topic 175 29.69
−0.2
Topic 163 56.73
−0.4
−0.6
−0.8
−1
151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175
Topic Identifier
85
unine UniNEen2 DS-MONO-EN-CLEF2006
Docs Cutoff Levels Precision at DCL (%)
Domain−Specific Monolingual English track − Retrieved documents vs Precision
100%
5 docs 69.60
UniNEen2
10 docs 69.60 90%
15 docs 68.00
80%
20 docs 66.20
30 docs 63.87 70%
100 docs 54.36
60%
200 docs 41.30
R−Precision
500 docs 22.50 50%
1000 docs 13.30 40%
R-Precision (precision after R document retrieved,
where R = Relevant retrieved) 30%
45.22
20%
10%
0%
5 10 15 20 30 100 200 500 1000
Retrieved Documents (logarithmic scale)
Exact R-Precision Domain−Specific Monolingual English track − Box plot of the Topics of the Experiment
Maximum 0.7537
Minimum 0.1000
First Quartile 0.2291
Second Quartile 0.5158
Third Quartile 0.5979
Interquartile range 0.3688
Mean 0.4522
Standard Deviation 0.2130
Lower Outlier Threshold 0.1000
Upper Outlier Threshold 0.7537 0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Exact R−Precision
Mean With No Outliers 0.4522
Std With No Outliers 0.2130
Domain−Specific Monolingual English track − Distribution of the Topics of the Experiment
5
Number of Topics of the Experiment
4
3
2
1
0
0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Exact R−Precision
Precision averages (%) for individual 1
Domain−Specific Monolingual English track − Comparison to Median Mean Exact R−Precision by Topic (Topics 151 to 175)
queries UniNEen2
Topic 151 58.82 Topic 164 11.84 0.8
Topic 152 59.53 Topic 165 73.33
Topic 153 17.30 Topic 166 62.11
0.6
Topic 154 72.46 Topic 167 13.04
Topic 155 52.63 Topic 168 19.19
Topic 156 53.77 Topic 169 14.29 0.4
Topic 157 51.58 Topic 170 50.00
Topic 158 44.64 Topic 171 58.79 0.2
Topic 159 75.37 Topic 172 10.00
Difference
Topic 160 24.15 Topic 173 58.26 0
Topic 161 40.42 Topic 174 66.67
Topic 162 51.44 Topic 175 30.19
−0.2
Topic 163 60.56
−0.4
−0.6
−0.8
−1
151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175
Topic Identifier
86
unine UniNEen3 DS-MONO-EN-CLEF2006
Overall statistics for 25 queries : Priority 1
Total number of documents over all queries Query Construction AUTOMATIC
Retrieved 25,000 Source Language English
Relevant 4,239 Topic Fields title, description, narrative
Relevant retrieved 3,396 Pooled true
Geometric Mean Average Precision 0.3806 data fusion of DFR, LM & Okapi
Binary Preference (BPREF) 0.5063
Interploated Recall (%) Precision Averages (%)
Domain−Specific Monolingual English track − Interpolated Recall vs Average Precision
100%
0 88.72
UniNEen3
10 75.70 90%
20 68.18
80%
30 62.82
40 56.57 70%
50 49.95
Average Precision
60%
60 40.01
70 32.60 50%
80 23.99 40%
90 13.41
30%
100 0.19
Average precision (non-interpolated) for all 20%
relevant documents (averaged over queries)
45.76 10%
0%
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
Interpolated Recall
Mean Average Precision Domain−Specific Monolingual English track − Box plot of the Topics of the Experiment
Maximum 0.8538
Minimum 0.0491
First Quartile 0.2818
Second Quartile 0.4910
Third Quartile 0.6269
Interquartile range 0.3452
Mean 0.4576
Standard Deviation 0.2318
Lower Outlier Threshold 0.0491
Upper Outlier Threshold 0.8538 0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Mean Average Precision
Mean With No Outliers 0.4576
Std With No Outliers 0.2318
Domain−Specific Monolingual English track − Distribution of the Topics of the Experiment
4
Number of Topics of the Experiment
3.5
3
2.5
2
1.5
1
0.5
0
0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Mean Average Precision
Precision averages (%) for individual 1
Domain−Specific Monolingual English track − Comparison to Median Mean Average Precision by Topic (Topics 151 to 175)
queries UniNEen3
Topic 151 68.21 Topic 164 4.91 0.8
Topic 152 49.10 Topic 165 85.38
Topic 153 37.89 Topic 166 62.61
0.6
Topic 154 81.69 Topic 167 21.34
Topic 155 62.94 Topic 168 12.04
Topic 156 49.84 Topic 169 15.63 0.4
Topic 157 53.52 Topic 170 33.08
Topic 158 34.07 Topic 171 58.65 0.2
Topic 159 81.63 Topic 172 13.82
Difference
Topic 160 19.81 Topic 173 63.91 0
Topic 161 35.87 Topic 174 61.74
Topic 162 48.90 Topic 175 30.45
−0.2
Topic 163 57.00
−0.4
−0.6
−0.8
−1
151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175
Topic Identifier
87
unine UniNEen3 DS-MONO-EN-CLEF2006
Docs Cutoff Levels Precision at DCL (%)
Domain−Specific Monolingual English track − Retrieved documents vs Precision
100%
5 docs 74.40
UniNEen3
10 docs 73.20 90%
15 docs 72.53
80%
20 docs 70.80
30 docs 68.13 70%
100 docs 57.28
60%
200 docs 43.28
R−Precision
500 docs 23.34 50%
1000 docs 13.58 40%
R-Precision (precision after R document retrieved,
where R = Relevant retrieved) 30%
47.71
20%
10%
0%
5 10 15 20 30 100 200 500 1000
Retrieved Documents (logarithmic scale)
Exact R-Precision Domain−Specific Monolingual English track − Box plot of the Topics of the Experiment
Maximum 0.7761
Minimum 0.1053
First Quartile 0.3274
Second Quartile 0.5158
Third Quartile 0.6054
Interquartile range 0.2781
Mean 0.4771
Standard Deviation 0.1947
Lower Outlier Threshold 0.1053
Upper Outlier Threshold 0.7761 0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Exact R−Precision
Mean With No Outliers 0.4771
Std With No Outliers 0.1947
Domain−Specific Monolingual English track − Distribution of the Topics of the Experiment
4
Number of Topics of the Experiment
3.5
3
2.5
2
1.5
1
0.5
0
0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Exact R−Precision
Precision averages (%) for individual 1
Domain−Specific Monolingual English track − Comparison to Median Mean Exact R−Precision by Topic (Topics 151 to 175)
queries UniNEen3
Topic 151 61.18 Topic 164 10.53 0.8
Topic 152 53.08 Topic 165 76.67
Topic 153 40.48 Topic 166 63.28
0.6
Topic 154 74.58 Topic 167 21.74
Topic 155 57.57 Topic 168 19.19
Topic 156 59.43 Topic 169 23.81 0.4
Topic 157 51.58 Topic 170 50.00
Topic 158 43.75 Topic 171 59.89 0.2
Topic 159 77.61 Topic 172 12.50
Difference
Topic 160 33.58 Topic 173 60.33 0
Topic 161 41.73 Topic 174 63.16
Topic 162 49.20 Topic 175 30.19
−0.2
Topic 163 57.75
−0.4
−0.6
−0.8
−1
151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175
Topic Identifier
88
berkeley BERK_MO_DE_T2FB DS-MONO-DE-CLEF2006
Overall statistics for 25 queries : Priority 1
Total number of documents over all queries Query Construction AUTOMATIC
Retrieved 25,000 Source Language German
Relevant 3,759 Topic Fields title, description
Relevant retrieved 2,867 Pooled true
Geometric Mean Average Precision 0.1440 Berkeley Trec2 algorithm with blind feedback - no
Binary Preference (BPREF) 0.4401 decompounding
Interploated Recall (%) Precision Averages (%)
Domain−Specific Monolingual German track − Interpolated Recall vs Average Precision
100%
0 79.80
BERK_MO_DE_T2FB
10 70.35 90%
20 60.44
80%
30 53.91
40 45.66 70%
50 39.88
Average Precision
60%
60 33.65
70 25.74 50%
80 18.32 40%
90 10.43
30%
100 0.70
Average precision (non-interpolated) for all 20%
relevant documents (averaged over queries)
39.17 10%
0%
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
Interpolated Recall
Mean Average Precision Domain−Specific Monolingual German track − Box plot of the Topics of the Experiment
Maximum 0.7969
Minimum 0.0000
First Quartile 0.1124
Second Quartile 0.4405
Third Quartile 0.5787
Interquartile range 0.4663
Mean 0.3917
Standard Deviation 0.2582
Lower Outlier Threshold 0.0000
Upper Outlier Threshold 0.7969 0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Mean Average Precision
Mean With No Outliers 0.3917
Std With No Outliers 0.2582
Domain−Specific Monolingual German track − Distribution of the Topics of the Experiment
3
Number of Topics of the Experiment
2.5
2
1.5
1
0.5
0
0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Mean Average Precision
Precision averages (%) for individual 1
Domain−Specific Monolingual German track − Comparison to Median Mean Average Precision by Topic (Topics 151 to 175)
queries BERK_MO_DE_T2FB
Topic 151 69.56 Topic 164 4.75 0.8
Topic 152 0.00 Topic 165 21.64
Topic 153 53.33 Topic 166 47.28
0.6
Topic 154 79.69 Topic 167 8.91
Topic 155 70.24 Topic 168 40.57
Topic 156 48.35 Topic 169 8.89 0.4
Topic 157 79.54 Topic 170 50.97
Topic 158 12.02 Topic 171 44.05 0.2
Topic 159 69.16 Topic 172 7.77
Difference
Topic 160 0.00 Topic 173 63.39 0
Topic 161 51.92 Topic 174 25.57
Topic 162 56.03 Topic 175 31.41
−0.2
Topic 163 34.23
−0.4
−0.6
−0.8
−1
151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175
Topic Identifier
89
berkeley BERK_MO_DE_T2FB DS-MONO-DE-CLEF2006
Docs Cutoff Levels Precision at DCL (%)
Domain−Specific Monolingual German track − Retrieved documents vs Precision
100%
5 docs 70.40
BERK_MO_DE_T2FB
10 docs 67.60 90%
15 docs 64.53
80%
20 docs 62.60
30 docs 60.13 70%
100 docs 46.84
60%
200 docs 33.80
R−Precision
500 docs 19.38 50%
1000 docs 11.47 40%
R-Precision (precision after R document retrieved,
where R = Relevant retrieved) 30%
39.05
20%
10%
0%
5 10 15 20 30 100 200 500 1000
Retrieved Documents (logarithmic scale)
Exact R-Precision Domain−Specific Monolingual German track − Box plot of the Topics of the Experiment
Maximum 0.6971
Minimum 0.0000
First Quartile 0.1977
Second Quartile 0.4545
Third Quartile 0.5581
Interquartile range 0.3604
Mean 0.3905
Standard Deviation 0.2253
Lower Outlier Threshold 0.0000
Upper Outlier Threshold 0.6971 0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Exact R−Precision
Mean With No Outliers 0.3905
Std With No Outliers 0.2253
Domain−Specific Monolingual German track − Distribution of the Topics of the Experiment
5
Number of Topics of the Experiment
4
3
2
1
0
0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Exact R−Precision
Precision averages (%) for individual 1
Domain−Specific Monolingual German track − Comparison to Median Mean Exact R−Precision by Topic (Topics 151 to 175)
queries BERK_MO_DE_T2FB
Topic 151 66.14 Topic 164 7.69 0.8
Topic 152 0.00 Topic 165 21.92
Topic 153 50.67 Topic 166 50.54
0.6
Topic 154 69.71 Topic 167 13.33
Topic 155 63.73 Topic 168 43.54
Topic 156 47.31 Topic 169 9.09 0.4
Topic 157 69.23 Topic 170 51.28
Topic 158 23.60 Topic 171 45.45 0.2
Topic 159 61.11 Topic 172 9.09
Difference
Topic 160 0.00 Topic 173 60.32 0
Topic 161 53.16 Topic 174 35.23
Topic 162 54.31 Topic 175 34.38
−0.2
Topic 163 35.41
−0.4
−0.6
−0.8
−1
151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175
Topic Identifier
90
hagen FUHggyynbfl500R DS-MONO-DE-CLEF2006
Overall statistics for 25 queries : Priority 1
Total number of documents over all queries Query Construction AUTOMATIC
Retrieved 25,000 Source Language German
Relevant 3,759 Topic Fields title, description
Relevant retrieved 2,920 Pooled true
Geometric Mean Average Precision 0.2869 First run
Binary Preference (BPREF) 0.4025
Interploated Recall (%) Precision Averages (%)
Domain−Specific Monolingual German track − Interpolated Recall vs Average Precision
100%
0 87.38
FUHggyynbfl500R
10 65.98 90%
20 55.89
80%
30 47.91
40 42.68 70%
50 35.82
Average Precision
60%
60 29.81
70 20.94 50%
80 12.22 40%
90 5.95
30%
100 0.15
Average precision (non-interpolated) for all 20%
relevant documents (averaged over queries)
35.39 10%
0%
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
Interpolated Recall
Mean Average Precision Domain−Specific Monolingual German track − Box plot of the Topics of the Experiment
Maximum 0.7362
Minimum 0.0696
First Quartile 0.1607
Second Quartile 0.3133
Third Quartile 0.5044
Interquartile range 0.3437
Mean 0.3539
Standard Deviation 0.2110
Lower Outlier Threshold 0.0696
Upper Outlier Threshold 0.7362 0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Mean Average Precision
Mean With No Outliers 0.3539
Std With No Outliers 0.2110
Domain−Specific Monolingual German track − Distribution of the Topics of the Experiment
3
Number of Topics of the Experiment
2.5
2
1.5
1
0.5
0
0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Mean Average Precision
Precision averages (%) for individual 1
Domain−Specific Monolingual German track − Comparison to Median Mean Average Precision by Topic (Topics 151 to 175)
queries FUHggyynbfl500R
Topic 151 73.46 Topic 164 13.60 0.8
Topic 152 51.75 Topic 165 73.62
Topic 153 30.71 Topic 166 50.01
0.6
Topic 154 64.19 Topic 167 6.96
Topic 155 24.91 Topic 168 31.58
Topic 156 14.34 Topic 169 25.68 0.4
Topic 157 12.90 Topic 170 46.94
Topic 158 16.65 Topic 171 31.33 0.2
Topic 159 46.79 Topic 172 8.15
Difference
Topic 160 55.88 Topic 173 67.70 0
Topic 161 37.49 Topic 174 46.39
Topic 162 24.76 Topic 175 9.86
−0.2
Topic 163 19.02
−0.4
−0.6
−0.8
−1
151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175
Topic Identifier
91
hagen FUHggyynbfl500R DS-MONO-DE-CLEF2006
Docs Cutoff Levels Precision at DCL (%)
Domain−Specific Monolingual German track − Retrieved documents vs Precision
100%
5 docs 72.00
FUHggyynbfl500R
10 docs 67.60 90%
15 docs 64.80
80%
20 docs 61.80
30 docs 58.80 70%
100 docs 43.36
60%
200 docs 33.84
R−Precision
500 docs 19.63 50%
1000 docs 11.68 40%
R-Precision (precision after R document retrieved,
where R = Relevant retrieved) 30%
39.88
20%
10%
0%
5 10 15 20 30 100 200 500 1000
Retrieved Documents (logarithmic scale)
Exact R-Precision Domain−Specific Monolingual German track − Box plot of the Topics of the Experiment
Maximum 0.6890
Minimum 0.1692
First Quartile 0.2419
Second Quartile 0.3788
Third Quartile 0.5389
Interquartile range 0.2970
Mean 0.3988
Standard Deviation 0.1739
Lower Outlier Threshold 0.1692
Upper Outlier Threshold 0.6890 0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Exact R−Precision
Mean With No Outliers 0.3988
Std With No Outliers 0.1739
Domain−Specific Monolingual German track − Distribution of the Topics of the Experiment
6
Number of Topics of the Experiment
5
4
3
2
1
0
0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Exact R−Precision
Precision averages (%) for individual 1
Domain−Specific Monolingual German track − Comparison to Median Mean Exact R−Precision by Topic (Topics 151 to 175)
queries FUHggyynbfl500R
Topic 151 68.90 Topic 164 19.23 0.8
Topic 152 50.96 Topic 165 68.49
Topic 153 37.33 Topic 166 54.03
0.6
Topic 154 60.95 Topic 167 17.78
Topic 155 32.64 Topic 168 43.54
Topic 156 17.20 Topic 169 27.27 0.4
Topic 157 16.92 Topic 170 53.85
Topic 158 25.84 Topic 171 37.88 0.2
Topic 159 49.07 Topic 172 18.18
Difference
Topic 160 56.25 Topic 173 62.70 0
Topic 161 44.30 Topic 174 53.41
Topic 162 36.21 Topic 175 17.19
−0.2
Topic 163 26.85
−0.4
−0.6
−0.8
−1
151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175
Topic Identifier
92
hagen FUHggyynbfl500 DS-MONO-DE-CLEF2006
Overall statistics for 25 queries : Priority 2
Total number of documents over all queries Query Construction AUTOMATIC
Retrieved 25,000 Source Language German
Relevant 3,759 Topic Fields title, description
Relevant retrieved 2,920 Pooled true
Geometric Mean Average Precision 0.2835 Second run
Binary Preference (BPREF) 0.4001
Interploated Recall (%) Precision Averages (%)
Domain−Specific Monolingual German track − Interpolated Recall vs Average Precision
100%
0 87.56
FUHggyynbfl500
10 64.75 90%
20 55.53
80%
30 47.53
40 42.35 70%
50 35.84
Average Precision
60%
60 29.59
70 20.91 50%
80 12.07 40%
90 5.92
30%
100 0.15
Average precision (non-interpolated) for all 20%
relevant documents (averaged over queries)
35.25 10%
0%
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
Interpolated Recall
Mean Average Precision Domain−Specific Monolingual German track − Box plot of the Topics of the Experiment
Maximum 0.7362
Minimum 0.0538
First Quartile 0.1616
Second Quartile 0.3256
Third Quartile 0.5002
Interquartile range 0.3386
Mean 0.3525
Standard Deviation 0.2106
Lower Outlier Threshold 0.0538
Upper Outlier Threshold 0.7362 0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Mean Average Precision
Mean With No Outliers 0.3525
Std With No Outliers 0.2106
Domain−Specific Monolingual German track − Distribution of the Topics of the Experiment
3
Number of Topics of the Experiment
2.5
2
1.5
1
0.5
0
0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Mean Average Precision
Precision averages (%) for individual 1
Domain−Specific Monolingual German track − Comparison to Median Mean Average Precision by Topic (Topics 151 to 175)
queries FUHggyynbfl500
Topic 151 73.04 Topic 164 13.20 0.8
Topic 152 49.01 Topic 165 73.62
Topic 153 37.00 Topic 166 53.03
0.6
Topic 154 63.55 Topic 167 5.38
Topic 155 24.72 Topic 168 32.21
Topic 156 15.79 Topic 169 24.95 0.4
Topic 157 12.65 Topic 170 41.24
Topic 158 16.28 Topic 171 32.56 0.2
Topic 159 46.99 Topic 172 7.82
Difference
Topic 160 55.73 Topic 173 67.70 0
Topic 161 37.49 Topic 174 45.64
Topic 162 23.67 Topic 175 10.19
−0.2
Topic 163 17.81
−0.4
−0.6
−0.8
−1
151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175
Topic Identifier
93
hagen FUHggyynbfl500 DS-MONO-DE-CLEF2006
Docs Cutoff Levels Precision at DCL (%)
Domain−Specific Monolingual German track − Retrieved documents vs Precision
100%
5 docs 72.00
FUHggyynbfl500
10 docs 68.40 90%
15 docs 64.00
80%
20 docs 60.60
30 docs 58.40 70%
100 docs 43.64
60%
200 docs 33.96
R−Precision
500 docs 19.70 50%
1000 docs 11.68 40%
R-Precision (precision after R document retrieved,
where R = Relevant retrieved) 30%
39.41
20%
10%
0%
5 10 15 20 30 100 200 500 1000
Retrieved Documents (logarithmic scale)
Exact R-Precision Domain−Specific Monolingual German track − Box plot of the Topics of the Experiment
Maximum 0.6850
Minimum 0.1333
First Quartile 0.2449
Second Quartile 0.4000
Third Quartile 0.5369
Interquartile range 0.2920
Mean 0.3941
Standard Deviation 0.1756
Lower Outlier Threshold 0.1333
Upper Outlier Threshold 0.6850 0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Exact R−Precision
Mean With No Outliers 0.3941
Std With No Outliers 0.1756
Domain−Specific Monolingual German track − Distribution of the Topics of the Experiment
3
Number of Topics of the Experiment
2.5
2
1.5
1
0.5
0
0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Exact R−Precision
Precision averages (%) for individual 1
Domain−Specific Monolingual German track − Comparison to Median Mean Exact R−Precision by Topic (Topics 151 to 175)
queries FUHggyynbfl500
Topic 151 68.50 Topic 164 17.31 0.8
Topic 152 48.08 Topic 165 68.49
Topic 153 40.00 Topic 166 56.99
0.6
Topic 154 60.22 Topic 167 13.33
Topic 155 32.64 Topic 168 41.50
Topic 156 20.43 Topic 169 27.27 0.4
Topic 157 18.46 Topic 170 46.15
Topic 158 25.84 Topic 171 38.89 0.2
Topic 159 52.78 Topic 172 13.64
Difference
Topic 160 56.25 Topic 173 62.70 0
Topic 161 44.30 Topic 174 52.84
Topic 162 32.76 Topic 175 18.75
−0.2
Topic 163 27.24
−0.4
−0.6
−0.8
−1
151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175
Topic Identifier
94
hagen FUHggyydbfl500R DS-MONO-DE-CLEF2006
Overall statistics for 25 queries : Priority 3
Total number of documents over all queries Query Construction AUTOMATIC
Retrieved 25,000 Source Language German
Relevant 3,759 Topic Fields title, description
Relevant retrieved 2,780 Pooled true
Geometric Mean Average Precision 0.2466 Third run
Binary Preference (BPREF) 0.3654
Interploated Recall (%) Precision Averages (%)
Domain−Specific Monolingual German track − Interpolated Recall vs Average Precision
100%
0 88.30
FUHggyydbfl500R
10 60.60 90%
20 50.56
80%
30 43.15
40 37.82 70%
50 31.65
Average Precision
60%
60 26.01
70 18.77 50%
80 12.70 40%
90 4.97
30%
100 0.58
Average precision (non-interpolated) for all 20%
relevant documents (averaged over queries)
31.79 10%
0%
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
Interpolated Recall
Mean Average Precision Domain−Specific Monolingual German track − Box plot of the Topics of the Experiment
Maximum 0.7053
Minimum 0.0703
First Quartile 0.1200
Second Quartile 0.2815
Third Quartile 0.5436
Interquartile range 0.4235
Mean 0.3179
Standard Deviation 0.2125
Lower Outlier Threshold 0.0703
Upper Outlier Threshold 0.7053 0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Mean Average Precision
Mean With No Outliers 0.3179
Std With No Outliers 0.2125
Domain−Specific Monolingual German track − Distribution of the Topics of the Experiment
5
Number of Topics of the Experiment
4
3
2
1
0
0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Mean Average Precision
Precision averages (%) for individual 1
Domain−Specific Monolingual German track − Comparison to Median Mean Average Precision by Topic (Topics 151 to 175)
queries FUHggyydbfl500R
Topic 151 70.53 Topic 164 12.00 0.8
Topic 152 9.81 Topic 165 36.10
Topic 153 32.60 Topic 166 61.62
0.6
Topic 154 58.98 Topic 167 8.85
Topic 155 28.15 Topic 168 33.02
Topic 156 17.01 Topic 169 18.74 0.4
Topic 157 13.95 Topic 170 53.68
Topic 158 8.70 Topic 171 38.54 0.2
Topic 159 56.37 Topic 172 7.03
Difference
Topic 160 62.11 Topic 173 60.62 0
Topic 161 12.00 Topic 174 48.20
Topic 162 22.11 Topic 175 10.86
−0.2
Topic 163 13.08
−0.4
−0.6
−0.8
−1
151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175
Topic Identifier
95
hagen FUHggyydbfl500R DS-MONO-DE-CLEF2006
Docs Cutoff Levels Precision at DCL (%)
Domain−Specific Monolingual German track − Retrieved documents vs Precision
100%
5 docs 64.80
FUHggyydbfl500R
10 docs 62.00 90%
15 docs 59.47
80%
20 docs 55.80
30 docs 51.87 70%
100 docs 40.96
60%
200 docs 31.64
R−Precision
500 docs 18.21 50%
1000 docs 11.12 40%
R-Precision (precision after R document retrieved,
where R = Relevant retrieved) 30%
36.15
20%
10%
0%
5 10 15 20 30 100 200 500 1000
Retrieved Documents (logarithmic scale)
Exact R-Precision Domain−Specific Monolingual German track − Box plot of the Topics of the Experiment
Maximum 0.6614
Minimum 0.1154
First Quartile 0.2078
Second Quartile 0.3600
Third Quartile 0.5702
Interquartile range 0.3624
Mean 0.3615
Standard Deviation 0.1835
Lower Outlier Threshold 0.1154
Upper Outlier Threshold 0.6614 0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Exact R−Precision
Mean With No Outliers 0.3615
Std With No Outliers 0.1835
Domain−Specific Monolingual German track − Distribution of the Topics of the Experiment
6
Number of Topics of the Experiment
5
4
3
2
1
0
0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Exact R−Precision
Precision averages (%) for individual 1
Domain−Specific Monolingual German track − Comparison to Median Mean Exact R−Precision by Topic (Topics 151 to 175)
queries FUHggyydbfl500R
Topic 151 66.14 Topic 164 11.54 0.8
Topic 152 15.38 Topic 165 43.84
Topic 153 36.00 Topic 166 59.41
0.6
Topic 154 59.49 Topic 167 13.33
Topic 155 36.79 Topic 168 42.18
Topic 156 22.58 Topic 169 27.27 0.4
Topic 157 23.08 Topic 170 53.85
Topic 158 15.73 Topic 171 41.92 0.2
Topic 159 58.33 Topic 172 13.64
Difference
Topic 160 57.64 Topic 173 58.33 0
Topic 161 22.47 Topic 174 56.82
Topic 162 30.60 Topic 175 14.06
−0.2
Topic 163 23.35
−0.4
−0.6
−0.8
−1
151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175
Topic Identifier
96
hagen FUHggyydbfl500 DS-MONO-DE-CLEF2006
Overall statistics for 25 queries : Priority 4
Total number of documents over all queries Query Construction AUTOMATIC
Retrieved 25,000 Source Language German
Relevant 3,759 Topic Fields title, description
Relevant retrieved 2,780 Pooled true
Geometric Mean Average Precision 0.2463 Fourth run
Binary Preference (BPREF) 0.3668
Interploated Recall (%) Precision Averages (%)
Domain−Specific Monolingual German track − Interpolated Recall vs Average Precision
100%
0 88.32
FUHggyydbfl500
10 59.75 90%
20 51.51
80%
30 43.97
40 38.38 70%
50 31.85
Average Precision
60%
60 26.80
70 19.14 50%
80 13.20 40%
90 5.04
30%
100 0.58
Average precision (non-interpolated) for all 20%
relevant documents (averaged over queries)
32.05 10%
0%
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
Interpolated Recall
Mean Average Precision Domain−Specific Monolingual German track − Box plot of the Topics of the Experiment
Maximum 0.7077
Minimum 0.0685
First Quartile 0.1191
Second Quartile 0.2809
Third Quartile 0.5077
Interquartile range 0.3886
Mean 0.3205
Standard Deviation 0.2134
Lower Outlier Threshold 0.0685
Upper Outlier Threshold 0.7077 0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Mean Average Precision
Mean With No Outliers 0.3205
Std With No Outliers 0.2134
Domain−Specific Monolingual German track − Distribution of the Topics of the Experiment
5
Number of Topics of the Experiment
4
3
2
1
0
0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Mean Average Precision
Precision averages (%) for individual 1
Domain−Specific Monolingual German track − Comparison to Median Mean Average Precision by Topic (Topics 151 to 175)
queries FUHggyydbfl500
Topic 151 70.77 Topic 164 11.63 0.8
Topic 152 8.16 Topic 165 36.10
Topic 153 40.89 Topic 166 64.36
0.6
Topic 154 57.64 Topic 167 6.85
Topic 155 28.09 Topic 168 34.62
Topic 156 19.12 Topic 169 18.76 0.4
Topic 157 13.83 Topic 170 48.85
Topic 158 8.85 Topic 171 39.94 0.2
Topic 159 56.53 Topic 172 7.01
Difference
Topic 160 61.96 Topic 173 60.62 0
Topic 161 12.00 Topic 174 47.48
Topic 162 21.85 Topic 175 10.80
−0.2
Topic 163 14.58
−0.4
−0.6
−0.8
−1
151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175
Topic Identifier
97
hagen FUHggyydbfl500 DS-MONO-DE-CLEF2006
Docs Cutoff Levels Precision at DCL (%)
Domain−Specific Monolingual German track − Retrieved documents vs Precision
100%
5 docs 64.80
FUHggyydbfl500
10 docs 60.80 90%
15 docs 59.47
80%
20 docs 55.00
30 docs 53.07 70%
100 docs 41.40
60%
200 docs 31.62
R−Precision
500 docs 18.46 50%
1000 docs 11.12 40%
R-Precision (precision after R document retrieved,
where R = Relevant retrieved) 30%
36.31
20%
10%
0%
5 10 15 20 30 100 200 500 1000
Retrieved Documents (logarithmic scale)
Exact R-Precision Domain−Specific Monolingual German track − Box plot of the Topics of the Experiment
Maximum 0.6575
Minimum 0.0889
First Quartile 0.2078
Second Quartile 0.3679
Third Quartile 0.5617
Interquartile range 0.3539
Mean 0.3631
Standard Deviation 0.1888
Lower Outlier Threshold 0.0889
Upper Outlier Threshold 0.6575 0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Exact R−Precision
Mean With No Outliers 0.3631
Std With No Outliers 0.1888
Domain−Specific Monolingual German track − Distribution of the Topics of the Experiment
5
Number of Topics of the Experiment
4
3
2
1
0
0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Exact R−Precision
Precision averages (%) for individual 1
Domain−Specific Monolingual German track − Comparison to Median Mean Exact R−Precision by Topic (Topics 151 to 175)
queries FUHggyydbfl500
Topic 151 65.75 Topic 164 11.54 0.8
Topic 152 14.42 Topic 165 43.84
Topic 153 45.33 Topic 166 63.17
0.6
Topic 154 58.39 Topic 167 8.89
Topic 155 36.79 Topic 168 44.22
Topic 156 24.73 Topic 169 27.27 0.4
Topic 157 23.08 Topic 170 48.72
Topic 158 15.73 Topic 171 42.93 0.2
Topic 159 59.26 Topic 172 9.09
Difference
Topic 160 57.64 Topic 173 58.33 0
Topic 161 22.47 Topic 174 55.68
Topic 162 31.03 Topic 175 14.06
−0.2
Topic 163 25.29
−0.4
−0.6
−0.8
−1
151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175
Topic Identifier
98
hagen FUHggyydbfl102 DS-MONO-DE-CLEF2006
Overall statistics for 25 queries : Priority 5
Total number of documents over all queries Query Construction AUTOMATIC
Retrieved 25,000 Source Language German
Relevant 3,759 Topic Fields title, description
Relevant retrieved 2,444 Pooled true
Geometric Mean Average Precision 0.1823 Fifth run
Binary Preference (BPREF) 0.3392
Interploated Recall (%) Precision Averages (%)
Domain−Specific Monolingual German track − Interpolated Recall vs Average Precision
100%
0 80.33
FUHggyydbfl102
10 54.08 90%
20 46.49
80%
30 41.25
40 35.15 70%
50 27.30
Average Precision
60%
60 19.71
70 14.61 50%
80 8.16 40%
90 2.44
30%
100 0.08
Average precision (non-interpolated) for all 20%
relevant documents (averaged over queries)
27.77 10%
0%
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
Interpolated Recall
Mean Average Precision Domain−Specific Monolingual German track − Box plot of the Topics of the Experiment
Maximum 0.6214
Minimum 0.0148
First Quartile 0.0698
Second Quartile 0.2078
Third Quartile 0.5057
Interquartile range 0.4359
Mean 0.2777
Standard Deviation 0.2138
Lower Outlier Threshold 0.0148
Upper Outlier Threshold 0.6214 0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Mean Average Precision
Mean With No Outliers 0.2777
Std With No Outliers 0.2138
Domain−Specific Monolingual German track − Distribution of the Topics of the Experiment
5
Number of Topics of the Experiment
4
3
2
1
0
0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Mean Average Precision
Precision averages (%) for individual 1
Domain−Specific Monolingual German track − Comparison to Median Mean Average Precision by Topic (Topics 151 to 175)
queries FUHggyydbfl102
Topic 151 60.10 Topic 164 16.48 0.8
Topic 152 4.57 Topic 165 28.70
Topic 153 5.47 Topic 166 40.46
0.6
Topic 154 50.48 Topic 167 28.62
Topic 155 40.01 Topic 168 20.78
Topic 156 1.48 Topic 169 6.61 0.4
Topic 157 10.87 Topic 170 54.99
Topic 158 4.72 Topic 171 55.30 0.2
Topic 159 50.84 Topic 172 8.43
Difference
Topic 160 53.72 Topic 173 47.95 0
Topic 161 6.67 Topic 174 62.14
Topic 162 14.12 Topic 175 7.08
−0.2
Topic 163 13.72
−0.4
−0.6
−0.8
−1
151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175
Topic Identifier
99
hagen FUHggyydbfl102 DS-MONO-DE-CLEF2006
Docs Cutoff Levels Precision at DCL (%)
Domain−Specific Monolingual German track − Retrieved documents vs Precision
100%
5 docs 61.60
FUHggyydbfl102
10 docs 55.20 90%
15 docs 52.00
80%
20 docs 50.40
30 docs 47.33 70%
100 docs 38.92
60%
200 docs 28.68
R−Precision
500 docs 15.91 50%
1000 docs 9.78 40%
R-Precision (precision after R document retrieved,
where R = Relevant retrieved) 30%
33.33
20%
10%
0%
5 10 15 20 30 100 200 500 1000
Retrieved Documents (logarithmic scale)
Exact R-Precision Domain−Specific Monolingual German track − Box plot of the Topics of the Experiment
Maximum 0.6932
Minimum 0.0800
First Quartile 0.1344
Second Quartile 0.3537
Third Quartile 0.5190
Interquartile range 0.3847
Mean 0.3333
Standard Deviation 0.1960
Lower Outlier Threshold 0.0800
Upper Outlier Threshold 0.6932 0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Exact R−Precision
Mean With No Outliers 0.3333
Std With No Outliers 0.1960
Domain−Specific Monolingual German track − Distribution of the Topics of the Experiment
4
Number of Topics of the Experiment
3.5
3
2.5
2
1.5
1
0.5
0
0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Exact R−Precision
Precision averages (%) for individual 1
Domain−Specific Monolingual German track − Comparison to Median Mean Exact R−Precision by Topic (Topics 151 to 175)
queries FUHggyydbfl102
Topic 151 57.87 Topic 164 19.23 0.8
Topic 152 10.10 Topic 165 39.73
Topic 153 8.00 Topic 166 42.74
0.6
Topic 154 57.66 Topic 167 35.56
Topic 155 43.52 Topic 168 35.37
Topic 156 9.68 Topic 169 9.09 0.4
Topic 157 20.00 Topic 170 56.41
Topic 158 13.48 Topic 171 54.04 0.2
Topic 159 49.07 Topic 172 18.18
Difference
Topic 160 58.33 Topic 173 51.19 0
Topic 161 13.29 Topic 174 69.32
Topic 162 21.98 Topic 175 12.50
−0.2
Topic 163 26.85
−0.4
−0.6
−0.8
−1
151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175
Topic Identifier
100
tuchemniz TUCMIgirtde2 DS-MONO-DE-CLEF2006
Overall statistics for 25 queries : Priority 2
Total number of documents over all queries Query Construction AUTOMATIC
Retrieved 25,000 Source Language German
Relevant 3,759 Topic Fields title, description
Relevant retrieved 3,139 Pooled true
Geometric Mean Average Precision 0.4138 Snowball-Ger2-QE-BF2-DC
Binary Preference (BPREF) 0.5617
Interploated Recall (%) Precision Averages (%)
Domain−Specific Monolingual German track − Interpolated Recall vs Average Precision
100%
0 93.44
TUCMIgirtde2
10 84.79 90%
20 80.36
80%
30 75.27
40 65.72 70%
50 55.47
Average Precision
60%
60 45.89
70 36.59 50%
80 26.32 40%
90 11.67
30%
100 0.59
Average precision (non-interpolated) for all 20%
relevant documents (averaged over queries)
52.02 10%
0%
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
Interpolated Recall
Mean Average Precision Domain−Specific Monolingual German track − Box plot of the Topics of the Experiment
Maximum 0.8573
Minimum 0.0028
First Quartile 0.4535
Second Quartile 0.5312
Third Quartile 0.6343
Interquartile range 0.1808
Mean 0.5202
Standard Deviation 0.1965
Lower Outlier Threshold 0.2212
Upper Outlier Threshold 0.8573 0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Mean Average Precision
Mean With No Outliers 0.5586
Std With No Outliers 0.1493
Domain−Specific Monolingual German track − Distribution of the Topics of the Experiment
4
Number of Topics of the Experiment
3.5
3
2.5
2
1.5
1
0.5
0
0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Mean Average Precision
Precision averages (%) for individual 1
Domain−Specific Monolingual German track − Comparison to Median Mean Average Precision by Topic (Topics 151 to 175)
queries TUCMIgirtde2
Topic 151 56.96 Topic 164 0.28 0.8
Topic 152 49.73 Topic 165 85.73
Topic 153 77.34 Topic 166 60.09
0.6
Topic 154 70.61 Topic 167 51.54
Topic 155 48.75 Topic 168 51.30
Topic 156 70.88 Topic 169 15.39 0.4
Topic 157 38.76 Topic 170 60.04
Topic 158 57.31 Topic 171 45.84 0.2
Topic 159 63.44 Topic 172 43.88
Difference
Topic 160 63.43 Topic 173 58.59 0
Topic 161 76.78 Topic 174 48.02
Topic 162 53.12 Topic 175 22.12
−0.2
Topic 163 30.61
−0.4
−0.6
−0.8
−1
151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175
Topic Identifier
101
tuchemniz TUCMIgirtde2 DS-MONO-DE-CLEF2006
Docs Cutoff Levels Precision at DCL (%)
Domain−Specific Monolingual German track − Retrieved documents vs Precision
100%
5 docs 89.60
TUCMIgirtde2
10 docs 84.40 90%
15 docs 80.27
80%
20 docs 78.60
30 docs 74.53 70%
100 docs 56.92
60%
200 docs 40.98
R−Precision
500 docs 22.04 50%
1000 docs 12.56 40%
R-Precision (precision after R document retrieved,
where R = Relevant retrieved) 30%
51.63
20%
10%
0%
5 10 15 20 30 100 200 500 1000
Retrieved Documents (logarithmic scale)
Exact R-Precision Domain−Specific Monolingual German track − Box plot of the Topics of the Experiment
Maximum 0.8219
Minimum 0.0192
First Quartile 0.4659
Second Quartile 0.5625
Third Quartile 0.6069
Interquartile range 0.1410
Mean 0.5163
Standard Deviation 0.1791
Lower Outlier Threshold 0.3846
Upper Outlier Threshold 0.7200 0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Exact R−Precision
Mean With No Outliers 0.5584
Std With No Outliers 0.0838
Domain−Specific Monolingual German track − Distribution of the Topics of the Experiment
6
Number of Topics of the Experiment
5
4
3
2
1
0
0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Exact R−Precision
Precision averages (%) for individual 1
Domain−Specific Monolingual German track − Comparison to Median Mean Exact R−Precision by Topic (Topics 151 to 175)
queries TUCMIgirtde2
Topic 151 54.33 Topic 164 1.92 0.8
Topic 152 51.44 Topic 165 82.19
Topic 153 72.00 Topic 166 53.76
0.6
Topic 154 62.04 Topic 167 57.78
Topic 155 54.40 Topic 168 52.38
Topic 156 64.52 Topic 169 9.09 0.4
Topic 157 38.46 Topic 170 56.41
Topic 158 57.30 Topic 171 46.97 0.2
Topic 159 59.26 Topic 172 45.45
Difference
Topic 160 60.42 Topic 173 61.51 0
Topic 161 69.62 Topic 174 56.25
Topic 162 57.76 Topic 175 25.00
−0.2
Topic 163 40.47
−0.4
−0.6
−0.8
−1
151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175
Topic Identifier
102
tuchemniz TUCMIgirtde3 DS-MONO-DE-CLEF2006
Overall statistics for 25 queries : Priority 3
Total number of documents over all queries Query Construction AUTOMATIC
Retrieved 25,000 Source Language German
Relevant 3,759 Topic Fields title, description
Relevant retrieved 3,138 Pooled true
Geometric Mean Average Precision 0.4132 Snowball-Ger2-QE-BF2-TC
Binary Preference (BPREF) 0.5616
Interploated Recall (%) Precision Averages (%)
Domain−Specific Monolingual German track − Interpolated Recall vs Average Precision
100%
0 93.25
TUCMIgirtde3
10 82.97 90%
20 77.70
80%
30 73.37
40 66.02 70%
50 54.44
Average Precision
60%
60 45.58
70 36.44 50%
80 25.60 40%
90 11.73
30%
100 0.28
Average precision (non-interpolated) for all 20%
relevant documents (averaged over queries)
51.36 10%
0%
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
Interpolated Recall
Mean Average Precision Domain−Specific Monolingual German track − Box plot of the Topics of the Experiment
Maximum 0.8771
Minimum 0.0060
First Quartile 0.4379
Second Quartile 0.5320
Third Quartile 0.6583
Interquartile range 0.2205
Mean 0.5136
Standard Deviation 0.2045
Lower Outlier Threshold 0.2202
Upper Outlier Threshold 0.8771 0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Mean Average Precision
Mean With No Outliers 0.5540
Std With No Outliers 0.1551
Domain−Specific Monolingual German track − Distribution of the Topics of the Experiment
5
Number of Topics of the Experiment
4
3
2
1
0
0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Mean Average Precision
Precision averages (%) for individual 1
Domain−Specific Monolingual German track − Comparison to Median Mean Average Precision by Topic (Topics 151 to 175)
queries TUCMIgirtde3
Topic 151 57.26 Topic 164 0.60 0.8
Topic 152 70.00 Topic 165 87.71
Topic 153 76.31 Topic 166 59.17
0.6
Topic 154 67.84 Topic 167 49.86
Topic 155 34.05 Topic 168 46.36
Topic 156 69.01 Topic 169 9.06 0.4
Topic 157 43.64 Topic 170 59.31
Topic 158 53.20 Topic 171 43.83 0.2
Topic 159 65.16 Topic 172 50.23
Difference
Topic 160 58.09 Topic 173 56.34 0
Topic 161 75.69 Topic 174 48.61
Topic 162 50.82 Topic 175 22.02
−0.2
Topic 163 29.78
−0.4
−0.6
−0.8
−1
151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175
Topic Identifier
103
tuchemniz TUCMIgirtde3 DS-MONO-DE-CLEF2006
Docs Cutoff Levels Precision at DCL (%)
Domain−Specific Monolingual German track − Retrieved documents vs Precision
100%
5 docs 88.80
TUCMIgirtde3
10 docs 83.20 90%
15 docs 78.67
80%
20 docs 76.80
30 docs 72.53 70%
100 docs 57.16
60%
200 docs 40.90
R−Precision
500 docs 22.15 50%
1000 docs 12.55 40%
R-Precision (precision after R document retrieved,
where R = Relevant retrieved) 30%
51.18
20%
10%
0%
5 10 15 20 30 100 200 500 1000
Retrieved Documents (logarithmic scale)
Exact R-Precision Domain−Specific Monolingual German track − Box plot of the Topics of the Experiment
Maximum 0.8493
Minimum 0.0192
First Quartile 0.4486
Second Quartile 0.5472
Third Quartile 0.6141
Interquartile range 0.1655
Mean 0.5118
Standard Deviation 0.1800
Lower Outlier Threshold 0.2656
Upper Outlier Threshold 0.8493 0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Exact R−Precision
Mean With No Outliers 0.5515
Std With No Outliers 0.1209
Domain−Specific Monolingual German track − Distribution of the Topics of the Experiment
5
Number of Topics of the Experiment
4
3
2
1
0
0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Exact R−Precision
Precision averages (%) for individual 1
Domain−Specific Monolingual German track − Comparison to Median Mean Exact R−Precision by Topic (Topics 151 to 175)
queries TUCMIgirtde3
Topic 151 54.72 Topic 164 1.92 0.8
Topic 152 64.90 Topic 165 84.93
Topic 153 73.33 Topic 166 53.23
0.6
Topic 154 58.03 Topic 167 55.56
Topic 155 42.49 Topic 168 48.98
Topic 156 63.44 Topic 169 9.09 0.4
Topic 157 43.08 Topic 170 56.41
Topic 158 58.43 Topic 171 45.45 0.2
Topic 159 61.11 Topic 172 45.45
Difference
Topic 160 55.56 Topic 173 62.30 0
Topic 161 67.41 Topic 174 52.27
Topic 162 54.31 Topic 175 26.56
−0.2
Topic 163 40.47
−0.4
−0.6
−0.8
−1
151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175
Topic Identifier
104
tuchemniz TUCMIgirtde1 DS-MONO-DE-CLEF2006
Overall statistics for 25 queries : Priority 1
Total number of documents over all queries Query Construction AUTOMATIC
Retrieved 25,000 Source Language German
Relevant 3,759 Topic Fields title, description
Relevant retrieved 3,126 Pooled true
Geometric Mean Average Precision 0.4193 Snowball-Ger2-QE-BF2
Binary Preference (BPREF) 0.5592
Interploated Recall (%) Precision Averages (%)
Domain−Specific Monolingual German track − Interpolated Recall vs Average Precision
100%
0 93.64
TUCMIgirtde1
10 84.22 90%
20 79.12
80%
30 74.12
40 66.11 70%
50 54.08
Average Precision
60%
60 45.45
70 35.55 50%
80 23.52 40%
90 9.36
30%
100 0.27
Average precision (non-interpolated) for all 20%
relevant documents (averaged over queries)
51.22 10%
0%
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
Interpolated Recall
Mean Average Precision Domain−Specific Monolingual German track − Box plot of the Topics of the Experiment
Maximum 0.8447
Minimum 0.0067
First Quartile 0.4379
Second Quartile 0.5232
Third Quartile 0.6261
Interquartile range 0.1882
Mean 0.5122
Standard Deviation 0.1960
Lower Outlier Threshold 0.1571
Upper Outlier Threshold 0.8447 0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Mean Average Precision
Mean With No Outliers 0.5333
Std With No Outliers 0.1688
Domain−Specific Monolingual German track − Distribution of the Topics of the Experiment
5
Number of Topics of the Experiment
4
3
2
1
0
0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Mean Average Precision
Precision averages (%) for individual 1
Domain−Specific Monolingual German track − Comparison to Median Mean Average Precision by Topic (Topics 151 to 175)
queries TUCMIgirtde1
Topic 151 57.35 Topic 164 0.67 0.8
Topic 152 61.39 Topic 165 84.47
Topic 153 76.31 Topic 166 57.73
0.6
Topic 154 69.28 Topic 167 50.25
Topic 155 43.45 Topic 168 45.17
Topic 156 67.92 Topic 169 15.84 0.4
Topic 157 43.80 Topic 170 58.82
Topic 158 52.01 Topic 171 43.77 0.2
Topic 159 66.28 Topic 172 50.05
Difference
Topic 160 54.31 Topic 173 59.81 0
Topic 161 75.32 Topic 174 48.73
Topic 162 52.32 Topic 175 15.71
−0.2
Topic 163 29.78
−0.4
−0.6
−0.8
−1
151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175
Topic Identifier
105
tuchemniz TUCMIgirtde1 DS-MONO-DE-CLEF2006
Docs Cutoff Levels Precision at DCL (%)
Domain−Specific Monolingual German track − Retrieved documents vs Precision
100%
5 docs 88.00
TUCMIgirtde1
10 docs 83.60 90%
15 docs 80.53
80%
20 docs 78.20
30 docs 74.80 70%
100 docs 56.60
60%
200 docs 40.28
R−Precision
500 docs 21.92 50%
1000 docs 12.50 40%
R-Precision (precision after R document retrieved,
where R = Relevant retrieved) 30%
50.97
20%
10%
0%
5 10 15 20 30 100 200 500 1000
Retrieved Documents (logarithmic scale)
Exact R-Precision Domain−Specific Monolingual German track − Box plot of the Topics of the Experiment
Maximum 0.8082
Minimum 0.0192
First Quartile 0.4545
Second Quartile 0.5349
Third Quartile 0.6019
Interquartile range 0.1474
Mean 0.5097
Standard Deviation 0.1648
Lower Outlier Threshold 0.2727
Upper Outlier Threshold 0.8082 0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Exact R−Precision
Mean With No Outliers 0.5451
Std With No Outliers 0.1125
Domain−Specific Monolingual German track − Distribution of the Topics of the Experiment
5
Number of Topics of the Experiment
4
3
2
1
0
0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Exact R−Precision
Precision averages (%) for individual 1
Domain−Specific Monolingual German track − Comparison to Median Mean Exact R−Precision by Topic (Topics 151 to 175)
queries TUCMIgirtde1
Topic 151 55.91 Topic 164 1.92 0.8
Topic 152 58.65 Topic 165 80.82
Topic 153 73.33 Topic 166 53.49
0.6
Topic 154 60.22 Topic 167 53.33
Topic 155 46.63 Topic 168 46.94
Topic 156 61.29 Topic 169 27.27 0.4
Topic 157 43.08 Topic 170 56.41
Topic 158 57.30 Topic 171 45.45 0.2
Topic 159 60.19 Topic 172 45.45
Difference
Topic 160 50.00 Topic 173 62.30 0
Topic 161 67.09 Topic 174 52.84
Topic 162 55.17 Topic 175 18.75
−0.2
Topic 163 40.47
−0.4
−0.6
−0.8
−1
151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175
Topic Identifier
106
tuchemniz TUCMIgirtde4 DS-MONO-DE-CLEF2006
Overall statistics for 25 queries : Priority 4
Total number of documents over all queries Query Construction AUTOMATIC
Retrieved 25,000 Source Language German
Relevant 3,759 Topic Fields title, description
Relevant retrieved 3,208 Pooled true
Geometric Mean Average Precision 0.4673 Merged-TUCMIgirtde1+PGD-Overst-QE-BF2
Binary Preference (BPREF) 0.5820
Interploated Recall (%) Precision Averages (%)
Domain−Specific Monolingual German track − Interpolated Recall vs Average Precision
100%
0 94.53
TUCMIgirtde4
10 83.78 90%
20 78.41
80%
30 73.58
40 68.68 70%
50 60.84
Average Precision
60%
60 51.78
70 42.69 50%
80 31.65 40%
90 13.59
30%
100 0.74
Average precision (non-interpolated) for all 20%
relevant documents (averaged over queries)
54.54 10%
0%
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
Interpolated Recall
Mean Average Precision Domain−Specific Monolingual German track − Box plot of the Topics of the Experiment
Maximum 0.8938
Minimum 0.0551
First Quartile 0.4830
Second Quartile 0.6154
Third Quartile 0.6922
Interquartile range 0.2091
Mean 0.5454
Standard Deviation 0.2191
Lower Outlier Threshold 0.1810
Upper Outlier Threshold 0.8938 0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Mean Average Precision
Mean With No Outliers 0.5869
Std With No Outliers 0.1728
Domain−Specific Monolingual German track − Distribution of the Topics of the Experiment
5
Number of Topics of the Experiment
4
3
2
1
0
0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Mean Average Precision
Precision averages (%) for individual 1
Domain−Specific Monolingual German track − Comparison to Median Mean Average Precision by Topic (Topics 151 to 175)
queries TUCMIgirtde4
Topic 151 77.66 Topic 164 5.51 0.8
Topic 152 76.57 Topic 165 89.38
Topic 153 71.69 Topic 166 61.84
0.6
Topic 154 80.12 Topic 167 48.45
Topic 155 49.26 Topic 168 48.42
Topic 156 69.11 Topic 169 19.57 0.4
Topic 157 52.63 Topic 170 61.54
Topic 158 40.22 Topic 171 56.99 0.2
Topic 159 62.26 Topic 172 56.97
Difference
Topic 160 62.61 Topic 173 66.30 0
Topic 161 69.54 Topic 174 47.96
Topic 162 62.60 Topic 175 18.10
−0.2
Topic 163 8.16
−0.4
−0.6
−0.8
−1
151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175
Topic Identifier
107
tuchemniz TUCMIgirtde4 DS-MONO-DE-CLEF2006
Docs Cutoff Levels Precision at DCL (%)
Domain−Specific Monolingual German track − Retrieved documents vs Precision
100%
5 docs 87.20
TUCMIgirtde4
10 docs 83.60 90%
15 docs 79.47
80%
20 docs 75.80
30 docs 71.87 70%
100 docs 58.80
60%
200 docs 43.40
R−Precision
500 docs 22.82 50%
1000 docs 12.83 40%
R-Precision (precision after R document retrieved,
where R = Relevant retrieved) 30%
53.68
20%
10%
0%
5 10 15 20 30 100 200 500 1000
Retrieved Documents (logarithmic scale)
Exact R-Precision Domain−Specific Monolingual German track − Box plot of the Topics of the Experiment
Maximum 0.8356
Minimum 0.0769
First Quartile 0.4840
Second Quartile 0.5618
Third Quartile 0.6654
Interquartile range 0.1814
Mean 0.5368
Standard Deviation 0.1830
Lower Outlier Threshold 0.2344
Upper Outlier Threshold 0.8356 0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Exact R−Precision
Mean With No Outliers 0.5727
Std With No Outliers 0.1397
Domain−Specific Monolingual German track − Distribution of the Topics of the Experiment
4
Number of Topics of the Experiment
3.5
3
2.5
2
1.5
1
0.5
0
0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Exact R−Precision
Precision averages (%) for individual 1
Domain−Specific Monolingual German track − Comparison to Median Mean Exact R−Precision by Topic (Topics 151 to 175)
queries TUCMIgirtde4
Topic 151 72.83 Topic 164 7.69 0.8
Topic 152 73.56 Topic 165 83.56
Topic 153 70.67 Topic 166 56.18
0.6
Topic 154 70.07 Topic 167 48.89
Topic 155 51.81 Topic 168 46.94
Topic 156 67.74 Topic 169 27.27 0.4
Topic 157 46.15 Topic 170 56.41
Topic 158 50.56 Topic 171 54.55 0.2
Topic 159 62.04 Topic 172 50.00
Difference
Topic 160 59.03 Topic 173 62.70 0
Topic 161 66.14 Topic 174 55.11
Topic 162 61.64 Topic 175 23.44
−0.2
Topic 163 17.12
−0.4
−0.6
−0.8
−1
151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175
Topic Identifier
108
unine UniNEde1 DS-MONO-DE-CLEF2006
Overall statistics for 25 queries : Priority 2
Total number of documents over all queries Query Construction AUTOMATIC
Retrieved 25,000 Source Language German
Relevant 3,759 Topic Fields title, description
Relevant retrieved 3,341 Pooled true
Geometric Mean Average Precision 0.4277 data fusion of Okapi & DFR
Binary Preference (BPREF) 0.5430
Interploated Recall (%) Precision Averages (%)
Domain−Specific Monolingual German track − Interpolated Recall vs Average Precision
100%
0 92.13
UniNEde1
10 74.93 90%
20 70.41
80%
30 62.76
40 57.91 70%
50 53.36
Average Precision
60%
60 46.21
70 39.14 50%
80 32.23 40%
90 21.22
30%
100 0.84
Average precision (non-interpolated) for all 20%
relevant documents (averaged over queries)
50.15 10%
0%
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
Interpolated Recall
Mean Average Precision Domain−Specific Monolingual German track − Box plot of the Topics of the Experiment
Maximum 0.9045
Minimum 0.0445
First Quartile 0.2610
Second Quartile 0.5200
Third Quartile 0.6943
Interquartile range 0.4333
Mean 0.5015
Standard Deviation 0.2350
Lower Outlier Threshold 0.0445
Upper Outlier Threshold 0.9045 0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Mean Average Precision
Mean With No Outliers 0.5015
Std With No Outliers 0.2350
Domain−Specific Monolingual German track − Distribution of the Topics of the Experiment
5
Number of Topics of the Experiment
4
3
2
1
0
0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Mean Average Precision
Precision averages (%) for individual 1
Domain−Specific Monolingual German track − Comparison to Median Mean Average Precision by Topic (Topics 151 to 175)
queries UniNEde1
Topic 151 74.19 Topic 164 4.45 0.8
Topic 152 81.64 Topic 165 34.89
Topic 153 79.24 Topic 166 74.91
0.6
Topic 154 90.45 Topic 167 23.19
Topic 155 21.72 Topic 168 49.13
Topic 156 50.97 Topic 169 22.35 0.4
Topic 157 62.60 Topic 170 56.44
Topic 158 44.76 Topic 171 58.29 0.2
Topic 159 68.98 Topic 172 26.69
Difference
Topic 160 63.04 Topic 173 67.50 0
Topic 161 70.76 Topic 174 52.00
Topic 162 24.32 Topic 175 26.87
−0.2
Topic 163 24.28
−0.4
−0.6
−0.8
−1
151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175
Topic Identifier
109
unine UniNEde1 DS-MONO-DE-CLEF2006
Docs Cutoff Levels Precision at DCL (%)
Domain−Specific Monolingual German track − Retrieved documents vs Precision
100%
5 docs 78.40
UniNEde1
10 docs 76.80 90%
15 docs 74.13
80%
20 docs 71.60
30 docs 67.20 70%
100 docs 52.52
60%
200 docs 42.62
R−Precision
500 docs 24.00 50%
1000 docs 13.36 40%
R-Precision (precision after R document retrieved,
where R = Relevant retrieved) 30%
49.91
20%
10%
0%
5 10 15 20 30 100 200 500 1000
Retrieved Documents (logarithmic scale)
Exact R-Precision Domain−Specific Monolingual German track − Box plot of the Topics of the Experiment
Maximum 0.8285
Minimum 0.0769
First Quartile 0.3554
Second Quartile 0.5227
Third Quartile 0.6667
Interquartile range 0.3113
Mean 0.4991
Standard Deviation 0.1934
Lower Outlier Threshold 0.0769
Upper Outlier Threshold 0.8285 0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Exact R−Precision
Mean With No Outliers 0.4991
Std With No Outliers 0.1934
Domain−Specific Monolingual German track − Distribution of the Topics of the Experiment
4
Number of Topics of the Experiment
3.5
3
2.5
2
1.5
1
0.5
0
0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Exact R−Precision
Precision averages (%) for individual 1
Domain−Specific Monolingual German track − Comparison to Median Mean Exact R−Precision by Topic (Topics 151 to 175)
queries UniNEde1
Topic 151 65.35 Topic 164 7.69 0.8
Topic 152 75.96 Topic 165 38.36
Topic 153 73.33 Topic 166 72.31
0.6
Topic 154 82.85 Topic 167 26.67
Topic 155 26.94 Topic 168 44.90
Topic 156 44.09 Topic 169 36.36 0.4
Topic 157 55.38 Topic 170 53.85
Topic 158 48.31 Topic 171 58.59 0.2
Topic 159 66.67 Topic 172 27.27
Difference
Topic 160 59.72 Topic 173 66.67 0
Topic 161 67.72 Topic 174 52.27
Topic 162 40.09 Topic 175 23.44
−0.2
Topic 163 33.07
−0.4
−0.6
−0.8
−1
151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175
Topic Identifier
110
unine UniNEde2 DS-MONO-DE-CLEF2006
Overall statistics for 25 queries : Priority 3
Total number of documents over all queries Query Construction AUTOMATIC
Retrieved 25,000 Source Language German
Relevant 3,759 Topic Fields title, description
Relevant retrieved 3,360 Pooled true
Geometric Mean Average Precision 0.4336 data fusion of DFR, LM & Okapi
Binary Preference (BPREF) 0.5451
Interploated Recall (%) Precision Averages (%)
Domain−Specific Monolingual German track − Interpolated Recall vs Average Precision
100%
0 92.11
UniNEde2
10 75.43 90%
20 70.85
80%
30 64.10
40 58.70 70%
50 54.22
Average Precision
60%
60 47.01
70 39.06 50%
80 32.38 40%
90 21.42
30%
100 0.94
Average precision (non-interpolated) for all 20%
relevant documents (averaged over queries)
50.51 10%
0%
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
Interpolated Recall
Mean Average Precision Domain−Specific Monolingual German track − Box plot of the Topics of the Experiment
Maximum 0.9124
Minimum 0.0480
First Quartile 0.2726
Second Quartile 0.5461
Third Quartile 0.6880
Interquartile range 0.4153
Mean 0.5051
Standard Deviation 0.2345
Lower Outlier Threshold 0.0480
Upper Outlier Threshold 0.9124 0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Mean Average Precision
Mean With No Outliers 0.5051
Std With No Outliers 0.2345
Domain−Specific Monolingual German track − Distribution of the Topics of the Experiment
4
Number of Topics of the Experiment
3.5
3
2.5
2
1.5
1
0.5
0
0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Mean Average Precision
Precision averages (%) for individual 1
Domain−Specific Monolingual German track − Comparison to Median Mean Average Precision by Topic (Topics 151 to 175)
queries UniNEde2
Topic 151 74.13 Topic 164 4.80 0.8
Topic 152 82.19 Topic 165 28.28
Topic 153 79.91 Topic 166 76.80
0.6
Topic 154 91.24 Topic 167 26.80
Topic 155 24.77 Topic 168 48.10
Topic 156 52.54 Topic 169 21.97 0.4
Topic 157 63.73 Topic 170 54.61
Topic 158 44.10 Topic 171 60.41 0.2
Topic 159 67.08 Topic 172 29.23
Difference
Topic 160 60.59 Topic 173 67.93 0
Topic 161 71.39 Topic 174 55.23
Topic 162 24.50 Topic 175 27.42
−0.2
Topic 163 24.91
−0.4
−0.6
−0.8
−1
151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175
Topic Identifier
111
unine UniNEde2 DS-MONO-DE-CLEF2006
Docs Cutoff Levels Precision at DCL (%)
Domain−Specific Monolingual German track − Retrieved documents vs Precision
100%
5 docs 77.60
UniNEde2
10 docs 76.80 90%
15 docs 74.67
80%
20 docs 71.80
30 docs 67.07 70%
100 docs 53.04
60%
200 docs 43.12
R−Precision
500 docs 24.09 50%
1000 docs 13.44 40%
R-Precision (precision after R document retrieved,
where R = Relevant retrieved) 30%
49.89
20%
10%
0%
5 10 15 20 30 100 200 500 1000
Retrieved Documents (logarithmic scale)
Exact R-Precision Domain−Specific Monolingual German track − Box plot of the Topics of the Experiment
Maximum 0.8248
Minimum 0.0769
First Quartile 0.3210
Second Quartile 0.5128
Third Quartile 0.6677
Interquartile range 0.3467
Mean 0.4989
Standard Deviation 0.1919
Lower Outlier Threshold 0.0769
Upper Outlier Threshold 0.8248 0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Exact R−Precision
Mean With No Outliers 0.4989
Std With No Outliers 0.1919
Domain−Specific Monolingual German track − Distribution of the Topics of the Experiment
4
Number of Topics of the Experiment
3.5
3
2.5
2
1.5
1
0.5
0
0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Exact R−Precision
Precision averages (%) for individual 1
Domain−Specific Monolingual German track − Comparison to Median Mean Exact R−Precision by Topic (Topics 151 to 175)
queries UniNEde2
Topic 151 66.54 Topic 164 7.69 0.8
Topic 152 76.44 Topic 165 31.51
Topic 153 73.33 Topic 166 71.77
0.6
Topic 154 82.48 Topic 167 31.11
Topic 155 31.09 Topic 168 46.94
Topic 156 44.09 Topic 169 36.36 0.4
Topic 157 58.46 Topic 170 51.28
Topic 158 49.44 Topic 171 59.09 0.2
Topic 159 62.04 Topic 172 27.27
Difference
Topic 160 56.25 Topic 173 67.46 0
Topic 161 68.35 Topic 174 53.98
Topic 162 40.09 Topic 175 21.88
−0.2
Topic 163 32.30
−0.4
−0.6
−0.8
−1
151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175
Topic Identifier
112
unine UniNEde3 DS-MONO-DE-CLEF2006
Overall statistics for 25 queries : Priority 1
Total number of documents over all queries Query Construction AUTOMATIC
Retrieved 25,000 Source Language German
Relevant 3,759 Topic Fields title, description, narrative
Relevant retrieved 3,389 Pooled true
Geometric Mean Average Precision 0.4553 data fusion of DFR, LM & Okapi
Binary Preference (BPREF) 0.5571
Interploated Recall (%) Precision Averages (%)
Domain−Specific Monolingual German track − Interpolated Recall vs Average Precision
100%
0 96.29
UniNEde3
10 80.98 90%
20 73.20
80%
30 68.16
40 61.09 70%
50 54.21
Average Precision
60%
60 46.60
70 38.24 50%
80 30.36 40%
90 19.89
30%
100 1.41
Average precision (non-interpolated) for all 20%
relevant documents (averaged over queries)
51.59 10%
0%
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
Interpolated Recall
Mean Average Precision Domain−Specific Monolingual German track − Box plot of the Topics of the Experiment
Maximum 0.8834
Minimum 0.0671
First Quartile 0.3210
Second Quartile 0.5437
Third Quartile 0.6812
Interquartile range 0.3602
Mean 0.5159
Standard Deviation 0.2169
Lower Outlier Threshold 0.0671
Upper Outlier Threshold 0.8834 0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Mean Average Precision
Mean With No Outliers 0.5159
Std With No Outliers 0.2169
Domain−Specific Monolingual German track − Distribution of the Topics of the Experiment
3
Number of Topics of the Experiment
2.5
2
1.5
1
0.5
0
0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Mean Average Precision
Precision averages (%) for individual 1
Domain−Specific Monolingual German track − Comparison to Median Mean Average Precision by Topic (Topics 151 to 175)
queries UniNEde3
Topic 151 77.93 Topic 164 6.71 0.8
Topic 152 77.34 Topic 165 40.68
Topic 153 78.64 Topic 166 66.91
0.6
Topic 154 88.34 Topic 167 39.84
Topic 155 46.60 Topic 168 64.18
Topic 156 31.95 Topic 169 18.47 0.4
Topic 157 53.15 Topic 170 54.37
Topic 158 42.59 Topic 171 57.99 0.2
Topic 159 62.37 Topic 172 22.30
Difference
Topic 160 73.48 Topic 173 71.76 0
Topic 161 65.49 Topic 174 59.26
Topic 162 29.11 Topic 175 28.08
−0.2
Topic 163 32.15
−0.4
−0.6
−0.8
−1
151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175
Topic Identifier
113
unine UniNEde3 DS-MONO-DE-CLEF2006
Docs Cutoff Levels Precision at DCL (%)
Domain−Specific Monolingual German track − Retrieved documents vs Precision
100%
5 docs 81.60
UniNEde3
10 docs 78.80 90%
15 docs 78.13
80%
20 docs 73.40
30 docs 69.47 70%
100 docs 56.00
60%
200 docs 43.28
R−Precision
500 docs 24.06 50%
1000 docs 13.56 40%
R-Precision (precision after R document retrieved,
where R = Relevant retrieved) 30%
50.21
20%
10%
0%
5 10 15 20 30 100 200 500 1000
Retrieved Documents (logarithmic scale)
Exact R-Precision Domain−Specific Monolingual German track − Box plot of the Topics of the Experiment
Maximum 0.7956
Minimum 0.0962
First Quartile 0.4041
Second Quartile 0.5556
Third Quartile 0.6361
Interquartile range 0.2321
Mean 0.5021
Standard Deviation 0.1846
Lower Outlier Threshold 0.0962
Upper Outlier Threshold 0.7956 0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Exact R−Precision
Mean With No Outliers 0.5021
Std With No Outliers 0.1846
Domain−Specific Monolingual German track − Distribution of the Topics of the Experiment
4
Number of Topics of the Experiment
3.5
3
2.5
2
1.5
1
0.5
0
0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Exact R−Precision
Precision averages (%) for individual 1
Domain−Specific Monolingual German track − Comparison to Median Mean Exact R−Precision by Topic (Topics 151 to 175)
queries UniNEde3
Topic 151 69.69 Topic 164 9.62 0.8
Topic 152 71.63 Topic 165 42.47
Topic 153 70.67 Topic 166 62.37
0.6
Topic 154 79.56 Topic 167 35.56
Topic 155 47.15 Topic 168 59.18
Topic 156 29.03 Topic 169 18.18 0.4
Topic 157 60.00 Topic 170 51.28
Topic 158 47.19 Topic 171 55.56 0.2
Topic 159 57.41 Topic 172 22.73
Difference
Topic 160 67.36 Topic 173 67.86 0
Topic 161 60.76 Topic 174 59.09
Topic 162 42.24 Topic 175 26.56
−0.2
Topic 163 42.02
−0.4
−0.6
−0.8
−1
151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175
Topic Identifier
114
berkeley BERK_MO_RU_T2FB DS-MONO-RU-CLEF2006
Overall statistics for 25 queries : Priority 1
Total number of documents over all queries Query Construction AUTOMATIC
Retrieved 25,000 Source Language Russian
Relevant 913 Topic Fields title, description
Relevant retrieved 792 Pooled true
Geometric Mean Average Precision 0.0728 Berkeley Trec2 algorithm with blind feedback
Binary Preference (BPREF) 0.2321
Interploated Recall (%) Precision Averages (%)
Domain−Specific Monolingual Russian track − Interpolated Recall vs Average Precision
100%
0 62.41
BERK_MO_RU_T2FB
10 46.67 90%
20 40.81
80%
30 34.50
40 30.55 70%
50 27.95
Average Precision
60%
60 18.74
70 16.12 50%
80 13.45 40%
90 8.17
30%
100 1.76
Average precision (non-interpolated) for all 20%
relevant documents (averaged over queries)
25.42 10%
0%
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
Interpolated Recall
Mean Average Precision Domain−Specific Monolingual Russian track − Box plot of the Topics of the Experiment
Maximum 0.8590
Minimum 0.0000
First Quartile 0.0611
Second Quartile 0.1418
Third Quartile 0.4248
Interquartile range 0.3637
Mean 0.2542
Standard Deviation 0.2547
Lower Outlier Threshold 0.0000
Upper Outlier Threshold 0.8590 0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Mean Average Precision
Mean With No Outliers 0.2542
Std With No Outliers 0.2547
Domain−Specific Monolingual Russian track − Distribution of the Topics of the Experiment
5
Number of Topics of the Experiment
4
3
2
1
0
0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Mean Average Precision
Precision averages (%) for individual 1
Domain−Specific Monolingual Russian track − Comparison to Median Mean Average Precision by Topic (Topics 151 to 175)
queries BERK_MO_RU_T2FB
Topic 151 10.89 Topic 164 5.28 0.8
Topic 152 58.46 Topic 165 10.75
Topic 153 14.18 Topic 166 5.94
0.6
Topic 154 6.17 Topic 167 9.10
Topic 155 21.49 Topic 168 26.19
Topic 156 20.18 Topic 169 2.00 0.4
Topic 157 55.88 Topic 170 7.47
Topic 158 12.80 Topic 171 50.31 0.2
Topic 159 57.00 Topic 172 0.30
Difference
Topic 160 80.15 Topic 173 20.85 0
Topic 161 34.39 Topic 174 0.00
Topic 162 39.87 Topic 175 0.00
−0.2
Topic 163 85.90
−0.4
−0.6
−0.8
−1
151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175
Topic Identifier
115
berkeley BERK_MO_RU_T2FB DS-MONO-RU-CLEF2006
Docs Cutoff Levels Precision at DCL (%)
Domain−Specific Monolingual Russian track − Retrieved documents vs Precision
100%
5 docs 32.80
BERK_MO_RU_T2FB
10 docs 30.80 90%
15 docs 27.20
80%
20 docs 25.80
30 docs 24.13 70%
100 docs 18.44
60%
200 docs 12.62
R−Precision
500 docs 5.94 50%
1000 docs 3.17 40%
R-Precision (precision after R document retrieved,
where R = Relevant retrieved) 30%
24.85
20%
10%
0%
5 10 15 20 30 100 200 500 1000
Retrieved Documents (logarithmic scale)
Exact R-Precision Domain−Specific Monolingual Russian track − Box plot of the Topics of the Experiment
Maximum 0.8584
Minimum 0.0000
First Quartile 0.0000
Second Quartile 0.1739
Third Quartile 0.4318
Interquartile range 0.4318
Mean 0.2485
Standard Deviation 0.2561
Lower Outlier Threshold 0.0000
Upper Outlier Threshold 0.8584 0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Exact R−Precision
Mean With No Outliers 0.2485
Std With No Outliers 0.2561
Domain−Specific Monolingual Russian track − Distribution of the Topics of the Experiment
8
Number of Topics of the Experiment
7
6
5
4
3
2
1
0
0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Exact R−Precision
Precision averages (%) for individual 1
Domain−Specific Monolingual Russian track − Comparison to Median Mean Exact R−Precision by Topic (Topics 151 to 175)
queries BERK_MO_RU_T2FB
Topic 151 18.75 Topic 164 0.00 0.8
Topic 152 55.81 Topic 165 11.11
Topic 153 17.39 Topic 166 0.00
0.6
Topic 154 12.12 Topic 167 0.00
Topic 155 13.51 Topic 168 25.00
Topic 156 25.00 Topic 169 0.00 0.4
Topic 157 50.00 Topic 170 4.35
Topic 158 14.29 Topic 171 50.00 0.2
Topic 159 52.38 Topic 172 0.00
Difference
Topic 160 81.98 Topic 173 30.00 0
Topic 161 40.91 Topic 174 0.00
Topic 162 32.76 Topic 175 0.00
−0.2
Topic 163 85.84
−0.4
−0.6
−0.8
−1
151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175
Topic Identifier
116
berkeley BERK_BI_RUEN_T2FB_B DS-BILI-X2EN-CLEF2006
Overall statistics for 25 queries : Priority 3
Total number of documents over all queries Query Construction AUTOMATIC
Retrieved 25,000 Source Language German
Relevant 4,239 Topic Fields title, description
Relevant retrieved 2,674 Pooled true
Geometric Mean Average Precision 0.2108 Russian to English using Babelfish
Binary Preference (BPREF) 0.3765
Interploated Recall (%) Precision Averages (%)
Domain−Specific Bilingual English track − Interpolated Recall vs Average Precision
100%
0 79.91
BERK_BI_RUEN_T2FB_B
10 59.42 90%
20 53.91
80%
30 46.84
40 40.64 70%
50 32.74
Average Precision
60%
60 25.29
70 17.61 50%
80 10.98 40%
90 6.21
30%
100 0.00
Average precision (non-interpolated) for all 20%
relevant documents (averaged over queries)
32.28 10%
0%
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
Interpolated Recall
Mean Average Precision Domain−Specific Bilingual English track − Box plot of the Topics of the Experiment
Maximum 0.8151
Minimum 0.0025
First Quartile 0.1356
Second Quartile 0.2490
Third Quartile 0.4725
Interquartile range 0.3369
Mean 0.3228
Standard Deviation 0.2296
Lower Outlier Threshold 0.0025
Upper Outlier Threshold 0.8151 0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Mean Average Precision
Mean With No Outliers 0.3228
Std With No Outliers 0.2296
Domain−Specific Bilingual English track − Distribution of the Topics of the Experiment
4
Number of Topics of the Experiment
3.5
3
2.5
2
1.5
1
0.5
0
0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Mean Average Precision
Precision averages (%) for individual 1
Domain−Specific Bilingual English track − Comparison to Median Mean Average Precision by Topic (Topics 151 to 175)
queries BERK_BI_RUEN_T2FB_B
Topic 151 71.74 Topic 164 6.08 0.8
Topic 152 57.51 Topic 165 81.51
Topic 153 2.41 Topic 166 20.56
0.6
Topic 154 56.77 Topic 167 13.64
Topic 155 37.93 Topic 168 55.32
Topic 156 38.09 Topic 169 16.83 0.4
Topic 157 41.54 Topic 170 24.90
Topic 158 8.00 Topic 171 11.19 0.2
Topic 159 70.05 Topic 172 22.09
Difference
Topic 160 19.96 Topic 173 0.25 0
Topic 161 33.66 Topic 174 37.45
Topic 162 44.56 Topic 175 21.67
−0.2
Topic 163 13.32
−0.4
−0.6
−0.8
−1
151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175
Topic Identifier
117
berkeley BERK_BI_RUEN_T2FB_B DS-BILI-X2EN-CLEF2006
Docs Cutoff Levels Precision at DCL (%)
Domain−Specific Bilingual English track − Retrieved documents vs Precision
100%
5 docs 56.80
BERK_BI_RUEN_T2FB_B
10 docs 58.00 90%
15 docs 57.60
80%
20 docs 56.60
30 docs 54.27 70%
100 docs 45.60
60%
200 docs 32.66
R−Precision
500 docs 17.66 50%
1000 docs 10.70 40%
R-Precision (precision after R document retrieved,
where R = Relevant retrieved) 30%
37.63
20%
10%
0%
5 10 15 20 30 100 200 500 1000
Retrieved Documents (logarithmic scale)
Exact R-Precision Domain−Specific Bilingual English track − Box plot of the Topics of the Experiment
Maximum 0.7333
Minimum 0.0289
First Quartile 0.2144
Second Quartile 0.4091
Third Quartile 0.4958
Interquartile range 0.2814
Mean 0.3763
Standard Deviation 0.1941
Lower Outlier Threshold 0.0289
Upper Outlier Threshold 0.7333 0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Exact R−Precision
Mean With No Outliers 0.3763
Std With No Outliers 0.1941
Domain−Specific Bilingual English track − Distribution of the Topics of the Experiment
4
Number of Topics of the Experiment
3.5
3
2.5
2
1.5
1
0.5
0
0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Exact R−Precision
Precision averages (%) for individual 1
Domain−Specific Bilingual English track − Comparison to Median Mean Exact R−Precision by Topic (Topics 151 to 175)
queries BERK_BI_RUEN_T2FB_B
Topic 151 67.06 Topic 164 17.11 0.8
Topic 152 56.60 Topic 165 73.33
Topic 153 10.03 Topic 166 30.47
0.6
Topic 154 55.51 Topic 167 21.74
Topic 155 42.76 Topic 168 60.61
Topic 156 46.23 Topic 169 19.05 0.4
Topic 157 45.26 Topic 170 40.91
Topic 158 20.54 Topic 171 18.13 0.2
Topic 159 70.90 Topic 172 22.50
Difference
Topic 160 32.08 Topic 173 2.89 0
Topic 161 44.62 Topic 174 42.11
Topic 162 47.60 Topic 175 30.19
−0.2
Topic 163 22.54
−0.4
−0.6
−0.8
−1
151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175
Topic Identifier
118
berkeley BERK_BI_DEEN_T2FB_B DS-BILI-X2EN-CLEF2006
Overall statistics for 25 queries : Priority 1
Total number of documents over all queries Query Construction AUTOMATIC
Retrieved 25,000 Source Language German
Relevant 4,239 Topic Fields title, description
Relevant retrieved 2,784 Pooled true
Geometric Mean Average Precision 0.2196 German to English using Babelfish translation
Binary Preference (BPREF) 0.3683
Interploated Recall (%) Precision Averages (%)
Domain−Specific Bilingual English track − Interpolated Recall vs Average Precision
100%
0 80.56
BERK_BI_DEEN_T2FB_B
10 59.08 90%
20 53.12
80%
30 48.49
40 43.27 70%
50 36.18
Average Precision
60%
60 25.80
70 17.92 50%
80 13.28 40%
90 7.19
30%
100 0.32
Average precision (non-interpolated) for all 20%
relevant documents (averaged over queries)
33.01 10%
0%
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
Interpolated Recall
Mean Average Precision Domain−Specific Bilingual English track − Box plot of the Topics of the Experiment
Maximum 0.7275
Minimum 0.0068
First Quartile 0.1497
Second Quartile 0.2911
Third Quartile 0.4851
Interquartile range 0.3354
Mean 0.3301
Standard Deviation 0.2214
Lower Outlier Threshold 0.0068
Upper Outlier Threshold 0.7275 0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Mean Average Precision
Mean With No Outliers 0.3301
Std With No Outliers 0.2214
Domain−Specific Bilingual English track − Distribution of the Topics of the Experiment
4
Number of Topics of the Experiment
3.5
3
2.5
2
1.5
1
0.5
0
0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Mean Average Precision
Precision averages (%) for individual 1
Domain−Specific Bilingual English track − Comparison to Median Mean Average Precision by Topic (Topics 151 to 175)
queries BERK_BI_DEEN_T2FB_B
Topic 151 66.01 Topic 164 28.78 0.8
Topic 152 29.11 Topic 165 30.54
Topic 153 45.68 Topic 166 48.29
0.6
Topic 154 72.75 Topic 167 20.00
Topic 155 2.35 Topic 168 0.68
Topic 156 12.44 Topic 169 16.20 0.4
Topic 157 55.46 Topic 170 40.09
Topic 158 3.95 Topic 171 49.19 0.2
Topic 159 69.29 Topic 172 23.26
Difference
Topic 160 46.89 Topic 173 41.67 0
Topic 161 28.42 Topic 174 62.53
Topic 162 4.57 Topic 175 11.38
−0.2
Topic 163 15.81
−0.4
−0.6
−0.8
−1
151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175
Topic Identifier
119
berkeley BERK_BI_DEEN_T2FB_B DS-BILI-X2EN-CLEF2006
Docs Cutoff Levels Precision at DCL (%)
Domain−Specific Bilingual English track − Retrieved documents vs Precision
100%
5 docs 63.20
BERK_BI_DEEN_T2FB_B
10 docs 57.60 90%
15 docs 57.33
80%
20 docs 55.80
30 docs 52.67 70%
100 docs 45.84
60%
200 docs 34.18
R−Precision
500 docs 18.66 50%
1000 docs 11.14 40%
R-Precision (precision after R document retrieved,
where R = Relevant retrieved) 30%
37.46
20%
10%
0%
5 10 15 20 30 100 200 500 1000
Retrieved Documents (logarithmic scale)
Exact R-Precision Domain−Specific Bilingual English track − Box plot of the Topics of the Experiment
Maximum 0.7015
Minimum 0.0101
First Quartile 0.2107
Second Quartile 0.3816
Third Quartile 0.5280
Interquartile range 0.3173
Mean 0.3746
Standard Deviation 0.2041
Lower Outlier Threshold 0.0101
Upper Outlier Threshold 0.7015 0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Exact R−Precision
Mean With No Outliers 0.3746
Std With No Outliers 0.2041
Domain−Specific Bilingual English track − Distribution of the Topics of the Experiment
5
Number of Topics of the Experiment
4
3
2
1
0
0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Exact R−Precision
Precision averages (%) for individual 1
Domain−Specific Bilingual English track − Comparison to Median Mean Exact R−Precision by Topic (Topics 151 to 175)
queries BERK_BI_DEEN_T2FB_B
Topic 151 63.53 Topic 164 38.16 0.8
Topic 152 39.88 Topic 165 36.67
Topic 153 46.71 Topic 166 51.17
0.6
Topic 154 69.07 Topic 167 21.74
Topic 155 7.89 Topic 168 1.01
Topic 156 22.64 Topic 169 19.05 0.4
Topic 157 52.63 Topic 170 50.00
Topic 158 11.61 Topic 171 54.95 0.2
Topic 159 70.15 Topic 172 32.50
Difference
Topic 160 47.92 Topic 173 53.31 0
Topic 161 35.96 Topic 174 63.16
Topic 162 10.86 Topic 175 11.32
−0.2
Topic 163 24.65
−0.4
−0.6
−0.8
−1
151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175
Topic Identifier
120
berkeley BERK_BI_DEEN_T2FB_P DS-BILI-X2EN-CLEF2006
Overall statistics for 25 queries : Priority 2
Total number of documents over all queries Query Construction AUTOMATIC
Retrieved 25,000 Source Language German
Relevant 4,239 Topic Fields title, description
Relevant retrieved 2,636 Pooled true
Geometric Mean Average Precision 0.1826 German to English translation using PROMT
Binary Preference (BPREF) 0.3577
Interploated Recall (%) Precision Averages (%)
Domain−Specific Bilingual English track − Interpolated Recall vs Average Precision
100%
0 84.27
BERK_BI_DEEN_T2FB_P
10 57.31 90%
20 51.66
80%
30 47.24
40 41.54 70%
50 32.98
Average Precision
60%
60 24.54
70 17.44 50%
80 9.95 40%
90 3.39
30%
100 0.33
Average precision (non-interpolated) for all 20%
relevant documents (averaged over queries)
31.76 10%
0%
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
Interpolated Recall
Mean Average Precision Domain−Specific Bilingual English track − Box plot of the Topics of the Experiment
Maximum 0.6929
Minimum 0.0006
First Quartile 0.1362
Second Quartile 0.2878
Third Quartile 0.5039
Interquartile range 0.3678
Mean 0.3176
Standard Deviation 0.2170
Lower Outlier Threshold 0.0006
Upper Outlier Threshold 0.6929 0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Mean Average Precision
Mean With No Outliers 0.3176
Std With No Outliers 0.2170
Domain−Specific Bilingual English track − Distribution of the Topics of the Experiment
4
Number of Topics of the Experiment
3.5
3
2.5
2
1.5
1
0.5
0
0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Mean Average Precision
Precision averages (%) for individual 1
Domain−Specific Bilingual English track − Comparison to Median Mean Average Precision by Topic (Topics 151 to 175)
queries BERK_BI_DEEN_T2FB_P
Topic 151 64.77 Topic 164 28.78 0.8
Topic 152 16.87 Topic 165 58.88
Topic 153 45.68 Topic 166 53.40
0.6
Topic 154 52.25 Topic 167 24.02
Topic 155 0.06 Topic 168 23.28
Topic 156 4.89 Topic 169 19.51 0.4
Topic 157 2.41 Topic 170 34.49
Topic 158 1.26 Topic 171 49.19 0.2
Topic 159 69.29 Topic 172 17.62
Difference
Topic 160 41.64 Topic 173 49.77 0
Topic 161 47.39 Topic 174 54.39
Topic 162 8.48 Topic 175 11.38
−0.2
Topic 163 14.36
−0.4
−0.6
−0.8
−1
151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175
Topic Identifier
121
berkeley BERK_BI_DEEN_T2FB_P DS-BILI-X2EN-CLEF2006
Docs Cutoff Levels Precision at DCL (%)
Domain−Specific Bilingual English track − Retrieved documents vs Precision
100%
5 docs 62.40
BERK_BI_DEEN_T2FB_P
10 docs 58.40 90%
15 docs 56.53
80%
20 docs 54.40
30 docs 52.00 70%
100 docs 44.96
60%
200 docs 32.62
R−Precision
500 docs 17.70 50%
1000 docs 10.54 40%
R-Precision (precision after R document retrieved,
where R = Relevant retrieved) 30%
36.19
20%
10%
0%
5 10 15 20 30 100 200 500 1000
Retrieved Documents (logarithmic scale)
Exact R-Precision Domain−Specific Bilingual English track − Box plot of the Topics of the Experiment
Maximum 0.7015
Minimum 0.0164
First Quartile 0.1728
Second Quartile 0.3816
Third Quartile 0.5259
Interquartile range 0.3531
Mean 0.3619
Standard Deviation 0.2029
Lower Outlier Threshold 0.0164
Upper Outlier Threshold 0.7015 0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Exact R−Precision
Mean With No Outliers 0.3619
Std With No Outliers 0.2029
Domain−Specific Bilingual English track − Distribution of the Topics of the Experiment
4
Number of Topics of the Experiment
3.5
3
2.5
2
1.5
1
0.5
0
0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Exact R−Precision
Precision averages (%) for individual 1
Domain−Specific Bilingual English track − Comparison to Median Mean Exact R−Precision by Topic (Topics 151 to 175)
queries BERK_BI_DEEN_T2FB_P
Topic 151 63.53 Topic 164 38.16 0.8
Topic 152 29.62 Topic 165 63.33
Topic 153 46.71 Topic 166 52.73
0.6
Topic 154 52.54 Topic 167 34.78
Topic 155 1.64 Topic 168 34.34
Topic 156 16.04 Topic 169 23.81 0.4
Topic 157 5.26 Topic 170 40.91
Topic 158 3.57 Topic 171 54.95 0.2
Topic 159 70.15 Topic 172 17.50
Difference
Topic 160 45.28 Topic 173 50.00 0
Topic 161 49.87 Topic 174 57.31
Topic 162 16.61 Topic 175 11.32
−0.2
Topic 163 24.65
−0.4
−0.6
−0.8
−1
151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175
Topic Identifier
122
berkeley BERK_BI_ENDE_T2FB_B DS-BILI-X2DE-CLEF2006
Overall statistics for 25 queries : Priority 1
Total number of documents over all queries Query Construction AUTOMATIC
Retrieved 25,000 Source Language English
Relevant 3,759 Topic Fields title, description
Relevant retrieved 1,994 Pooled true
Geometric Mean Average Precision 0.0479 English to German using Babelfish
Binary Preference (BPREF) 0.2915
Interploated Recall (%) Precision Averages (%)
Domain−Specific Bilingual German track − Interpolated Recall vs Average Precision
100%
0 54.98
BERK_BI_ENDE_T2FB_B
10 41.59 90%
20 38.87
80%
30 33.56
40 27.51 70%
50 22.98
Average Precision
60%
60 19.74
70 13.51 50%
80 10.61 40%
90 6.42
30%
100 0.64
Average precision (non-interpolated) for all 20%
relevant documents (averaged over queries)
23.66 10%
0%
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
Interpolated Recall
Mean Average Precision Domain−Specific Bilingual German track − Box plot of the Topics of the Experiment
Maximum 0.7927
Minimum 0.0000
First Quartile 0.0215
Second Quartile 0.0767
Third Quartile 0.4403
Interquartile range 0.4188
Mean 0.2366
Standard Deviation 0.2716
Lower Outlier Threshold 0.0000
Upper Outlier Threshold 0.7927 0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Mean Average Precision
Mean With No Outliers 0.2366
Std With No Outliers 0.2716
Domain−Specific Bilingual German track − Distribution of the Topics of the Experiment
15
Number of Topics of the Experiment
10
5
0
0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Mean Average Precision
Precision averages (%) for individual 1
Domain−Specific Bilingual German track − Comparison to Median Mean Average Precision by Topic (Topics 151 to 175)
queries BERK_BI_ENDE_T2FB_B
Topic 151 78.41 Topic 164 4.75 0.8
Topic 152 0.10 Topic 165 1.97
Topic 153 53.16 Topic 166 36.02
0.6
Topic 154 7.67 Topic 167 2.41
Topic 155 0.18 Topic 168 40.99
Topic 156 0.00 Topic 169 4.55 0.4
Topic 157 79.27 Topic 170 56.63
Topic 158 2.40 Topic 171 19.05 0.2
Topic 159 66.62 Topic 172 2.21
Difference
Topic 160 0.04 Topic 173 0.27 0
Topic 161 60.59 Topic 174 5.64
Topic 162 29.02 Topic 175 19.23
−0.2
Topic 163 20.26
−0.4
−0.6
−0.8
−1
151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175
Topic Identifier
123
berkeley BERK_BI_ENDE_T2FB_B DS-BILI-X2DE-CLEF2006
Docs Cutoff Levels Precision at DCL (%)
Domain−Specific Bilingual German track − Retrieved documents vs Precision
100%
5 docs 40.00
BERK_BI_ENDE_T2FB_B
10 docs 41.20 90%
15 docs 39.20
80%
20 docs 39.60
30 docs 38.53 70%
100 docs 30.24
60%
200 docs 21.96
R−Precision
500 docs 12.85 50%
1000 docs 7.98 40%
R-Precision (precision after R document retrieved,
where R = Relevant retrieved) 30%
26.63
20%
10%
0%
5 10 15 20 30 100 200 500 1000
Retrieved Documents (logarithmic scale)
Exact R-Precision Domain−Specific Bilingual German track − Box plot of the Topics of the Experiment
Maximum 0.6969
Minimum 0.0000
First Quartile 0.0444
Second Quartile 0.1861
Third Quartile 0.4583
Interquartile range 0.4139
Mean 0.2663
Standard Deviation 0.2432
Lower Outlier Threshold 0.0000
Upper Outlier Threshold 0.6969 0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Exact R−Precision
Mean With No Outliers 0.2663
Std With No Outliers 0.2432
Domain−Specific Bilingual German track − Distribution of the Topics of the Experiment
8
Number of Topics of the Experiment
7
6
5
4
3
2
1
0
0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Exact R−Precision
Precision averages (%) for individual 1
Domain−Specific Bilingual German track − Comparison to Median Mean Exact R−Precision by Topic (Topics 151 to 175)
queries BERK_BI_ENDE_T2FB_B
Topic 151 69.69 Topic 164 7.69 0.8
Topic 152 1.92 Topic 165 4.11
Topic 153 50.67 Topic 166 43.28
0.6
Topic 154 18.61 Topic 167 6.67
Topic 155 1.04 Topic 168 44.22
Topic 156 0.00 Topic 169 9.09 0.4
Topic 157 69.23 Topic 170 58.97
Topic 158 11.24 Topic 171 27.27 0.2
Topic 159 61.11 Topic 172 4.55
Difference
Topic 160 0.00 Topic 173 1.59 0
Topic 161 58.54 Topic 174 12.50
Topic 162 40.52 Topic 175 31.25
−0.2
Topic 163 31.91
−0.4
−0.6
−0.8
−1
151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175
Topic Identifier
124
hagen FUHegpyynl500 DS-BILI-X2DE-CLEF2006
Overall statistics for 25 queries : Priority 1
Total number of documents over all queries Query Construction AUTOMATIC
Retrieved 25,000 Source Language English
Relevant 3,759 Topic Fields title, description
Relevant retrieved 2,569 Pooled true
Geometric Mean Average Precision 0.1807 First run, EN
Binary Preference (BPREF) 0.2910
Interploated Recall (%) Precision Averages (%)
Domain−Specific Bilingual German track − Interpolated Recall vs Average Precision
100%
0 74.37
FUHegpyynl500
10 47.16 90%
20 40.42
80%
30 33.57
40 28.47 70%
50 22.95
Average Precision
60%
60 18.44
70 12.56 50%
80 8.25 40%
90 4.32
30%
100 0.07
Average precision (non-interpolated) for all 20%
relevant documents (averaged over queries)
24.48 10%
0%
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
Interpolated Recall
Mean Average Precision Domain−Specific Bilingual German track − Box plot of the Topics of the Experiment
Maximum 0.6908
Minimum 0.0443
First Quartile 0.0842
Second Quartile 0.1907
Third Quartile 0.3691
Interquartile range 0.2849
Mean 0.2448
Standard Deviation 0.1863
Lower Outlier Threshold 0.0443
Upper Outlier Threshold 0.6908 0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Mean Average Precision
Mean With No Outliers 0.2448
Std With No Outliers 0.1863
Domain−Specific Bilingual German track − Distribution of the Topics of the Experiment
6
Number of Topics of the Experiment
5
4
3
2
1
0
0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Mean Average Precision
Precision averages (%) for individual 1
Domain−Specific Bilingual German track − Comparison to Median Mean Average Precision by Topic (Topics 151 to 175)
queries FUHegpyynl500
Topic 151 4.67 Topic 164 13.20 0.8
Topic 152 8.13 Topic 165 25.53
Topic 153 37.00 Topic 166 53.08
0.6
Topic 154 69.08 Topic 167 4.43
Topic 155 25.51 Topic 168 33.46
Topic 156 8.52 Topic 169 24.47 0.4
Topic 157 10.87 Topic 170 26.08
Topic 158 8.91 Topic 171 19.07 0.2
Topic 159 43.19 Topic 172 7.82
Difference
Topic 160 6.95 Topic 173 60.57 0
Topic 161 36.88 Topic 174 47.02
Topic 162 12.26 Topic 175 7.45
−0.2
Topic 163 17.81
−0.4
−0.6
−0.8
−1
151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175
Topic Identifier
125
hagen FUHegpyynl500 DS-BILI-X2DE-CLEF2006
Docs Cutoff Levels Precision at DCL (%)
Domain−Specific Bilingual German track − Retrieved documents vs Precision
100%
5 docs 50.40
FUHegpyynl500
10 docs 49.60 90%
15 docs 46.93
80%
20 docs 46.40
30 docs 43.73 70%
100 docs 34.04
60%
200 docs 27.20
R−Precision
500 docs 16.80 50%
1000 docs 10.28 40%
R-Precision (precision after R document retrieved,
where R = Relevant retrieved) 30%
30.38
20%
10%
0%
5 10 15 20 30 100 200 500 1000
Retrieved Documents (logarithmic scale)
Exact R-Precision Domain−Specific Bilingual German track − Box plot of the Topics of the Experiment
Maximum 0.6350
Minimum 0.0667
First Quartile 0.1521
Second Quartile 0.2727
Third Quartile 0.4169
Interquartile range 0.2647
Mean 0.3038
Standard Deviation 0.1671
Lower Outlier Threshold 0.0667
Upper Outlier Threshold 0.6350 0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Exact R−Precision
Mean With No Outliers 0.3038
Std With No Outliers 0.1671
Domain−Specific Bilingual German track − Distribution of the Topics of the Experiment
5
Number of Topics of the Experiment
4
3
2
1
0
0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Exact R−Precision
Precision averages (%) for individual 1
Domain−Specific Bilingual German track − Comparison to Median Mean Exact R−Precision by Topic (Topics 151 to 175)
queries FUHegpyynl500
Topic 151 13.78 Topic 164 17.31 0.8
Topic 152 13.46 Topic 165 31.51
Topic 153 40.00 Topic 166 56.99
0.6
Topic 154 63.50 Topic 167 6.67
Topic 155 32.64 Topic 168 40.82
Topic 156 13.98 Topic 169 27.27 0.4
Topic 157 20.00 Topic 170 35.90
Topic 158 13.48 Topic 171 30.81 0.2
Topic 159 49.07 Topic 172 13.64
Difference
Topic 160 15.97 Topic 173 59.92 0
Topic 161 44.30 Topic 174 52.84
Topic 162 22.84 Topic 175 15.62
−0.2
Topic 163 27.24
−0.4
−0.6
−0.8
−1
151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175
Topic Identifier
126
hagen FUHegpyydl500 DS-BILI-X2DE-CLEF2006
Overall statistics for 25 queries : Priority 2
Total number of documents over all queries Query Construction AUTOMATIC
Retrieved 25,000 Source Language English
Relevant 3,759 Topic Fields title, description
Relevant retrieved 2,422 Pooled true
Geometric Mean Average Precision 0.1226 Second run, EN
Binary Preference (BPREF) 0.2720
Interploated Recall (%) Precision Averages (%)
Domain−Specific Bilingual German track − Interpolated Recall vs Average Precision
100%
0 67.04
FUHegpyydl500
10 39.89 90%
20 35.99
80%
30 32.09
40 26.30 70%
50 20.56
Average Precision
60%
60 16.90
70 12.45 50%
80 6.95 40%
90 1.91
30%
100 0.00
Average precision (non-interpolated) for all 20%
relevant documents (averaged over queries)
21.90 10%
0%
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
Interpolated Recall
Mean Average Precision Domain−Specific Bilingual German track − Box plot of the Topics of the Experiment
Maximum 0.6919
Minimum 0.0019
First Quartile 0.0737
Second Quartile 0.1565
Third Quartile 0.3487
Interquartile range 0.2751
Mean 0.2190
Standard Deviation 0.1966
Lower Outlier Threshold 0.0019
Upper Outlier Threshold 0.6919 0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Mean Average Precision
Mean With No Outliers 0.2190
Std With No Outliers 0.1966
Domain−Specific Bilingual German track − Distribution of the Topics of the Experiment
5
Number of Topics of the Experiment
4
3
2
1
0
0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Mean Average Precision
Precision averages (%) for individual 1
Domain−Specific Bilingual German track − Comparison to Median Mean Average Precision by Topic (Topics 151 to 175)
queries FUHegpyydl500
Topic 151 8.36 Topic 164 1.95 0.8
Topic 152 1.11 Topic 165 20.68
Topic 153 40.89 Topic 166 60.51
0.6
Topic 154 28.12 Topic 167 6.57
Topic 155 15.65 Topic 168 32.87
Topic 156 9.97 Topic 169 11.72 0.4
Topic 157 0.19 Topic 170 16.17
Topic 158 2.24 Topic 171 20.91 0.2
Topic 159 69.19 Topic 172 7.01
Difference
Topic 160 16.05 Topic 173 48.68 0
Topic 161 48.50 Topic 174 45.81
Topic 162 14.35 Topic 175 7.48
−0.2
Topic 163 12.52
−0.4
−0.6
−0.8
−1
151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175
Topic Identifier
127
hagen FUHegpyydl500 DS-BILI-X2DE-CLEF2006
Docs Cutoff Levels Precision at DCL (%)
Domain−Specific Bilingual German track − Retrieved documents vs Precision
100%
5 docs 44.80
FUHegpyydl500
10 docs 39.60 90%
15 docs 39.47
80%
20 docs 38.20
30 docs 37.33 70%
100 docs 31.48
60%
200 docs 25.86
R−Precision
500 docs 15.51 50%
1000 docs 9.69 40%
R-Precision (precision after R document retrieved,
where R = Relevant retrieved) 30%
27.81
20%
10%
0%
5 10 15 20 30 100 200 500 1000
Retrieved Documents (logarithmic scale)
Exact R-Precision Domain−Specific Bilingual German track − Box plot of the Topics of the Experiment
Maximum 0.6204
Minimum 0.0000
First Quartile 0.1356
Second Quartile 0.2412
Third Quartile 0.4399
Interquartile range 0.3042
Mean 0.2781
Standard Deviation 0.1868
Lower Outlier Threshold 0.0000
Upper Outlier Threshold 0.6204 0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Exact R−Precision
Mean With No Outliers 0.2781
Std With No Outliers 0.1868
Domain−Specific Bilingual German track − Distribution of the Topics of the Experiment
5
Number of Topics of the Experiment
4
3
2
1
0
0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Exact R−Precision
Precision averages (%) for individual 1
Domain−Specific Bilingual German track − Comparison to Median Mean Exact R−Precision by Topic (Topics 151 to 175)
queries FUHegpyydl500
Topic 151 17.32 Topic 164 0.00 0.8
Topic 152 7.21 Topic 165 31.51
Topic 153 45.33 Topic 166 60.75
0.6
Topic 154 39.05 Topic 167 15.56
Topic 155 19.17 Topic 168 43.54
Topic 156 15.05 Topic 169 9.09 0.4
Topic 157 1.54 Topic 170 23.08
Topic 158 7.87 Topic 171 34.85 0.2
Topic 159 62.04 Topic 172 9.09
Difference
Topic 160 27.78 Topic 173 52.38 0
Topic 161 49.37 Topic 174 52.84
Topic 162 31.03 Topic 175 15.62
−0.2
Topic 163 24.12
−0.4
−0.6
−0.8
−1
151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175
Topic Identifier
128
hagen FUHegpyynl102 DS-BILI-X2DE-CLEF2006
Overall statistics for 25 queries : Priority 3
Total number of documents over all queries Query Construction AUTOMATIC
Retrieved 25,000 Source Language English
Relevant 3,759 Topic Fields title, description
Relevant retrieved 2,134 Pooled true
Geometric Mean Average Precision 0.1176 Third run, EN
Binary Preference (BPREF) 0.2533
Interploated Recall (%) Precision Averages (%)
Domain−Specific Bilingual German track − Interpolated Recall vs Average Precision
100%
0 67.18
FUHegpyynl102
10 39.14 90%
20 34.09
80%
30 27.77
40 22.30 70%
50 16.92
Average Precision
60%
60 12.92
70 9.30 50%
80 5.82 40%
90 3.59
30%
100 0.16
Average precision (non-interpolated) for all 20%
relevant documents (averaged over queries)
19.80 10%
0%
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
Interpolated Recall
Mean Average Precision Domain−Specific Bilingual German track − Box plot of the Topics of the Experiment
Maximum 0.8019
Minimum 0.0067
First Quartile 0.0544
Second Quartile 0.1790
Third Quartile 0.2578
Interquartile range 0.2034
Mean 0.1980
Standard Deviation 0.1887
Lower Outlier Threshold 0.0067
Upper Outlier Threshold 0.5205 0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Mean Average Precision
Mean With No Outliers 0.1729
Std With No Outliers 0.1437
Domain−Specific Bilingual German track − Distribution of the Topics of the Experiment
6
Number of Topics of the Experiment
5
4
3
2
1
0
0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Mean Average Precision
Precision averages (%) for individual 1
Domain−Specific Bilingual German track − Comparison to Median Mean Average Precision by Topic (Topics 151 to 175)
queries FUHegpyynl102
Topic 151 5.67 Topic 164 23.38 0.8
Topic 152 1.89 Topic 165 25.94
Topic 153 3.97 Topic 166 25.25
0.6
Topic 154 80.19 Topic 167 2.43
Topic 155 26.94 Topic 168 14.61
Topic 156 0.67 Topic 169 24.17 0.4
Topic 157 6.07 Topic 170 28.12
Topic 158 7.83 Topic 171 17.90 0.2
Topic 159 25.72 Topic 172 9.86
Difference
Topic 160 2.01 Topic 173 52.05 0
Topic 161 25.36 Topic 174 51.90
Topic 162 9.66 Topic 175 4.76
−0.2
Topic 163 18.72
−0.4
−0.6
−0.8
−1
151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175
Topic Identifier
129
hagen FUHegpyynl102 DS-BILI-X2DE-CLEF2006
Docs Cutoff Levels Precision at DCL (%)
Domain−Specific Bilingual German track − Retrieved documents vs Precision
100%
5 docs 49.60
FUHegpyynl102
10 docs 47.20 90%
15 docs 42.67
80%
20 docs 40.20
30 docs 36.53 70%
100 docs 30.16
60%
200 docs 23.86
R−Precision
500 docs 13.78 50%
1000 docs 8.54 40%
R-Precision (precision after R document retrieved,
where R = Relevant retrieved) 30%
25.52
20%
10%
0%
5 10 15 20 30 100 200 500 1000
Retrieved Documents (logarithmic scale)
Exact R-Precision Domain−Specific Bilingual German track − Box plot of the Topics of the Experiment
Maximum 0.7226
Minimum 0.0215
First Quartile 0.1223
Second Quartile 0.2245
Third Quartile 0.3296
Interquartile range 0.2074
Mean 0.2552
Standard Deviation 0.1772
Lower Outlier Threshold 0.0215
Upper Outlier Threshold 0.6023 0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Exact R−Precision
Mean With No Outliers 0.2357
Std With No Outliers 0.1512
Domain−Specific Bilingual German track − Distribution of the Topics of the Experiment
4
Number of Topics of the Experiment
3.5
3
2.5
2
1.5
1
0.5
0
0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Exact R−Precision
Precision averages (%) for individual 1
Domain−Specific Bilingual German track − Comparison to Median Mean Exact R−Precision by Topic (Topics 151 to 175)
queries FUHegpyynl102
Topic 151 17.32 Topic 164 26.92 0.8
Topic 152 6.73 Topic 165 32.88
Topic 153 8.00 Topic 166 32.26
0.6
Topic 154 72.26 Topic 167 4.44
Topic 155 35.23 Topic 168 22.45
Topic 156 2.15 Topic 169 18.18 0.4
Topic 157 18.46 Topic 170 35.90
Topic 158 16.85 Topic 171 31.31 0.2
Topic 159 28.70 Topic 172 13.64
Difference
Topic 160 6.25 Topic 173 56.75 0
Topic 161 33.23 Topic 174 60.23
Topic 162 21.12 Topic 175 7.81
−0.2
Topic 163 28.79
−0.4
−0.6
−0.8
−1
151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175
Topic Identifier
130
hagen FUHegpyydl102 DS-BILI-X2DE-CLEF2006
Overall statistics for 25 queries : Priority 4
Total number of documents over all queries Query Construction AUTOMATIC
Retrieved 25,000 Source Language English
Relevant 3,759 Topic Fields title, description
Relevant retrieved 2,120 Pooled true
Geometric Mean Average Precision 0.0897 Fourth run, EN
Binary Preference (BPREF) 0.2350
Interploated Recall (%) Precision Averages (%)
Domain−Specific Bilingual German track − Interpolated Recall vs Average Precision
100%
0 64.89
FUHegpyydl102
10 38.07 90%
20 30.28
80%
30 25.24
40 20.82 70%
50 15.84
Average Precision
60%
60 12.37
70 7.94 50%
80 5.09 40%
90 1.32
30%
100 0.00
Average precision (non-interpolated) for all 20%
relevant documents (averaged over queries)
17.68 10%
0%
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
Interpolated Recall
Mean Average Precision Domain−Specific Bilingual German track − Box plot of the Topics of the Experiment
Maximum 0.5752
Minimum 0.0016
First Quartile 0.0438
Second Quartile 0.0951
Third Quartile 0.2709
Interquartile range 0.2272
Mean 0.1768
Standard Deviation 0.1732
Lower Outlier Threshold 0.0016
Upper Outlier Threshold 0.5752 0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Mean Average Precision
Mean With No Outliers 0.1768
Std With No Outliers 0.1732
Domain−Specific Bilingual German track − Distribution of the Topics of the Experiment
8
Number of Topics of the Experiment
7
6
5
4
3
2
1
0
0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Mean Average Precision
Precision averages (%) for individual 1
Domain−Specific Bilingual German track − Comparison to Median Mean Average Precision by Topic (Topics 151 to 175)
queries FUHegpyydl102
Topic 151 9.51 Topic 164 1.89 0.8
Topic 152 0.77 Topic 165 18.68
Topic 153 5.47 Topic 166 31.06
0.6
Topic 154 43.94 Topic 167 2.65
Topic 155 21.00 Topic 168 12.33
Topic 156 4.87 Topic 169 6.96 0.4
Topic 157 0.16 Topic 170 25.77
Topic 158 1.20 Topic 171 22.22 0.2
Topic 159 53.97 Topic 172 8.43
Difference
Topic 160 2.89 Topic 173 45.84 0
Topic 161 32.37 Topic 174 57.52
Topic 162 9.22 Topic 175 13.99
−0.2
Topic 163 9.28
−0.4
−0.6
−0.8
−1
151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175
Topic Identifier
131
hagen FUHegpyydl102 DS-BILI-X2DE-CLEF2006
Docs Cutoff Levels Precision at DCL (%)
Domain−Specific Bilingual German track − Retrieved documents vs Precision
100%
5 docs 44.00
FUHegpyydl102
10 docs 38.80 90%
15 docs 37.33
80%
20 docs 36.60
30 docs 33.20 70%
100 docs 28.40
60%
200 docs 22.14
R−Precision
500 docs 13.40 50%
1000 docs 8.48 40%
R-Precision (precision after R document retrieved,
where R = Relevant retrieved) 30%
23.26
20%
10%
0%
5 10 15 20 30 100 200 500 1000
Retrieved Documents (logarithmic scale)
Exact R-Precision Domain−Specific Bilingual German track − Box plot of the Topics of the Experiment
Maximum 0.6364
Minimum 0.0000
First Quartile 0.0740
Second Quartile 0.2041
Third Quartile 0.3606
Interquartile range 0.2866
Mean 0.2326
Standard Deviation 0.1827
Lower Outlier Threshold 0.0000
Upper Outlier Threshold 0.6364 0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Exact R−Precision
Mean With No Outliers 0.2326
Std With No Outliers 0.1827
Domain−Specific Bilingual German track − Distribution of the Topics of the Experiment
4
Number of Topics of the Experiment
3.5
3
2.5
2
1.5
1
0.5
0
0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Exact R−Precision
Precision averages (%) for individual 1
Domain−Specific Bilingual German track − Comparison to Median Mean Exact R−Precision by Topic (Topics 151 to 175)
queries FUHegpyydl102
Topic 151 22.83 Topic 164 0.00 0.8
Topic 152 2.88 Topic 165 27.40
Topic 153 8.00 Topic 166 36.56
0.6
Topic 154 54.01 Topic 167 4.44
Topic 155 25.91 Topic 168 20.41
Topic 156 10.75 Topic 169 9.09 0.4
Topic 157 0.00 Topic 170 35.90
Topic 158 5.62 Topic 171 30.81 0.2
Topic 159 52.78 Topic 172 18.18
Difference
Topic 160 5.56 Topic 173 50.40 0
Topic 161 38.92 Topic 174 63.64
Topic 162 17.24 Topic 175 18.75
−0.2
Topic 163 21.40
−0.4
−0.6
−0.8
−1
151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175
Topic Identifier
132
hagen FUHegpyydl500R DS-BILI-X2DE-CLEF2006
Overall statistics for 25 queries : Priority 5
Total number of documents over all queries Query Construction AUTOMATIC
Retrieved 25,000 Source Language English
Relevant 3,759 Topic Fields title, description
Relevant retrieved 2,422 Pooled true
Geometric Mean Average Precision 0.1266 Fifth run, EN
Binary Preference (BPREF) 0.2731
Interploated Recall (%) Precision Averages (%)
Domain−Specific Bilingual German track − Interpolated Recall vs Average Precision
100%
0 65.87
FUHegpyydl500R
10 41.03 90%
20 35.77
80%
30 31.78
40 26.09 70%
50 20.49
Average Precision
60%
60 16.11
70 11.69 50%
80 6.82 40%
90 1.80
30%
100 0.00
Average precision (non-interpolated) for all 20%
relevant documents (averaged over queries)
21.80 10%
0%
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
Interpolated Recall
Mean Average Precision Domain−Specific Bilingual German track − Box plot of the Topics of the Experiment
Maximum 0.6746
Minimum 0.0019
First Quartile 0.0750
Second Quartile 0.1603
Third Quartile 0.3162
Interquartile range 0.2413
Mean 0.2180
Standard Deviation 0.1884
Lower Outlier Threshold 0.0019
Upper Outlier Threshold 0.6746 0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Mean Average Precision
Mean With No Outliers 0.2180
Std With No Outliers 0.1884
Domain−Specific Bilingual German track − Distribution of the Topics of the Experiment
4
Number of Topics of the Experiment
3.5
3
2.5
2
1.5
1
0.5
0
0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Mean Average Precision
Precision averages (%) for individual 1
Domain−Specific Bilingual German track − Comparison to Median Mean Average Precision by Topic (Topics 151 to 175)
queries FUHegpyydl500R
Topic 151 7.35 Topic 164 2.06 0.8
Topic 152 1.59 Topic 165 20.68
Topic 153 32.60 Topic 166 57.06
0.6
Topic 154 29.35 Topic 167 9.32
Topic 155 15.92 Topic 168 31.30
Topic 156 10.36 Topic 169 11.72 0.4
Topic 157 0.19 Topic 170 22.28
Topic 158 2.29 Topic 171 21.20 0.2
Topic 159 67.46 Topic 172 7.03
Difference
Topic 160 16.43 Topic 173 48.69 0
Topic 161 48.50 Topic 174 46.77
Topic 162 16.03 Topic 175 7.55
−0.2
Topic 163 11.33
−0.4
−0.6
−0.8
−1
151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175
Topic Identifier
133
hagen FUHegpyydl500R DS-BILI-X2DE-CLEF2006
Docs Cutoff Levels Precision at DCL (%)
Domain−Specific Bilingual German track − Retrieved documents vs Precision
100%
5 docs 42.40
FUHegpyydl500R
10 docs 40.00 90%
15 docs 40.53
80%
20 docs 39.00
30 docs 37.87 70%
100 docs 31.76
60%
200 docs 25.72
R−Precision
500 docs 15.29 50%
1000 docs 9.69 40%
R-Precision (precision after R document retrieved,
where R = Relevant retrieved) 30%
27.77
20%
10%
0%
5 10 15 20 30 100 200 500 1000
Retrieved Documents (logarithmic scale)
Exact R-Precision Domain−Specific Bilingual German track − Box plot of the Topics of the Experiment
Maximum 0.6019
Minimum 0.0000
First Quartile 0.1470
Second Quartile 0.2564
Third Quartile 0.4089
Interquartile range 0.2619
Mean 0.2777
Standard Deviation 0.1761
Lower Outlier Threshold 0.0000
Upper Outlier Threshold 0.6019 0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Exact R−Precision
Mean With No Outliers 0.2777
Std With No Outliers 0.1761
Domain−Specific Bilingual German track − Distribution of the Topics of the Experiment
5
Number of Topics of the Experiment
4
3
2
1
0
0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Exact R−Precision
Precision averages (%) for individual 1
Domain−Specific Bilingual German track − Comparison to Median Mean Exact R−Precision by Topic (Topics 151 to 175)
queries FUHegpyydl500R
Topic 151 16.54 Topic 164 0.00 0.8
Topic 152 10.10 Topic 165 31.51
Topic 153 36.00 Topic 166 54.84
0.6
Topic 154 39.78 Topic 167 17.78
Topic 155 19.69 Topic 168 44.22
Topic 156 15.05 Topic 169 9.09 0.4
Topic 157 1.54 Topic 170 25.64
Topic 158 7.87 Topic 171 35.35 0.2
Topic 159 60.19 Topic 172 13.64
Difference
Topic 160 28.47 Topic 173 52.38 0
Topic 161 49.37 Topic 174 53.98
Topic 162 32.33 Topic 175 17.19
−0.2
Topic 163 21.79
−0.4
−0.6
−0.8
−1
151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175
Topic Identifier
134
berkeley BERK_BI_ENRU_T2FB_P DS-BILI-X2RU-CLEF2006
Overall statistics for 25 queries : Priority 2
Total number of documents over all queries Query Construction AUTOMATIC
Retrieved 25,000 Source Language German
Relevant 913 Topic Fields title, description
Relevant retrieved 652 Pooled true
Geometric Mean Average Precision 0.0216 English to Russian using PROMT
Binary Preference (BPREF) 0.1646
Interploated Recall (%) Precision Averages (%)
Domain−Specific Bilingual Russian track − Interpolated Recall vs Average Precision
100%
0 43.99
BERK_BI_ENRU_T2FB_P
10 30.68 90%
20 27.38
80%
30 20.29
40 18.04 70%
50 16.84
Average Precision
60%
60 14.27
70 11.21 50%
80 8.30 40%
90 4.58
30%
100 0.27
Average precision (non-interpolated) for all 20%
relevant documents (averaged over queries)
16.48 10%
0%
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
Interpolated Recall
Mean Average Precision Domain−Specific Bilingual Russian track − Box plot of the Topics of the Experiment
Maximum 0.7333
Minimum 0.0000
First Quartile 0.0102
Second Quartile 0.0483
Third Quartile 0.2641
Interquartile range 0.2538
Mean 0.1648
Standard Deviation 0.2264
Lower Outlier Threshold 0.0000
Upper Outlier Threshold 0.6215 0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Mean Average Precision
Mean With No Outliers 0.1411
Std With No Outliers 0.1971
Domain−Specific Bilingual Russian track − Distribution of the Topics of the Experiment
15
Number of Topics of the Experiment
10
5
0
0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Mean Average Precision
Precision averages (%) for individual 1
Domain−Specific Bilingual Russian track − Comparison to Median Mean Average Precision by Topic (Topics 151 to 175)
queries BERK_BI_ENRU_T2FB_P
Topic 151 7.19 Topic 164 2.27 0.8
Topic 152 50.03 Topic 165 1.91
Topic 153 14.10 Topic 166 5.42
0.6
Topic 154 28.74 Topic 167 1.18
Topic 155 73.33 Topic 168 25.63
Topic 156 0.78 Topic 169 11.51 0.4
Topic 157 0.00 Topic 170 3.96
Topic 158 3.75 Topic 171 1.10 0.2
Topic 159 56.16 Topic 172 0.30
Difference
Topic 160 0.08 Topic 173 11.90 0
Topic 161 62.15 Topic 174 0.00
Topic 162 4.83 Topic 175 0.00
−0.2
Topic 163 45.73
−0.4
−0.6
−0.8
−1
151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175
Topic Identifier
135
berkeley BERK_BI_ENRU_T2FB_P DS-BILI-X2RU-CLEF2006
Docs Cutoff Levels Precision at DCL (%)
Domain−Specific Bilingual Russian track − Retrieved documents vs Precision
100%
5 docs 23.20
BERK_BI_ENRU_T2FB_P
10 docs 22.00 90%
15 docs 21.07
80%
20 docs 21.20
30 docs 19.73 70%
100 docs 15.76
60%
200 docs 10.44
R−Precision
500 docs 4.82 50%
1000 docs 2.61 40%
R-Precision (precision after R document retrieved,
where R = Relevant retrieved) 30%
17.38
20%
10%
0%
5 10 15 20 30 100 200 500 1000
Retrieved Documents (logarithmic scale)
Exact R-Precision Domain−Specific Bilingual Russian track − Box plot of the Topics of the Experiment
Maximum 0.7027
Minimum 0.0000
First Quartile 0.0000
Second Quartile 0.0435
Third Quartile 0.2961
Interquartile range 0.2961
Mean 0.1738
Standard Deviation 0.2327
Lower Outlier Threshold 0.0000
Upper Outlier Threshold 0.7027 0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Exact R−Precision
Mean With No Outliers 0.1738
Std With No Outliers 0.2327
Domain−Specific Bilingual Russian track − Distribution of the Topics of the Experiment
15
Number of Topics of the Experiment
10
5
0
0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Exact R−Precision
Precision averages (%) for individual 1
Domain−Specific Bilingual Russian track − Comparison to Median Mean Exact R−Precision by Topic (Topics 151 to 175)
queries BERK_BI_ENRU_T2FB_P
Topic 151 6.25 Topic 164 0.00 0.8
Topic 152 45.35 Topic 165 0.00
Topic 153 13.04 Topic 166 0.00
0.6
Topic 154 43.43 Topic 167 0.00
Topic 155 70.27 Topic 168 25.00
Topic 156 0.00 Topic 169 25.00 0.4
Topic 157 0.00 Topic 170 4.35
Topic 158 0.00 Topic 171 0.00 0.2
Topic 159 60.00 Topic 172 0.00
Difference
Topic 160 2.70 Topic 173 20.00 0
Topic 161 55.68 Topic 174 0.00
Topic 162 8.62 Topic 175 0.00
−0.2
Topic 163 54.87
−0.4
−0.6
−0.8
−1
151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175
Topic Identifier
136
berkeley BERK_BI_ENRU_T2FB_B DS-BILI-X2RU-CLEF2006
Overall statistics for 25 queries : Priority 1
Total number of documents over all queries Query Construction AUTOMATIC
Retrieved 25,000 Source Language German
Relevant 913 Topic Fields title, description
Relevant retrieved 571 Pooled true
Geometric Mean Average Precision 0.0086 English to Russian using Babelfish
Binary Preference (BPREF) 0.1156
Interploated Recall (%) Precision Averages (%)
Domain−Specific Bilingual Russian track − Interpolated Recall vs Average Precision
100%
0 29.02
BERK_BI_ENRU_T2FB_B
10 23.91 90%
20 16.13
80%
30 13.98
40 12.90 70%
50 11.21
Average Precision
60%
60 10.29
70 8.63 50%
80 6.43 40%
90 3.25
30%
100 0.18
Average precision (non-interpolated) for all 20%
relevant documents (averaged over queries)
11.55 10%
0%
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
Interpolated Recall
Mean Average Precision Domain−Specific Bilingual Russian track − Box plot of the Topics of the Experiment
Maximum 0.7004
Minimum 0.0000
First Quartile 0.0028
Second Quartile 0.0131
Third Quartile 0.1162
Interquartile range 0.1134
Mean 0.1155
Standard Deviation 0.1982
Lower Outlier Threshold 0.0000
Upper Outlier Threshold 0.2756 0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Mean Average Precision
Mean With No Outliers 0.0392
Std With No Outliers 0.0682
Domain−Specific Bilingual Russian track − Distribution of the Topics of the Experiment
20
Number of Topics of the Experiment
15
10
5
0
0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Mean Average Precision
Precision averages (%) for individual 1
Domain−Specific Bilingual Russian track − Comparison to Median Mean Average Precision by Topic (Topics 151 to 175)
queries BERK_BI_ENRU_T2FB_B
Topic 151 10.69 Topic 164 0.29 0.8
Topic 152 31.74 Topic 165 1.31
Topic 153 9.27 Topic 166 0.00
0.6
Topic 154 2.63 Topic 167 0.96
Topic 155 0.06 Topic 168 0.23
Topic 156 1.30 Topic 169 0.35 0.4
Topic 157 0.00 Topic 170 8.08
Topic 158 14.41 Topic 171 1.53 0.2
Topic 159 59.91 Topic 172 1.81
Difference
Topic 160 1.02 Topic 173 0.77 0
Topic 161 70.04 Topic 174 0.00
Topic 162 27.56 Topic 175 0.00
−0.2
Topic 163 44.88
−0.4
−0.6
−0.8
−1
151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175
Topic Identifier
137
berkeley BERK_BI_ENRU_T2FB_B DS-BILI-X2RU-CLEF2006
Docs Cutoff Levels Precision at DCL (%)
Domain−Specific Bilingual Russian track − Retrieved documents vs Precision
100%
5 docs 19.20
BERK_BI_ENRU_T2FB_B
10 docs 16.00 90%
15 docs 16.53
80%
20 docs 15.80
30 docs 14.80 70%
100 docs 11.72
60%
200 docs 8.26
R−Precision
500 docs 4.14 50%
1000 docs 2.28 40%
R-Precision (precision after R document retrieved,
where R = Relevant retrieved) 30%
12.91
20%
10%
0%
5 10 15 20 30 100 200 500 1000
Retrieved Documents (logarithmic scale)
Exact R-Precision Domain−Specific Bilingual Russian track − Box plot of the Topics of the Experiment
Maximum 0.6591
Minimum 0.0000
First Quartile 0.0000
Second Quartile 0.0135
Third Quartile 0.1475
Interquartile range 0.1475
Mean 0.1291
Standard Deviation 0.2020
Lower Outlier Threshold 0.0000
Upper Outlier Threshold 0.2414 0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Exact R−Precision
Mean With No Outliers 0.0490
Std With No Outliers 0.0718
Domain−Specific Bilingual Russian track − Distribution of the Topics of the Experiment
15
Number of Topics of the Experiment
10
5
0
0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Exact R−Precision
Precision averages (%) for individual 1
Domain−Specific Bilingual Russian track − Comparison to Median Mean Exact R−Precision by Topic (Topics 151 to 175)
queries BERK_BI_ENRU_T2FB_B
Topic 151 12.50 Topic 164 0.00 0.8
Topic 152 40.70 Topic 165 5.56
Topic 153 13.04 Topic 166 0.00
0.6
Topic 154 16.16 Topic 167 0.00
Topic 155 1.35 Topic 168 0.00
Topic 156 0.00 Topic 169 0.00 0.4
Topic 157 0.00 Topic 170 8.70
Topic 158 14.29 Topic 171 0.00 0.2
Topic 159 60.00 Topic 172 0.00
Difference
Topic 160 7.21 Topic 173 0.00 0
Topic 161 65.91 Topic 174 0.00
Topic 162 24.14 Topic 175 0.00
−0.2
Topic 163 53.10
−0.4
−0.6
−0.8
−1
151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175
Topic Identifier
138
berkeley BERK_BI_DERU_T2FB_P DS-BILI-X2RU-CLEF2006
Overall statistics for 25 queries : Priority 3
Total number of documents over all queries Query Construction AUTOMATIC
Retrieved 25,000 Source Language German
Relevant 913 Topic Fields title, description
Relevant retrieved 456 Pooled true
Geometric Mean Average Precision 0.0077 German to Russian using PROMT
Binary Preference (BPREF) 0.1034
Interploated Recall (%) Precision Averages (%)
Domain−Specific Bilingual Russian track − Interpolated Recall vs Average Precision
100%
0 31.91
BERK_BI_DERU_T2FB_P
10 21.87 90%
20 20.61
80%
30 14.56
40 12.49 70%
50 11.38
Average Precision
60%
60 6.50
70 5.51 50%
80 3.97 40%
90 2.37
30%
100 1.04
Average precision (non-interpolated) for all 20%
relevant documents (averaged over queries)
10.83 10%
0%
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
Interpolated Recall
Mean Average Precision Domain−Specific Bilingual Russian track − Box plot of the Topics of the Experiment
Maximum 0.6726
Minimum 0.0000
First Quartile 0.0012
Second Quartile 0.0358
Third Quartile 0.1411
Interquartile range 0.1399
Mean 0.1083
Standard Deviation 0.1720
Lower Outlier Threshold 0.0000
Upper Outlier Threshold 0.3344 0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Mean Average Precision
Mean With No Outliers 0.0666
Std With No Outliers 0.0944
Domain−Specific Bilingual Russian track − Distribution of the Topics of the Experiment
15
Number of Topics of the Experiment
10
5
0
0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Mean Average Precision
Precision averages (%) for individual 1
Domain−Specific Bilingual Russian track − Comparison to Median Mean Average Precision by Topic (Topics 151 to 175)
queries BERK_BI_DERU_T2FB_P
Topic 151 13.94 Topic 164 8.13 0.8
Topic 152 0.06 Topic 165 14.05
Topic 153 14.29 Topic 166 3.66
0.6
Topic 154 7.52 Topic 167 1.41
Topic 155 0.00 Topic 168 29.52
Topic 156 2.68 Topic 169 3.58 0.4
Topic 157 50.29 Topic 170 1.26
Topic 158 0.27 Topic 171 0.33 0.2
Topic 159 67.26 Topic 172 0.14
Difference
Topic 160 0.00 Topic 173 0.00 0
Topic 161 4.14 Topic 174 0.00
Topic 162 33.44 Topic 175 0.00
−0.2
Topic 163 14.68
−0.4
−0.6
−0.8
−1
151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175
Topic Identifier
139
berkeley BERK_BI_DERU_T2FB_P DS-BILI-X2RU-CLEF2006
Docs Cutoff Levels Precision at DCL (%)
Domain−Specific Bilingual Russian track − Retrieved documents vs Precision
100%
5 docs 12.00
BERK_BI_DERU_T2FB_P
10 docs 10.80 90%
15 docs 10.13
80%
20 docs 10.20
30 docs 10.00 70%
100 docs 8.32
60%
200 docs 5.50
R−Precision
500 docs 3.01 50%
1000 docs 1.82 40%
R-Precision (precision after R document retrieved,
where R = Relevant retrieved) 30%
11.87
20%
10%
0%
5 10 15 20 30 100 200 500 1000
Retrieved Documents (logarithmic scale)
Exact R-Precision Domain−Specific Bilingual Russian track − Box plot of the Topics of the Experiment
Maximum 0.7143
Minimum 0.0000
First Quartile 0.0000
Second Quartile 0.0000
Third Quartile 0.2107
Interquartile range 0.2107
Mean 0.1187
Standard Deviation 0.1866
Lower Outlier Threshold 0.0000
Upper Outlier Threshold 0.5000 0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Exact R−Precision
Mean With No Outliers 0.0939
Std With No Outliers 0.1424
Domain−Specific Bilingual Russian track − Distribution of the Topics of the Experiment
15
Number of Topics of the Experiment
10
5
0
0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Exact R−Precision
Precision averages (%) for individual 1
Domain−Specific Bilingual Russian track − Comparison to Median Mean Exact R−Precision by Topic (Topics 151 to 175)
queries BERK_BI_DERU_T2FB_P
Topic 151 25.00 Topic 164 0.00 0.8
Topic 152 2.33 Topic 165 19.44
Topic 153 13.04 Topic 166 0.00
0.6
Topic 154 23.23 Topic 167 0.00
Topic 155 0.00 Topic 168 25.00
Topic 156 0.00 Topic 169 0.00 0.4
Topic 157 50.00 Topic 170 0.00
Topic 158 0.00 Topic 171 0.00 0.2
Topic 159 71.43 Topic 172 0.00
Difference
Topic 160 0.00 Topic 173 0.00 0
Topic 161 9.09 Topic 174 0.00
Topic 162 37.93 Topic 175 0.00
−0.2
Topic 163 20.35
−0.4
−0.6
−0.8
−1
151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175
Topic Identifier
140
berkeley BERK_MU_DE_T2FB_B_CMBZ DS-MULTI-CLEF2006
Overall statistics for 25 queries : Priority 1
Total number of documents over all queries Query Construction AUTOMATIC
Retrieved 25,000 Source Language German
Relevant 8,911 Topic Fields title, description
Relevant retrieved 1,578 Pooled false
Geometric Mean Average Precision 0.0143 German to Russian using PROMT, and German to
Binary Preference (BPREF) 0.1606 English using Babelfish
Interploated Recall (%) Precision Averages (%)
Domain−Specific Multilingual track − Interpolated Recall vs Average Precision
100%
0 31.82
BERK_MU_DE_T2FB_B_CMBZ
10 14.65 90%
20 11.17
80%
30 6.05
40 3.21 70%
50 2.24
Average Precision
60%
60 1.44
70 0.00 50%
80 0.00 40%
90 0.00
30%
100 0.00
Average precision (non-interpolated) for all 20%
relevant documents (averaged over queries)
4.67 10%
0%
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
Interpolated Recall
Mean Average Precision Domain−Specific Multilingual track − Box plot of the Topics of the Experiment
Maximum 0.3352
Minimum 0.0001
First Quartile 0.0037
Second Quartile 0.0240
Third Quartile 0.0521
Interquartile range 0.0484
Mean 0.0467
Standard Deviation 0.0740
Lower Outlier Threshold 0.0001
Upper Outlier Threshold 0.0961 0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Mean Average Precision
Mean With No Outliers 0.0277
Std With No Outliers 0.0285
Domain−Specific Multilingual track − Distribution of the Topics of the Experiment
20
Number of Topics of the Experiment
15
10
5
0
0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Mean Average Precision
Precision averages (%) for individual 1
Domain−Specific Multilingual track − Comparison to Median Mean Average Precision by Topic (Topics 151 to 175)
queries BERK_MU_DE_T2FB_B_CMBZ
Topic 151 1.12 Topic 164 5.47 0.8
Topic 152 0.01 Topic 165 2.40
Topic 153 9.16 Topic 166 0.28
0.6
Topic 154 3.42 Topic 167 1.96
Topic 155 0.72 Topic 168 4.21
Topic 156 6.43 Topic 169 0.03 0.4
Topic 157 5.12 Topic 170 19.52
Topic 158 0.62 Topic 171 33.52 0.2
Topic 159 2.67 Topic 172 4.68
Difference
Topic 160 0.17 Topic 173 2.35 0
Topic 161 9.61 Topic 174 0.39
Topic 162 0.30 Topic 175 0.17
−0.2
Topic 163 2.52
−0.4
−0.6
−0.8
−1
151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175
Topic Identifier
141
berkeley BERK_MU_DE_T2FB_B_CMBZ DS-MULTI-CLEF2006
Docs Cutoff Levels Precision at DCL (%)
Domain−Specific Multilingual track − Retrieved documents vs Precision
100%
5 docs 12.00
BERK_MU_DE_T2FB_B_CMBZ
10 docs 14.40 90%
15 docs 15.73
80%
20 docs 17.20
30 docs 18.27 70%
100 docs 16.36
60%
200 docs 13.24
R−Precision
500 docs 8.52 50%
1000 docs 6.31 40%
R-Precision (precision after R document retrieved,
where R = Relevant retrieved) 30%
11.15
20%
10%
0%
5 10 15 20 30 100 200 500 1000
Retrieved Documents (logarithmic scale)
Exact R-Precision Domain−Specific Multilingual track − Box plot of the Topics of the Experiment
Maximum 0.4476
Minimum 0.0000
First Quartile 0.0336
Second Quartile 0.0778
Third Quartile 0.1409
Interquartile range 0.1073
Mean 0.1115
Standard Deviation 0.1085
Lower Outlier Threshold 0.0000
Upper Outlier Threshold 0.2522 0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Exact R−Precision
Mean With No Outliers 0.0873
Std With No Outliers 0.0698
Domain−Specific Multilingual track − Distribution of the Topics of the Experiment
8
Number of Topics of the Experiment
7
6
5
4
3
2
1
0
0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Exact R−Precision
Precision averages (%) for individual 1
Domain−Specific Multilingual track − Comparison to Median Mean Exact R−Precision by Topic (Topics 151 to 175)
queries BERK_MU_DE_T2FB_B_CMBZ
Topic 151 5.33 Topic 164 17.56 0.8
Topic 152 0.47 Topic 165 5.33
Topic 153 22.74 Topic 166 2.05
0.6
Topic 154 12.48 Topic 167 8.45
Topic 155 4.20 Topic 168 13.20
Topic 156 13.04 Topic 169 0.00 0.4
Topic 157 13.58 Topic 170 33.33
Topic 158 5.12 Topic 171 44.76 0.2
Topic 159 7.78 Topic 172 15.62
Difference
Topic 160 3.08 Topic 173 7.63 0
Topic 161 25.22 Topic 174 3.46
Topic 162 2.32 Topic 175 1.71
−0.2
Topic 163 10.35
−0.4
−0.6
−0.8
−1
151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175
Topic Identifier
142
berkeley BERK_MU_EN_T2FB_B_CMBZ DS-MULTI-CLEF2006
Overall statistics for 25 queries : Priority 2
Total number of documents over all queries Query Construction AUTOMATIC
Retrieved 25,000 Source Language German
Relevant 8,911 Topic Fields title, description
Relevant retrieved 2,422 Pooled false
Geometric Mean Average Precision 0.0298 English to German and Russian using Babelfish
Binary Preference (BPREF) 0.2097
Interploated Recall (%) Precision Averages (%)
Domain−Specific Multilingual track − Interpolated Recall vs Average Precision
100%
0 34.88
BERK_MU_EN_T2FB_B_CMBZ
10 25.04 90%
20 18.87
80%
30 12.88
40 6.47 70%
50 2.99
Average Precision
60%
60 0.30
70 0.00 50%
80 0.00 40%
90 0.00
30%
100 0.00
Average precision (non-interpolated) for all 20%
relevant documents (averaged over queries)
7.53 10%
0%
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
Interpolated Recall
Mean Average Precision Domain−Specific Multilingual track − Box plot of the Topics of the Experiment
Maximum 0.2452
Minimum 0.0013
First Quartile 0.0061
Second Quartile 0.0430
Third Quartile 0.1221
Interquartile range 0.1161
Mean 0.0753
Standard Deviation 0.0784
Lower Outlier Threshold 0.0013
Upper Outlier Threshold 0.2452 0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Mean Average Precision
Mean With No Outliers 0.0753
Std With No Outliers 0.0784
Domain−Specific Multilingual track − Distribution of the Topics of the Experiment
15
Number of Topics of the Experiment
10
5
0
0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Mean Average Precision
Precision averages (%) for individual 1
Domain−Specific Multilingual track − Comparison to Median Mean Average Precision by Topic (Topics 151 to 175)
queries BERK_MU_EN_T2FB_B_CMBZ
Topic 151 24.52 Topic 164 1.38 0.8
Topic 152 8.86 Topic 165 0.33
Topic 153 10.64 Topic 166 14.87
0.6
Topic 154 4.30 Topic 167 0.62
Topic 155 10.44 Topic 168 5.50
Topic 156 1.05 Topic 169 1.94 0.4
Topic 157 22.74 Topic 170 14.57
Topic 158 11.43 Topic 171 19.86 0.2
Topic 159 0.29 Topic 172 4.23
Difference
Topic 160 2.03 Topic 173 0.58 0
Topic 161 19.31 Topic 174 8.15
Topic 162 0.16 Topic 175 0.13
−0.2
Topic 163 0.41
−0.4
−0.6
−0.8
−1
151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175
Topic Identifier
143
berkeley BERK_MU_EN_T2FB_B_CMBZ DS-MULTI-CLEF2006
Docs Cutoff Levels Precision at DCL (%)
Domain−Specific Multilingual track − Retrieved documents vs Precision
100%
5 docs 8.80
BERK_MU_EN_T2FB_B_CMBZ
10 docs 8.80 90%
15 docs 14.67
80%
20 docs 17.00
30 docs 21.33 70%
100 docs 24.56
60%
200 docs 20.72
R−Precision
500 docs 14.18 50%
1000 docs 9.69 40%
R-Precision (precision after R document retrieved,
where R = Relevant retrieved) 30%
15.97
20%
10%
0%
5 10 15 20 30 100 200 500 1000
Retrieved Documents (logarithmic scale)
Exact R-Precision Domain−Specific Multilingual track − Box plot of the Topics of the Experiment
Maximum 0.3848
Minimum 0.0085
First Quartile 0.0298
Second Quartile 0.1379
Third Quartile 0.2564
Interquartile range 0.2266
Mean 0.1597
Standard Deviation 0.1299
Lower Outlier Threshold 0.0085
Upper Outlier Threshold 0.3848 0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Exact R−Precision
Mean With No Outliers 0.1597
Std With No Outliers 0.1299
Domain−Specific Multilingual track − Distribution of the Topics of the Experiment
10
Number of Topics of the Experiment
8
6
4
2
0
0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Exact R−Precision
Precision averages (%) for individual 1
Domain−Specific Multilingual track − Comparison to Median Mean Exact R−Precision by Topic (Topics 151 to 175)
queries BERK_MU_EN_T2FB_B_CMBZ
Topic 151 38.48 Topic 164 5.34 0.8
Topic 152 24.09 Topic 165 1.78
Topic 153 25.06 Topic 166 29.86
0.6
Topic 154 13.79 Topic 167 1.41
Topic 155 24.52 Topic 168 18.40
Topic 156 2.42 Topic 169 4.92 0.4
Topic 157 37.04 Topic 170 27.38
Topic 158 22.79 Topic 171 35.60 0.2
Topic 159 3.17 Topic 172 12.50
Difference
Topic 160 9.42 Topic 173 4.39 0
Topic 161 33.76 Topic 174 17.87
Topic 162 2.16 Topic 175 0.85
−0.2
Topic 163 2.34
−0.4
−0.6
−0.8
−1
151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175
Topic Identifier
144