Appendix B
Results of the Core Tracks
Ad-hoc Robust Tasks
Prepared by:
Giorgio Maria Di Nunzio and Nicola Ferro
{dinunzio, ferro}@dei.unipd.it
Department of Information Engineering
University of Padua
Italy
1
2
Introduction
3
4
Results for CLEF 2006 Ad-Hoc Robust Tracks
The following pages contain the results and graphs for all the experiments that have been officially submitted to the CLEF 2006
campaign for the Ad-Hoc Robust track.
This document is divided in three main parts:
1. List of submitted experiments
2. Track overview results and graphs
3. Individual experiment results and graphs
1. List of Submitted Experiments
This section gives a listing of all experiments and their characteristics:
Participant: the name of the participant who submitted the experiment.
Country: country of the participant.
Identifier: unique identifier for each experiment.
Task: track/task to which the experiment belongs.
Topic language: language of the topics used to create the experiment (ISO identifiers for language).
Topic fields: identifies the parts of the topics used to create the experiment (T = title, D = Description, N = Narrative).
Query constr.: identifies how the query has been constructed from topic fields (manual/automatic).
Pool: specifies if experiment was used for relevance assessment pooling.
2. Track Overview Results and Graphs
For each track/task graphs and tables are shown in order to compare the experiments.
The graphs and tables contain the following information:
- Mandatory experiments title + description (TD) of at most top five participants
- Interpolated recall vs precision averages plot
- Average precision comparison to median plot
- All experiments
- Average precision box plot
- Average precision Tukey t-test plot
- Mandatory experiments title + description (TD) of at most top five participants
- Document cutoff levels (DCL) vs precision at DCL plot
- R-Precision comparison to median plot
- All experiments
- R-Precision box plot
- R-Precision Tukey t-test plot
- A table with descriptive statistics of performance figures for each topic
3. Individual Experiment Results and Graphs
This section provides the individual results for each official experiment. For each experiment the following tables and graphs are
shown:
- Overall statistics and information
- Interpolated recall vs precision averages plot
- Average precision statistics and box plot
- Average precision comparison to median plot
- Document cutoff levels vs precision at DCL plot
- R-Precision statistics and box plot
- R-Precision comparison to median plot
5
6
List of Submitted Experiments
7
8
Participant Country Experiment ID Task Topic Topic Query Pool
Lang. Fields Construction
colesir Spain CoLesIRdeTst AH-ROBUST-MONO-DE-CLEF2006 de TD AUTOMATIC no
daedalus Spain deFSdeR2S AH-ROBUST-MONO-DE-CLEF2006 de TD AUTOMATIC no
daedalus Spain deFSdeR3S AH-ROBUST-MONO-DE-CLEF2006 de TDN AUTOMATIC no
daedalus Spain dex021deRFW3FS3 AH-ROBUST-MONO-DE-CLEF2006 de TDN AUTOMATIC no
daedalus Spain dex011deRFW3FS3 AH-ROBUST-MONO-DE-CLEF2006 de TDN AUTOMATIC no
hummingbird Canada humDE06Rtde AH-ROBUST-MONO-DE-CLEF2006 de TD AUTOMATIC no
hummingbird Canada humDE06Rtd AH-ROBUST-MONO-DE-CLEF2006 de TD AUTOMATIC no
colesir Spain CoLesIRenTst AH-ROBUST-MONO-EN-CLEF2006 en TD AUTOMATIC no
daedalus Spain enFSenR2S AH-ROBUST-MONO-EN-CLEF2006 en TD AUTOMATIC no
daedalus Spain enFSenR3S AH-ROBUST-MONO-EN-CLEF2006 en TDN AUTOMATIC no
daedalus Spain enFSenR5S AH-ROBUST-MONO-EN-CLEF2006 en TDN AUTOMATIC no
daedalus Spain enyenRFS3456 AH-ROBUST-MONO-EN-CLEF2006 en TDN AUTOMATIC no
dcu Ireland dcunarreng1005 AH-ROBUST-MONO-EN-CLEF2006 en TDN AUTOMATIC no
dcu Ireland dcudesceng12075 AH-ROBUST-MONO-EN-CLEF2006 en TD AUTOMATIC no
hildesheim Germany uhienmo2 AH-ROBUST-MONO-EN-CLEF2006 en TD AUTOMATIC no
hildesheim Germany uhienmo1 AH-ROBUST-MONO-EN-CLEF2006 en TD AUTOMATIC no
hummingbird Canada humEN06Rtd AH-ROBUST-MONO-EN-CLEF2006 en TD AUTOMATIC no
hummingbird Canada humEN06Rtde AH-ROBUST-MONO-EN-CLEF2006 en TD AUTOMATIC no
reina Spain reinaENttest AH-ROBUST-MONO-EN-CLEF2006 en T AUTOMATIC no
reina Spain reinaENtdtest AH-ROBUST-MONO-EN-CLEF2006 en TD AUTOMATIC no
colesir Spain CoLesIResTst AH-ROBUST-MONO-ES-CLEF2006 es TD AUTOMATIC no
daedalus Spain esx011esRFW3FS3 AH-ROBUST-MONO-ES-CLEF2006 es TDN AUTOMATIC no
daedalus Spain esFSesR2S AH-ROBUST-MONO-ES-CLEF2006 es TD AUTOMATIC no
daedalus Spain esx021esRFW3FS3 AH-ROBUST-MONO-ES-CLEF2006 es TDN AUTOMATIC no
daedalus Spain esFSesR3S AH-ROBUST-MONO-ES-CLEF2006 es TDN AUTOMATIC no
dcu Ireland dcu1500narrsp1005.qy1 AH-ROBUST-MONO-ES-CLEF2006 es TDN AUTOMATIC no
dcu Ireland dcudescsp12075 AH-ROBUST-MONO-ES-CLEF2006 es TD AUTOMATIC no
hummingbird Canada humES06Rtde AH-ROBUST-MONO-ES-CLEF2006 es TD AUTOMATIC no
hummingbird Canada humES06Rtd AH-ROBUST-MONO-ES-CLEF2006 es TD AUTOMATIC no
reina Spain reinaEStdtest AH-ROBUST-MONO-ES-CLEF2006 es TD AUTOMATIC no
reina Spain reinaESttest AH-ROBUST-MONO-ES-CLEF2006 es T AUTOMATIC no
colesir Spain CoLesIRfrTst AH-ROBUST-MONO-FR-CLEF2006 fr TD AUTOMATIC no
daedalus Spain frFSfrR2S AH-ROBUST-MONO-FR-CLEF2006 fr TD AUTOMATIC no
daedalus Spain fryfrRFS3456 AH-ROBUST-MONO-FR-CLEF2006 fr TDN AUTOMATIC no
daedalus Spain frx011frRFW3FS3 AH-ROBUST-MONO-FR-CLEF2006 fr TDN AUTOMATIC no
daedalus Spain frFSfrR3S AH-ROBUST-MONO-FR-CLEF2006 fr TDN AUTOMATIC no
dcu Ireland dcudescfr12075 AH-ROBUST-MONO-FR-CLEF2006 fr TD AUTOMATIC no
dcu Ireland dcudescfr12075 AH-ROBUST-MONO-FR-CLEF2006 fr TD AUTOMATIC no
dcu Ireland dcunarrfr1005 AH-ROBUST-MONO-FR-CLEF2006 fr TDN AUTOMATIC no
hildesheim Germany uhifrmo2 AH-ROBUST-MONO-FR-CLEF2006 fr TD AUTOMATIC no
hildesheim Germany uhifrmo1 AH-ROBUST-MONO-FR-CLEF2006 fr TD AUTOMATIC no
hildesheim Germany uhifrmo1 AH-ROBUST-MONO-FR-CLEF2006 fr TD AUTOMATIC no
hummingbird Canada humFR06Rtde AH-ROBUST-MONO-FR-CLEF2006 fr TD AUTOMATIC no
hummingbird Canada humFR06Rtd AH-ROBUST-MONO-FR-CLEF2006 fr TD AUTOMATIC no
reina Spain reinaFRtdtest AH-ROBUST-MONO-FR-CLEF2006 fr TD AUTOMATIC no
reina Spain reinaFRttest AH-ROBUST-MONO-FR-CLEF2006 fr T AUTOMATIC no
unine Switzerland UniNEfrr1 AH-ROBUST-MONO-FR-CLEF2006 fr TD AUTOMATIC no
unine Switzerland UniNEfrr3 AH-ROBUST-MONO-FR-CLEF2006 fr TD AUTOMATIC no
unine Switzerland UniNEfrr2 AH-ROBUST-MONO-FR-CLEF2006 fr T AUTOMATIC no
colesir Spain CoLesIRitTst AH-ROBUST-MONO-IT-CLEF2006 it TD AUTOMATIC no
daedalus Spain itFSitR4S AH-ROBUST-MONO-IT-CLEF2006 it TDN AUTOMATIC no
daedalus Spain ityitRFS3456 AH-ROBUST-MONO-IT-CLEF2006 it TDN AUTOMATIC no
daedalus Spain itFSitR2S AH-ROBUST-MONO-IT-CLEF2006 it TD AUTOMATIC no
daedalus Spain itFSitR6S AH-ROBUST-MONO-IT-CLEF2006 it TDN AUTOMATIC no
dcu Ireland dcu1500narrit1005.qy1 AH-ROBUST-MONO-IT-CLEF2006 it TDN AUTOMATIC no
dcu Ireland dcudescit1005 AH-ROBUST-MONO-IT-CLEF2006 it TD AUTOMATIC no
9
Participant Country Experiment ID Task Topic Topic Query Pool
Lang. Fields Construction
hummingbird Canada humIT06Rtd AH-ROBUST-MONO-IT-CLEF2006 it TD AUTOMATIC no
hummingbird Canada humIT06Rtde AH-ROBUST-MONO-IT-CLEF2006 it TD AUTOMATIC no
reina Spain reinaITttest AH-ROBUST-MONO-IT-CLEF2006 it T AUTOMATIC no
reina Spain reinaITtdtest AH-ROBUST-MONO-IT-CLEF2006 it TD AUTOMATIC no
colesir Spain CoLesIRnlTst AH-ROBUST-MONO-NL-CLEF2006 nl TD AUTOMATIC no
daedalus Spain nlFSnlR4S AH-ROBUST-MONO-NL-CLEF2006 nl TDN AUTOMATIC no
daedalus Spain nlynlRFS3456 AH-ROBUST-MONO-NL-CLEF2006 nl TDN AUTOMATIC no
daedalus Spain nlx011nlRFW4FS4 AH-ROBUST-MONO-NL-CLEF2006 nl TDN AUTOMATIC no
daedalus Spain nlFSnlR2S AH-ROBUST-MONO-NL-CLEF2006 nl TD AUTOMATIC no
hummingbird Canada humNL06Rtde AH-ROBUST-MONO-NL-CLEF2006 nl TD AUTOMATIC no
hummingbird Canada humNL06Rtd AH-ROBUST-MONO-NL-CLEF2006 nl TD AUTOMATIC no
colesir Spain CoLesIRmultTst AH-ROBUST-MULTI-CLEF2006 en TD AUTOMATIC no
daedalus Spain ml4XRSFSen4S AH-ROBUST-MULTI-CLEF2006 en TDN AUTOMATIC no
daedalus Spain ml5XRSFSen4S AH-ROBUST-MULTI-CLEF2006 en TDN AUTOMATIC no
daedalus Spain ml6XRSFSen4S AH-ROBUST-MULTI-CLEF2006 en TDN AUTOMATIC no
daedalus Spain mlRSFSen2S AH-ROBUST-MULTI-CLEF2006 en TD AUTOMATIC no
jaen Spain ujamlblr AH-ROBUST-MULTI-CLEF2006 en TD AUTOMATIC no
jaen Spain ujamlrsv2 AH-ROBUST-MULTI-CLEF2006 en TD AUTOMATIC no
jaen Spain ujamllr AH-ROBUST-MULTI-CLEF2006 en TD AUTOMATIC no
reina Spain reinaES2mtdtest AH-ROBUST-MULTI-CLEF2006 es TD AUTOMATIC no
reina Spain reinaES2mttest AH-ROBUST-MULTI-CLEF2006 es T AUTOMATIC no
colesir Spain CoLesIRendeTst AH-ROBUST-BILI-X2DE-CLEF2006 en TD AUTOMATIC no
daedalus Spain deFSdeRSen2S AH-ROBUST-BILI-X2DE-CLEF2006 en TD AUTOMATIC no
daedalus Spain deFSdeRSen4S AH-ROBUST-BILI-X2DE-CLEF2006 en TDN AUTOMATIC no
daedalus Spain deFSdeRSen3S AH-ROBUST-BILI-X2DE-CLEF2006 en TDN AUTOMATIC no
daedalus Spain deydeRSenFS3456 AH-ROBUST-BILI-X2DE-CLEF2006 en TDN AUTOMATIC no
daedalus Spain esx011esRLitFW3FS3 AH-ROBUST-BILI-X2ES-CLEF2006 it TDN AUTOMATIC no
daedalus Spain esFSesRLit2S AH-ROBUST-BILI-X2ES-CLEF2006 it TD AUTOMATIC no
daedalus Spain esFSesRLit3S AH-ROBUST-BILI-X2ES-CLEF2006 it TDN AUTOMATIC no
daedalus Spain esx021esRLitFW3FS3 AH-ROBUST-BILI-X2ES-CLEF2006 it TDN AUTOMATIC no
dcu Ireland dcuitqynarrsp1005 AH-ROBUST-BILI-X2ES-CLEF2006 it TDN AUTOMATIC no
dcu Ireland dcuitqydescsp12075 AH-ROBUST-BILI-X2ES-CLEF2006 it TD AUTOMATIC no
reina Spain reinaIT2EStdtest AH-ROBUST-BILI-X2ES-CLEF2006 it TD AUTOMATIC no
reina Spain reinaIT2ESttest AH-ROBUST-BILI-X2ES-CLEF2006 it T AUTOMATIC no
daedalus Spain nlFSnlRLfr2S AH-ROBUST-BILI-X2NL-CLEF2006 fr TD AUTOMATIC no
daedalus Spain nlFSnlRLfr6S AH-ROBUST-BILI-X2NL-CLEF2006 fr TDN AUTOMATIC no
daedalus Spain nlx021nlRLfrFW4FS6 AH-ROBUST-BILI-X2NL-CLEF2006 fr TDN AUTOMATIC no
daedalus Spain nlx011nlRLfrFW4FS6 AH-ROBUST-BILI-X2NL-CLEF2006 fr TDN AUTOMATIC no
10
Track Overview Results and Graphs
11
12
AH-ROBUST-CLEF2006 Track Overview Results and Graphs AH-ROBUST-MONO-DE-CLEF2006
Ad−Hoc Robust Monolingual German track Top 5 Participants − Interpolated Recall vs Average Precision
100%
hummingbird [humDE06Rtde; MAP 48.30%; Not Pooled]
colesir [CoLesIRdeTst; MAP 37.21%; Not Pooled]
90%
daedalus [deFSdeR2S; MAP 34.06%; Not Pooled]
80%
70%
Average Precision
60%
50%
40%
30%
20%
10%
0%
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
Interpolated Recall
Ad−Hoc Robust Monolingual German track Top 5 Participants − Comparison to Median Mean Average Precision by Topic (Topics 041to 085)
1
hummingbird [humDE06Rtde; MAP 48.30%; Not Pooled]
colesir [CoLesIRdeTst; MAP 37.21%; Not Pooled]
0.5 daedalus [deFSdeR2S; MAP 34.06%; Not Pooled]
Difference
0
−0.5
−1
041 042 043 044 045 046 047 048 049 060 061 062 063 064 065 066 067 068 069 080 081 082 083 084 085
Topic Identifier
Ad−Hoc Robust Monolingual German track Top 5 Participants − Comparison to Median Mean Average Precision by Topic (Topics 086to 130)
1
hummingbird [humDE06Rtde; MAP 48.30%; Not Pooled]
colesir [CoLesIRdeTst; MAP 37.21%; Not Pooled]
0.5 daedalus [deFSdeR2S; MAP 34.06%; Not Pooled]
Difference
0
−0.5
−1
086 087 088 089 090 091 092 093 094 095 096 097 098 099 110 111 112 113 114 115 116 117 118 119 130
Topic Identifier
Ad−Hoc Robust Monolingual German track Top 5 Participants − Comparison to Median Mean Average Precision by Topic (Topics 131to 165)
1
hummingbird [humDE06Rtde; MAP 48.30%; Not Pooled]
colesir [CoLesIRdeTst; MAP 37.21%; Not Pooled]
0.5 daedalus [deFSdeR2S; MAP 34.06%; Not Pooled]
Difference
0
−0.5
−1
131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 160 161 162 163 164 165
Topic Identifier
Ad−Hoc Robust Monolingual German track Top 5 Participants − Comparison to Median Mean Average Precision by Topic (Topics 166to 200)
1
hummingbird [humDE06Rtde; MAP 48.30%; Not Pooled]
colesir [CoLesIRdeTst; MAP 37.21%; Not Pooled]
0.5 daedalus [deFSdeR2S; MAP 34.06%; Not Pooled]
Difference
0
−0.5
−1
166 167 168 169 170 171 172 173 174 175 176 177 178 179 190 191 192 193 194 195 196 197 198 199 200
Topic Identifier
13
AH-ROBUST-CLEF2006 Track Overview Results and Graphs AH-ROBUST-MONO-DE-CLEF2006
Ad−Hoc Robust Monolingual German track − Box Plot of the Topics
humDE06Rtde [MAP 48.30%; Not Pooled]
humDE06Rtd [MAP 42.45%; Not Pooled]
dex011deRFW3FS3 [MAP 38.63%; Not Pooled]
Experiments
dex021deRFW3FS3 [MAP 38.31%; Not Pooled]
deFSdeR3S [MAP 38.03%; Not Pooled]
CoLesIRdeTst [MAP 37.21%; Not Pooled]
deFSdeR2S [MAP 34.06%; Not Pooled]
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
Mean Average Precision
14
AH-ROBUST-CLEF2006 Track Overview Results and Graphs AH-ROBUST-MONO-DE-CLEF2006
Ad−Hoc Robust Monolingual German track − Tukey T test with "top group" highlighted
humDE06Rtde
humDE06Rtd
dex011deRFW3FS3
Experiments
dex021deRFW3FS3
deFSdeR3S
CoLesIRdeTst
deFSdeR2S
0.55 0.6 0.65 0.7 0.75 0.8
arcsin(sqrt(Mean average precision))
15
AH-ROBUST-CLEF2006 Track Overview Results and Graphs AH-ROBUST-MONO-DE-CLEF2006
Ad−Hoc Robust Monolingual German track Top 5 Participants − Retrieved documents vs Precision
100%
hummingbird [humDE06Rtde; R−Prec 46.34%; Not Pooled]
colesir [CoLesIRdeTst; R−Prec 36.49%; Not Pooled]
90%
daedalus [deFSdeR2S; R−Prec 34.80%; Not Pooled]
80%
70%
60%
R−Precision
50%
40%
30%
20%
10%
0%
5 10 15 20 30 100 200 500 1000
Retrieved Documents (logarithmic scale)
Ad−Hoc Robust Monolingual German track Top 5 Participants − Comparison to Median R−Precision by Topic (Topics 041 to 085)
1
hummingbird [humDE06Rtde; R−Prec 46.34%; Not Pooled]
colesir [CoLesIRdeTst; R−Prec 36.49%; Not Pooled]
0.5 daedalus [deFSdeR2S; R−Prec 34.80%; Not Pooled]
Difference
0
−0.5
−1
041 042 043 044 045 046 047 048 049 060 061 062 063 064 065 066 067 068 069 080 081 082 083 084 085
Topic Identifier
Ad−Hoc Robust Monolingual German track Top 5 Participants − Comparison to Median R−Precision by Topic (Topics 086 to 130)
1
hummingbird [humDE06Rtde; R−Prec 46.34%; Not Pooled]
colesir [CoLesIRdeTst; R−Prec 36.49%; Not Pooled]
0.5 daedalus [deFSdeR2S; R−Prec 34.80%; Not Pooled]
Difference
0
−0.5
−1
086 087 088 089 090 091 092 093 094 095 096 097 098 099 110 111 112 113 114 115 116 117 118 119 130
Topic Identifier
Ad−Hoc Robust Monolingual German track Top 5 Participants − Comparison to Median R−Precision by Topic (Topics 131 to 165)
1
hummingbird [humDE06Rtde; R−Prec 46.34%; Not Pooled]
colesir [CoLesIRdeTst; R−Prec 36.49%; Not Pooled]
0.5 daedalus [deFSdeR2S; R−Prec 34.80%; Not Pooled]
Difference
0
−0.5
−1
131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 160 161 162 163 164 165
Topic Identifier
Ad−Hoc Robust Monolingual German track Top 5 Participants − Comparison to Median R−Precision by Topic (Topics 166 to 200)
1
hummingbird [humDE06Rtde; R−Prec 46.34%; Not Pooled]
colesir [CoLesIRdeTst; R−Prec 36.49%; Not Pooled]
0.5 daedalus [deFSdeR2S; R−Prec 34.80%; Not Pooled]
Difference
0
−0.5
−1
166 167 168 169 170 171 172 173 174 175 176 177 178 179 190 191 192 193 194 195 196 197 198 199 200
Topic Identifier
16
AH-ROBUST-CLEF2006 Track Overview Results and Graphs AH-ROBUST-MONO-DE-CLEF2006
Ad−Hoc Robust Monolingual German track − Box Plot of the Topics
humDE06Rtde [R−Prec 46.34%; Not Pooled]
humDE06Rtd [R−Prec 41.68%; Not Pooled]
dex011deRFW3FS3 [R−Prec 38.76%; Not Pooled]
Experiments
dex021deRFW3FS3 [R−Prec 38.15%; Not Pooled]
deFSdeR3S [R−Prec 37.76%; Not Pooled]
CoLesIRdeTst [R−Prec 36.49%; Not Pooled]
deFSdeR2S [R−Prec 34.80%; Not Pooled]
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
R−Precision
17
AH-ROBUST-CLEF2006 Track Overview Results and Graphs AH-ROBUST-MONO-DE-CLEF2006
Ad−Hoc Robust Monolingual German track − Tukey T test with "top group" highlighted
humDE06Rtde
humDE06Rtd
dex011deRFW3FS3
Experiments
dex021deRFW3FS3
deFSdeR3S
CoLesIRdeTst
deFSdeR2S
0.5 0.55 0.6 0.65 0.7 0.75 0.8 0.85 0.9
arcsin(sqrt(R Precision))
18
AH-ROBUST-CLEF2006 Track Overview Results and Graphs AH-ROBUST-MONO-DE-CLEF2006
Average Precision R-Precision
Topic Minimum 1st Q. Median 3rd Q. Maximum Mean Std Minimum 1st Q. Median 3rd Q. Maximum Mean Std
041 0.6119 0.6119 0.6392 0.7928 0.8497 0.6971 0.1026 0.5000 0.5000 0.5714 0.7321 0.7500 0.6071 0.1166
042 0.7170 0.7170 0.7248 0.7355 0.7423 0.7262 0.0105 0.6792 0.6804 0.6934 0.6934 0.6934 0.6880 0.0069
043 0.0239 0.0368 0.0368 0.3310 0.4167 0.1488 0.1836 0.0000 0.0000 0.0000 0.0000 0.5000 0.0714 0.1890
044 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000
045 0.5134 0.5497 0.6660 0.6756 0.6756 0.6179 0.0716 0.5377 0.5684 0.6226 0.6415 0.6415 0.6038 0.0432
046 0.0492 0.0520 0.0866 0.1383 0.1383 0.0940 0.0432 0.0000 0.0000 0.0000 0.2000 0.2000 0.0857 0.1069
047 0.0674 0.1205 0.1277 0.1342 0.1589 0.1234 0.0278 0.0417 0.1094 0.1250 0.1875 0.1875 0.1369 0.0549
048 0.0805 0.1093 0.1093 0.1895 0.2142 0.1374 0.0529 0.1455 0.1818 0.1818 0.2318 0.2909 0.2052 0.0478
049 0.1405 0.1567 0.1936 0.2371 0.2912 0.2014 0.0534 0.1111 0.1111 0.1667 0.2083 0.2222 0.1587 0.0500
060 0.1931 0.1989 0.2204 0.3034 0.3471 0.2537 0.0625 0.2807 0.2807 0.2982 0.4254 0.5439 0.3584 0.1027
061 0.0385 0.0385 0.0455 0.3483 0.4612 0.1760 0.1841 0.0233 0.0233 0.0233 0.4477 0.5814 0.2126 0.2477
062 0.4278 0.4718 0.5187 0.5522 0.5621 0.5068 0.0495 0.3333 0.4444 0.4444 0.6667 0.6667 0.5238 0.1393
063 0.4154 0.4729 0.5422 0.6704 0.7915 0.5709 0.1355 0.3421 0.4934 0.5526 0.5921 0.6842 0.5376 0.1073
064 1.0000 1.0000 1.0000 1.0000 1.0000 1.0000 0.0000 1.0000 1.0000 1.0000 1.0000 1.0000 1.0000 0.0000
065 0.0553 0.0719 0.1086 0.1193 0.1327 0.0990 0.0292 0.1538 0.1692 0.2000 0.2154 0.2154 0.1912 0.0264
066 0.7684 0.7736 0.7736 0.8336 0.8415 0.7984 0.0333 0.6667 0.6667 0.6863 0.7010 0.7059 0.6835 0.0176
067 0.1578 0.1630 0.2006 0.3604 0.6823 0.2858 0.1952 0.1800 0.2600 0.2600 0.3850 0.6600 0.3314 0.1616
068 0.1586 0.2085 0.2872 0.3144 0.3297 0.2605 0.0660 0.2045 0.3239 0.3864 0.3864 0.4091 0.3474 0.0703
069 0.0754 0.0754 0.0828 0.1257 0.1346 0.0983 0.0270 0.0000 0.0000 0.1429 0.2500 0.2857 0.1224 0.1285
080 0.0707 0.4341 0.4834 0.5020 0.5077 0.4213 0.1574 0.1167 0.4896 0.5500 0.5667 0.5667 0.4821 0.1645
081 0.3859 0.5184 0.5280 0.6194 0.6450 0.5481 0.0893 0.4200 0.4800 0.5800 0.5800 0.5800 0.5286 0.0672
082 0.3365 0.5521 0.5521 0.6782 0.8460 0.5946 0.1562 0.2963 0.5926 0.5926 0.7037 0.7778 0.6085 0.1555
083 0.1620 0.1784 0.1825 0.2081 0.2658 0.1955 0.0350 0.1111 0.1111 0.1111 0.3056 0.3333 0.1905 0.1057
084 0.0022 0.0022 0.1000 0.1661 0.2167 0.0874 0.0903 0.0000 0.0000 0.2000 0.2000 0.2000 0.1143 0.1069
085 0.0725 0.1645 0.2417 0.3969 0.3969 0.2635 0.1342 0.0405 0.2230 0.2703 0.3919 0.3919 0.2780 0.1293
086 0.3530 0.4205 0.4457 0.4811 0.4824 0.4423 0.0470 0.4414 0.4685 0.4685 0.4842 0.4865 0.4710 0.0154
087 0.2292 0.2872 0.3249 0.3451 0.3708 0.3154 0.0480 0.2353 0.3088 0.3529 0.3529 0.3529 0.3277 0.0463
088 0.5302 0.5869 0.6077 0.6110 0.6410 0.5981 0.0348 0.5517 0.5517 0.5517 0.5517 0.5690 0.5542 0.0065
089 0.5092 0.5134 0.5209 0.6142 0.6554 0.5552 0.0643 0.4966 0.5310 0.5310 0.6431 0.6759 0.5734 0.0705
090 0.1004 0.1147 0.1569 0.1786 0.2216 0.1539 0.0427 0.1250 0.1406 0.1875 0.1875 0.3125 0.1875 0.0625
091 0.1020 0.1133 0.1237 0.1919 0.3210 0.1597 0.0799 0.1562 0.1562 0.1562 0.1797 0.3750 0.1920 0.0815
092 0.1654 0.1729 0.1999 0.1999 0.2349 0.1935 0.0235 0.1765 0.1765 0.2353 0.2353 0.2941 0.2185 0.0445
093 0.7219 0.8344 0.8672 0.8949 0.8949 0.8493 0.0621 0.7222 0.7500 0.7500 0.7986 0.8056 0.7659 0.0315
094 0.7555 0.7648 0.7799 0.7854 0.7985 0.7770 0.0148 0.7188 0.7500 0.7500 0.8125 0.8125 0.7723 0.0392
095 0.0171 0.0421 0.1214 0.1214 0.1294 0.0876 0.0473 0.0455 0.0773 0.1455 0.1659 0.2455 0.1364 0.0688
096 0.1941 0.2067 0.2067 0.2330 0.3245 0.2279 0.0447 0.2222 0.2222 0.2593 0.2963 0.4444 0.2804 0.0796
097 0.3134 0.3628 0.5065 0.6579 0.6579 0.5164 0.1538 0.3333 0.3833 0.5333 0.5333 0.6000 0.4857 0.1069
098 0.3035 0.3688 0.3948 0.4263 0.4368 0.3881 0.0459 0.4400 0.4800 0.4800 0.5200 0.5200 0.4914 0.0302
099 0.1350 0.2300 0.2655 0.2751 0.5475 0.2819 0.1271 0.2584 0.2978 0.3820 0.3820 0.5393 0.3676 0.0912
110 0.4242 0.4564 0.4564 0.5131 0.5639 0.4822 0.0476 0.5417 0.5417 0.5417 0.5833 0.6250 0.5655 0.0328
111 0.0042 0.0077 0.0077 0.0499 0.1671 0.0390 0.0598 0.0000 0.0000 0.0000 0.0938 0.1250 0.0357 0.0610
112 0.7102 0.7602 0.7602 0.8229 0.8729 0.7845 0.0540 0.6429 0.7143 0.7143 0.7143 0.8571 0.7245 0.0643
113 0.0336 0.0337 0.0881 0.1278 0.1538 0.0821 0.0511 0.0800 0.0800 0.1200 0.1600 0.2400 0.1314 0.0598
114 0.0946 0.1393 0.2368 0.2521 0.2607 0.2006 0.0673 0.1071 0.1964 0.2500 0.2768 0.2857 0.2296 0.0647
115 0.1332 0.1434 0.1639 0.1916 0.2682 0.1765 0.0460 0.1667 0.2500 0.2500 0.2812 0.3750 0.2619 0.0623
116 0.4683 0.4848 0.5688 0.5783 0.5903 0.5364 0.0530 0.4516 0.4758 0.5484 0.5484 0.5806 0.5253 0.0517
117 0.1785 0.1839 0.1918 0.2237 0.4553 0.2314 0.1005 0.2222 0.2333 0.3111 0.3111 0.5333 0.3111 0.1058
118 0.0632 0.1875 0.2448 0.2646 0.2683 0.2114 0.0733 0.0000 0.2222 0.2222 0.4167 0.4444 0.2698 0.1553
119 0.6202 0.6206 0.6297 0.7161 0.7250 0.6619 0.0498 0.6667 0.6667 0.6771 0.7318 0.7396 0.6949 0.0339
130 0.5621 0.6112 0.7276 0.7843 0.7843 0.6952 0.0970 0.6000 0.6000 0.6667 0.6667 0.7333 0.6476 0.0504
19
AH-ROBUST-CLEF2006 Track Overview Results and Graphs AH-ROBUST-MONO-DE-CLEF2006
Average Precision R-Precision
Topic Minimum 1st Q. Median 3rd Q. Maximum Mean Std Minimum 1st Q. Median 3rd Q. Maximum Mean Std
131 0.0047 0.0062 0.0093 0.0218 0.0283 0.0132 0.0098 0.0152 0.0303 0.0455 0.0606 0.0606 0.0433 0.0184
132 0.7359 0.7359 0.7907 0.7975 0.8465 0.7766 0.0425 0.6071 0.6071 0.7143 0.7411 0.7857 0.6837 0.0756
133 0.5730 0.5863 0.6338 0.6338 0.6563 0.6180 0.0318 0.5909 0.6227 0.6455 0.6455 0.6636 0.6351 0.0237
134 0.8452 0.8607 0.8839 0.9235 0.9571 0.8920 0.0411 0.7143 0.7143 0.7143 0.8214 0.8571 0.7551 0.0697
135 0.0630 0.1126 0.1161 0.3416 0.6049 0.2404 0.1959 0.1609 0.2184 0.2184 0.4167 0.5862 0.3136 0.1537
136 0.4581 0.4880 0.5118 0.5668 0.5959 0.5208 0.0509 0.4286 0.4286 0.4286 0.5357 0.5714 0.4694 0.0697
137 0.0000 0.0000 0.0000 0.0716 0.1429 0.0393 0.0557 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000
138 0.4094 0.4668 0.6590 0.7109 0.7197 0.5944 0.1320 0.3548 0.4113 0.6452 0.7419 0.7419 0.5853 0.1731
139 0.0106 0.0106 0.0153 0.1305 0.1610 0.0625 0.0677 0.0357 0.0357 0.0357 0.1071 0.1071 0.0663 0.0382
140 0.0799 0.0832 0.1369 0.3537 0.4962 0.2181 0.1676 0.1852 0.1852 0.2716 0.4198 0.5679 0.3157 0.1498
141 0.7500 0.7500 0.7500 0.8520 0.9705 0.8097 0.0849 0.6250 0.7500 0.7500 0.7500 0.8750 0.7500 0.0722
142 0.3251 0.3398 0.3398 0.8916 0.8984 0.5750 0.2980 0.4000 0.4000 0.4000 0.8115 0.8462 0.5802 0.2252
143 0.5072 0.5608 0.5768 0.6218 0.6941 0.5931 0.0592 0.5873 0.5873 0.5873 0.5992 0.6667 0.6009 0.0296
144 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000
145 0.0833 0.1812 0.3281 0.4349 0.7874 0.3472 0.2322 0.1000 0.2500 0.4000 0.4750 0.6000 0.3714 0.1704
146 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000
147 0.0297 0.0308 0.0319 0.2085 0.3384 0.1106 0.1314 0.0345 0.0345 0.0690 0.2241 0.3793 0.1281 0.1404
148 0.0941 0.1141 0.1345 0.1416 0.1666 0.1303 0.0235 0.0833 0.1042 0.2500 0.2500 0.2500 0.1905 0.0793
149 0.5387 0.8176 0.8468 0.8648 0.9338 0.8131 0.1269 0.5000 0.7708 0.8333 0.8333 0.9167 0.7857 0.1349
160 0.1000 0.1571 0.2500 0.2500 0.2500 0.2061 0.0619 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000
161 0.0068 0.0160 0.0714 0.8333 1.0000 0.3491 0.4589 0.0000 0.0000 0.0000 0.7500 1.0000 0.2857 0.4880
162 0.3220 0.5773 0.7497 0.8033 0.8921 0.6863 0.1971 0.4167 0.5833 0.7083 0.7708 0.7917 0.6667 0.1382
163 0.1284 0.5627 0.5864 0.6165 0.6920 0.5373 0.1856 0.1702 0.5106 0.5745 0.5904 0.6596 0.5198 0.1620
164 0.0706 0.0937 0.0973 0.2839 0.3965 0.1745 0.1327 0.1389 0.1389 0.1667 0.3333 0.4444 0.2262 0.1317
165 0.4091 0.4091 0.4483 0.4633 0.6500 0.4645 0.0855 0.2500 0.2500 0.2500 0.2500 0.5000 0.2857 0.0945
166 0.0130 0.0170 0.0170 0.0292 0.0476 0.0238 0.0121 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000
167 0.5738 0.6266 0.6750 0.6821 0.8042 0.6711 0.0720 0.5000 0.5000 0.5000 0.5000 0.7500 0.5357 0.0945
168 0.0190 0.0238 0.0238 0.0654 0.1512 0.0518 0.0477 0.0000 0.0588 0.0588 0.1029 0.2353 0.0840 0.0748
169 0.0038 0.0628 0.1389 0.1632 0.1632 0.1085 0.0644 0.0000 0.1000 0.2000 0.2000 0.2000 0.1429 0.0787
170 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000
171 0.0271 0.0596 0.0596 0.3628 0.5888 0.2135 0.2180 0.0000 0.0000 0.0000 0.3438 0.5000 0.1607 0.2130
172 0.0000 0.0000 0.0000 0.8333 1.0000 0.3333 0.4714 0.0000 0.0000 0.0000 0.7500 1.0000 0.2857 0.4880
173 0.9184 0.9672 0.9733 0.9790 0.9809 0.9665 0.0219 0.8000 0.8250 0.9000 0.9000 0.9000 0.8714 0.0488
174 0.2760 0.6322 0.6322 0.8692 0.8936 0.6846 0.2172 0.3056 0.6389 0.6389 0.7986 0.8333 0.6627 0.1789
175 0.5282 0.5282 0.6036 0.8001 0.8742 0.6610 0.1492 0.5455 0.5455 0.5455 0.6364 0.8182 0.6104 0.1012
176 0.9703 0.9753 0.9753 0.9820 0.9877 0.9777 0.0060 0.8919 0.9054 0.9054 0.9189 0.9324 0.9112 0.0132
177 0.0000 0.0000 0.0007 0.2074 0.2881 0.0905 0.1258 0.0000 0.0000 0.0000 0.2500 0.2857 0.1020 0.1359
178 0.1027 0.1027 0.2404 0.4007 0.4377 0.2513 0.1523 0.2381 0.2381 0.3810 0.4286 0.4286 0.3401 0.0969
179 0.5368 0.6265 0.6265 0.6762 0.8069 0.6501 0.0824 0.5306 0.5510 0.5510 0.6378 0.7551 0.5977 0.0805
190 0.0708 0.1284 0.5493 0.5726 0.5726 0.3739 0.2421 0.1754 0.2281 0.5614 0.5789 0.5789 0.4185 0.1955
191 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000
192 0.3487 0.4851 0.4851 0.7007 0.8497 0.5676 0.1724 0.4615 0.5000 0.5000 0.6154 0.7308 0.5604 0.0964
193 0.5460 0.6167 0.6296 0.6346 0.6458 0.6185 0.0335 0.5294 0.5368 0.5588 0.5809 0.6176 0.5630 0.0314
194 0.8333 0.8333 0.8333 1.0000 1.0000 0.9048 0.0891 0.5000 0.5000 0.5000 1.0000 1.0000 0.7143 0.2673
195 0.2174 0.2351 0.2467 0.4384 0.5345 0.3322 0.1288 0.2414 0.2414 0.2759 0.4569 0.5862 0.3498 0.1385
196 0.8733 0.9479 0.9583 0.9583 0.9762 0.9468 0.0337 0.8333 0.8333 0.8333 0.8333 0.8333 0.8333 0.0000
197 0.5766 0.5793 0.5856 0.6534 0.7321 0.6228 0.0586 0.5820 0.5820 0.5820 0.6373 0.6885 0.6101 0.0423
198 0.1028 0.1091 0.1195 0.1378 0.1429 0.1228 0.0157 0.0833 0.1667 0.1667 0.1667 0.1667 0.1548 0.0315
199 0.7229 0.7400 0.7831 0.7831 0.8227 0.7713 0.0349 0.7083 0.7344 0.7500 0.7500 0.7917 0.7470 0.0253
200 0.1002 0.2354 0.2405 0.3256 0.6916 0.3003 0.1874 0.1061 0.3068 0.3182 0.3864 0.5909 0.3377 0.1445
ALL 0.3406 0.3742 0.3831 0.4149 0.4830 0.3957 0.0457 0.3480 0.3680 0.3815 0.4095 0.4634 0.3914 0.0381
20
21
AH-ROBUST-CLEF2006 Track Overview Results and Graphs AH-ROBUST-MONO-EN-CLEF2006
Ad−Hoc Robust Monolingual English track Top 5 Participants − Interpolated Recall vs Average Precision
100%
hummingbird [humEN06Rtde; MAP 47.63%; Not Pooled]
reina [reinaENtdtest; MAP 43.66%; Not Pooled]
90% dcu [dcudesceng12075; MAP 43.48%; Not Pooled]
daedalus [enFSenR2S; MAP 39.69%; Not Pooled]
colesir [CoLesIRenTst; MAP 37.64%; Not Pooled]
80%
70%
Average Precision
60%
50%
40%
30%
20%
10%
0%
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
Interpolated Recall
Ad−Hoc Robust Monolingual English track Top 5 Participants − Comparison to Median Mean Average Precision by Topic (Topics 041to 085)
1
hummingbird [humEN06Rtde; MAP 47.63%; Not Pooled]
reina [reinaENtdtest; MAP 43.66%; Not Pooled]
0.5 dcu [dcudesceng12075; MAP 43.48%; Not Pooled]
Difference
daedalus [enFSenR2S; MAP 39.69%; Not Pooled]
0 colesir [CoLesIRenTst; MAP 37.64%; Not Pooled]
−0.5
−1
041 042 043 044 045 046 047 048 049 060 061 062 063 064 065 066 067 068 069 080 081 082 083 084 085
Topic Identifier
Ad−Hoc Robust Monolingual English track Top 5 Participants − Comparison to Median Mean Average Precision by Topic (Topics 086to 130)
1
hummingbird [humEN06Rtde; MAP 47.63%; Not Pooled]
reina [reinaENtdtest; MAP 43.66%; Not Pooled]
0.5 dcu [dcudesceng12075; MAP 43.48%; Not Pooled]
Difference
daedalus [enFSenR2S; MAP 39.69%; Not Pooled]
0 colesir [CoLesIRenTst; MAP 37.64%; Not Pooled]
−0.5
−1
086 087 088 089 090 091 092 093 094 095 096 097 098 099 110 111 112 113 114 115 116 117 118 119 130
Topic Identifier
Ad−Hoc Robust Monolingual English track Top 5 Participants − Comparison to Median Mean Average Precision by Topic (Topics 131to 165)
1
hummingbird [humEN06Rtde; MAP 47.63%; Not Pooled]
reina [reinaENtdtest; MAP 43.66%; Not Pooled]
0.5 dcu [dcudesceng12075; MAP 43.48%; Not Pooled]
Difference
daedalus [enFSenR2S; MAP 39.69%; Not Pooled]
0 colesir [CoLesIRenTst; MAP 37.64%; Not Pooled]
−0.5
−1
131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 160 161 162 163 164 165
Topic Identifier
Ad−Hoc Robust Monolingual English track Top 5 Participants − Comparison to Median Mean Average Precision by Topic (Topics 166to 200)
1
hummingbird [humEN06Rtde; MAP 47.63%; Not Pooled]
reina [reinaENtdtest; MAP 43.66%; Not Pooled]
0.5 dcu [dcudesceng12075; MAP 43.48%; Not Pooled]
Difference
daedalus [enFSenR2S; MAP 39.69%; Not Pooled]
0 colesir [CoLesIRenTst; MAP 37.64%; Not Pooled]
−0.5
−1
166 167 168 169 170 171 172 173 174 175 176 177 178 179 190 191 192 193 194 195 196 197 198 199 200
Topic Identifier
22
AH-ROBUST-CLEF2006 Track Overview Results and Graphs AH-ROBUST-MONO-EN-CLEF2006
Ad−Hoc Robust Monolingual English track − Box Plot of the Topics
dcunarreng1005 [MAP 49.12%; Not Pooled]
humEN06Rtde [MAP 47.63%; Not Pooled]
reinaENtdtest [MAP 43.66%; Not Pooled]
dcudesceng12075 [MAP 43.48%; Not Pooled]
enyenRFS3456 [MAP 43.38%; Not Pooled]
enFSenR5S [MAP 43.02%; Not Pooled]
Experiments
enFSenR3S [MAP 42.89%; Not Pooled]
humEN06Rtd [MAP 42.80%; Not Pooled]
reinaENttest [MAP 40.66%; Not Pooled]
enFSenR2S [MAP 39.69%; Not Pooled]
CoLesIRenTst [MAP 37.64%; Not Pooled]
uhienmo1 [MAP 7.98%; Not Pooled]
uhienmo2 [MAP 7.12%; Not Pooled]
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
Mean Average Precision
23
AH-ROBUST-CLEF2006 Track Overview Results and Graphs AH-ROBUST-MONO-EN-CLEF2006
Ad−Hoc Robust Monolingual English track − Tukey T test with "top group" highlighted
dcunarreng1005
humEN06Rtde
reinaENtdtest
dcudesceng12075
humEN06Rtd
enyenRFS3456
Experiments
enFSenR3S
enFSenR5S
reinaENttest
enFSenR2S
CoLesIRenTst
uhienmo1
uhienmo2
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9
arcsin(sqrt(Mean average precision))
24
AH-ROBUST-CLEF2006 Track Overview Results and Graphs AH-ROBUST-MONO-EN-CLEF2006
Ad−Hoc Robust Monolingual English track Top 5 Participants − Retrieved documents vs Precision
100%
hummingbird [humEN06Rtde; R−Prec 44.80%; Not Pooled]
reina [reinaENtdtest; R−Prec 40.74%; Not Pooled]
90% dcu [dcudesceng12075; R−Prec 40.51%; Not Pooled]
daedalus [enFSenR2S; R−Prec 36.69%; Not Pooled]
colesir [CoLesIRenTst; R−Prec 36.21%; Not Pooled]
80%
70%
60%
R−Precision
50%
40%
30%
20%
10%
0%
5 10 15 20 30 100 200 500 1000
Retrieved Documents (logarithmic scale)
Ad−Hoc Robust Monolingual English track Top 5 Participants − Comparison to Median R−Precision by Topic (Topics 041 to 085)
1
hummingbird [humEN06Rtde; R−Prec 44.80%; Not Pooled]
reina [reinaENtdtest; R−Prec 40.74%; Not Pooled]
0.5 dcu [dcudesceng12075; R−Prec 40.51%; Not Pooled]
Difference
daedalus [enFSenR2S; R−Prec 36.69%; Not Pooled]
0 colesir [CoLesIRenTst; R−Prec 36.21%; Not Pooled]
−0.5
−1
041 042 043 044 045 046 047 048 049 060 061 062 063 064 065 066 067 068 069 080 081 082 083 084 085
Topic Identifier
Ad−Hoc Robust Monolingual English track Top 5 Participants − Comparison to Median R−Precision by Topic (Topics 086 to 130)
1
hummingbird [humEN06Rtde; R−Prec 44.80%; Not Pooled]
reina [reinaENtdtest; R−Prec 40.74%; Not Pooled]
0.5 dcu [dcudesceng12075; R−Prec 40.51%; Not Pooled]
Difference
daedalus [enFSenR2S; R−Prec 36.69%; Not Pooled]
0 colesir [CoLesIRenTst; R−Prec 36.21%; Not Pooled]
−0.5
−1
086 087 088 089 090 091 092 093 094 095 096 097 098 099 110 111 112 113 114 115 116 117 118 119 130
Topic Identifier
Ad−Hoc Robust Monolingual English track Top 5 Participants − Comparison to Median R−Precision by Topic (Topics 131 to 165)
1
hummingbird [humEN06Rtde; R−Prec 44.80%; Not Pooled]
reina [reinaENtdtest; R−Prec 40.74%; Not Pooled]
0.5 dcu [dcudesceng12075; R−Prec 40.51%; Not Pooled]
Difference
daedalus [enFSenR2S; R−Prec 36.69%; Not Pooled]
0 colesir [CoLesIRenTst; R−Prec 36.21%; Not Pooled]
−0.5
−1
131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 160 161 162 163 164 165
Topic Identifier
Ad−Hoc Robust Monolingual English track Top 5 Participants − Comparison to Median R−Precision by Topic (Topics 166 to 200)
1
hummingbird [humEN06Rtde; R−Prec 44.80%; Not Pooled]
reina [reinaENtdtest; R−Prec 40.74%; Not Pooled]
0.5 dcu [dcudesceng12075; R−Prec 40.51%; Not Pooled]
Difference
daedalus [enFSenR2S; R−Prec 36.69%; Not Pooled]
0 colesir [CoLesIRenTst; R−Prec 36.21%; Not Pooled]
−0.5
−1
166 167 168 169 170 171 172 173 174 175 176 177 178 179 190 191 192 193 194 195 196 197 198 199 200
Topic Identifier
25
AH-ROBUST-CLEF2006 Track Overview Results and Graphs AH-ROBUST-MONO-EN-CLEF2006
Ad−Hoc Robust Monolingual English track − Box Plot of the Topics
dcunarreng1005 [R−Prec 45.72%; Not Pooled]
humEN06Rtde [R−Prec 44.80%; Not Pooled]
reinaENtdtest [R−Prec 40.74%; Not Pooled]
dcudesceng12075 [R−Prec 40.51%; Not Pooled]
enyenRFS3456 [R−Prec 40.19%; Not Pooled]
enFSenR5S [R−Prec 40.01%; Not Pooled]
Experiments
humEN06Rtd [R−Prec 39.78%; Not Pooled]
enFSenR3S [R−Prec 39.70%; Not Pooled]
reinaENttest [R−Prec 39.47%; Not Pooled]
enFSenR2S [R−Prec 36.69%; Not Pooled]
CoLesIRenTst [R−Prec 36.21%; Not Pooled]
uhienmo1 [R−Prec 7.29%; Not Pooled]
uhienmo2 [R−Prec 5.78%; Not Pooled]
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
R−Precision
26
AH-ROBUST-CLEF2006 Track Overview Results and Graphs AH-ROBUST-MONO-EN-CLEF2006
Ad−Hoc Robust Monolingual English track − Tukey T test with "top group" highlighted
dcunarreng1005
humEN06Rtde
reinaENtdtest
reinaENttest
dcudesceng12075
enFSenR5S
Experiments
humEN06Rtd
enyenRFS3456
enFSenR3S
CoLesIRenTst
enFSenR2S
uhienmo1
uhienmo2
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8
arcsin(sqrt(R Precision))
27
AH-ROBUST-CLEF2006 Track Overview Results and Graphs AH-ROBUST-MONO-EN-CLEF2006
Average Precision R-Precision
Topic Minimum 1st Q. Median 3rd Q. Maximum Mean Std Minimum 1st Q. Median 3rd Q. Maximum Mean Std
041 0.0000 0.1644 0.1794 0.2168 0.2749 0.1724 0.0855 0.0000 0.1250 0.2500 0.2500 0.3750 0.1923 0.1096
042 0.0000 0.2546 0.2982 0.3555 0.3870 0.2689 0.1298 0.0000 0.3088 0.3529 0.4118 0.4510 0.3122 0.1462
043 0.0000 0.4425 0.6002 0.8193 0.8349 0.5441 0.2916 0.0000 0.4091 0.5455 0.7273 0.8182 0.5105 0.2664
044 0.0000 0.0668 0.5200 0.5255 0.5294 0.3348 0.2296 0.0000 0.0000 0.5000 0.5000 0.5000 0.3462 0.2402
045 0.0000 0.5857 0.6450 0.7025 0.7445 0.5604 0.2542 0.0000 0.5909 0.6364 0.6667 0.6970 0.5361 0.2410
046 0.0000 0.3113 0.3610 0.3819 0.3913 0.3020 0.1391 0.0000 0.3333 0.4444 0.5556 0.5556 0.3761 0.1900
047 0.0000 0.5638 0.6192 0.6614 0.7132 0.5374 0.2430 0.0000 0.5244 0.5854 0.6098 0.6585 0.4972 0.2250
048 0.0000 0.0528 0.1114 0.2479 0.3611 0.1467 0.1187 0.0000 0.0000 0.1515 0.3106 0.3636 0.1515 0.1464
049 0.0000 0.1512 0.1687 0.2286 0.3154 0.1740 0.0946 0.0000 0.1429 0.1429 0.2857 0.3571 0.1813 0.1076
060 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000
061 0.0000 0.3811 0.5614 0.6084 0.7200 0.4694 0.2328 0.0000 0.3500 0.4000 0.6000 0.8000 0.4308 0.2428
062 0.0000 1.0000 1.0000 1.0000 1.0000 0.8462 0.3755 0.0000 1.0000 1.0000 1.0000 1.0000 0.8462 0.3755
063 0.0000 0.0032 0.0175 0.0274 0.5000 0.0534 0.1349 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000
064 0.0000 0.2255 0.3066 0.5725 0.8354 0.3710 0.2529 0.0000 0.2000 0.3000 0.7000 0.9000 0.4077 0.2957
065 0.0000 0.2726 0.3751 0.4718 0.6051 0.3470 0.1905 0.0000 0.3000 0.3333 0.5333 0.6000 0.3538 0.1970
066 0.0000 0.7014 0.7548 0.7722 0.8184 0.6352 0.2846 0.0000 0.7273 0.7273 0.7273 0.8182 0.6224 0.2774
067 0.0000 0.0250 0.1262 0.2514 0.2557 0.1264 0.1018 0.0000 0.0000 0.0000 0.5000 0.5000 0.1538 0.2402
068 0.0000 0.1804 0.3788 0.4100 0.8016 0.3274 0.2196 0.0000 0.2857 0.4286 0.4643 0.7143 0.3736 0.2224
069 0.0000 0.2376 0.3377 0.4254 0.6135 0.3103 0.1756 0.0000 0.2500 0.3750 0.3750 0.6250 0.2981 0.1657
080 0.0000 0.2674 0.3938 0.4497 0.4709 0.3219 0.1735 0.0000 0.3214 0.3929 0.4286 0.4821 0.3352 0.1675
081 0.0000 0.7543 0.8058 0.8590 0.9308 0.6992 0.3153 0.0000 0.5938 0.7500 0.7812 0.8750 0.6250 0.2976
082 0.0000 0.4122 0.6241 0.6458 0.7985 0.5100 0.2501 0.0000 0.4000 0.6000 0.6000 0.8000 0.4923 0.2431
083 0.0000 0.1279 0.2319 0.2854 0.3649 0.2033 0.1232 0.0000 0.0000 0.2000 0.2000 0.4000 0.1385 0.1261
084 0.0000 0.5751 0.6788 0.7027 0.7416 0.5605 0.2593 0.0000 0.5667 0.6000 0.6667 0.6667 0.5231 0.2386
085 0.0000 0.1880 0.4541 0.4823 0.5948 0.3621 0.2010 0.0000 0.1719 0.4375 0.4688 0.5000 0.3317 0.1860
086 0.0000 0.3222 0.4457 0.5230 0.5542 0.3882 0.1919 0.0000 0.3872 0.4878 0.5244 0.5610 0.4081 0.1923
087 0.0000 0.4672 0.5852 0.6157 0.7753 0.4957 0.2414 0.0000 0.2917 0.5000 0.5000 0.6667 0.3846 0.2056
088 0.0000 0.1349 0.2000 0.6250 1.0000 0.3644 0.3824 0.0000 0.0000 0.0000 0.2500 1.0000 0.2308 0.4385
089 0.0000 0.2429 0.6451 0.7738 1.0000 0.5341 0.3456 0.0000 0.2500 0.5000 0.5625 1.0000 0.4615 0.3203
090 0.0000 0.0509 0.3086 0.3845 0.6236 0.2675 0.1974 0.0000 0.0625 0.2500 0.2708 0.5833 0.2308 0.1805
091 0.0000 0.0773 0.2076 0.5009 0.5922 0.2731 0.2223 0.0000 0.1071 0.2857 0.4286 0.4286 0.2637 0.1735
092 0.0000 0.5872 0.7298 0.7452 0.7972 0.6017 0.2760 0.0000 0.5741 0.6667 0.6667 0.7407 0.5527 0.2510
093 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000
094 0.0000 0.6987 0.7739 0.7869 0.8194 0.6454 0.2896 0.0000 0.6458 0.6667 0.6875 0.7500 0.5769 0.2601
095 0.0000 0.2525 0.3171 0.4055 0.4750 0.2890 0.1482 0.0000 0.2630 0.3333 0.4401 0.5208 0.3149 0.1651
096 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000
097 0.0000 1.0000 1.0000 1.0000 1.0000 0.8462 0.3755 0.0000 1.0000 1.0000 1.0000 1.0000 0.8462 0.3755
098 0.0000 0.2500 0.5000 1.0000 1.0000 0.5385 0.4147 0.0000 0.0000 0.0000 1.0000 1.0000 0.3846 0.5064
099 0.0000 0.1059 0.3562 0.5062 0.6208 0.3162 0.2094 0.0000 0.2500 0.2857 0.4643 0.5714 0.3187 0.1948
110 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000
111 0.0000 0.1421 0.3483 0.3987 0.4869 0.2831 0.1706 0.0000 0.2000 0.4000 0.4125 0.4500 0.3154 0.1625
112 0.0000 0.4418 0.5422 0.5856 0.6257 0.4546 0.2219 0.0000 0.3750 0.5000 0.5250 0.6000 0.4231 0.2048
113 0.0000 0.0026 0.0084 0.0373 0.3244 0.0397 0.0872 0.0000 0.0000 0.0000 0.0000 0.2500 0.0192 0.0693
114 0.0000 0.4356 0.5306 0.6154 0.6408 0.4610 0.2178 0.0000 0.4286 0.5714 0.7143 0.7143 0.4835 0.2442
115 0.0000 0.1473 0.1716 0.1883 0.2181 0.1475 0.0693 0.0000 0.1600 0.2000 0.2400 0.2400 0.1754 0.0825
116 0.0000 0.3248 0.3869 0.4091 0.4984 0.3328 0.1554 0.0000 0.2727 0.3636 0.4545 0.4545 0.3287 0.1641
117 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000
118 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000
119 0.0000 0.6730 0.7685 0.7864 0.8384 0.6415 0.2927 0.0000 0.5938 0.6786 0.7009 0.7500 0.5714 0.2589
130 0.0000 0.3799 0.4739 0.5688 0.6709 0.4231 0.2092 0.0000 0.4167 0.5079 0.5714 0.6032 0.4371 0.2047
28
AH-ROBUST-CLEF2006 Track Overview Results and Graphs AH-ROBUST-MONO-EN-CLEF2006
Average Precision R-Precision
Topic Minimum 1st Q. Median 3rd Q. Maximum Mean Std Minimum 1st Q. Median 3rd Q. Maximum Mean Std
131 0.0000 0.3712 0.4469 0.4868 0.6263 0.3943 0.1976 0.0000 0.3250 0.4250 0.4937 0.6500 0.3885 0.1989
132 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000
133 0.0000 0.1354 0.2261 0.2747 0.5552 0.2147 0.1514 0.0000 0.1000 0.3000 0.3000 0.5000 0.2462 0.1506
134 0.0000 0.7135 0.7500 0.7513 0.7541 0.6241 0.2799 0.0000 0.7500 0.7500 0.7500 0.7500 0.6346 0.2817
135 0.0000 0.0828 0.1720 0.3379 0.4126 0.2028 0.1481 0.0000 0.1500 0.4000 0.4000 0.4000 0.2769 0.1739
136 0.0000 0.2500 0.3333 0.5000 0.5000 0.3462 0.1827 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000
137 0.0000 0.1406 0.5000 0.5000 0.5000 0.3189 0.2141 0.0000 0.0000 0.5000 0.5000 0.5000 0.3077 0.2532
138 0.0000 0.2724 0.4129 0.5655 0.5833 0.3769 0.2028 0.0000 0.3750 0.5000 0.5000 0.5000 0.3942 0.1829
139 0.0000 0.0397 0.0422 0.0434 0.0489 0.0361 0.0162 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000
140 0.0000 0.2601 0.3509 0.4544 0.6148 0.3415 0.2028 0.0000 0.3393 0.4429 0.5179 0.6000 0.3912 0.1978
141 0.0032 1.0000 1.0000 1.0000 1.0000 0.8469 0.3738 0.0000 1.0000 1.0000 1.0000 1.0000 0.8462 0.3755
142 0.6250 0.7714 0.8874 0.9778 1.0000 0.8609 0.1323 0.6250 0.8438 0.8750 0.8750 1.0000 0.8462 0.1159
143 0.0155 0.7136 0.8170 0.8517 0.9366 0.6997 0.3103 0.0256 0.6538 0.7436 0.7756 0.8718 0.6391 0.2729
144 0.0556 0.5056 0.5868 0.7056 0.8667 0.5436 0.2323 0.0000 0.3333 0.3333 0.6667 0.6667 0.4615 0.2168
145 0.0000 0.4162 0.5909 0.6685 0.7901 0.5019 0.2761 0.0000 0.4276 0.5789 0.6447 0.7368 0.4858 0.2602
146 0.0035 0.2885 0.5026 0.5082 0.5135 0.3900 0.1976 0.0000 0.3750 0.5000 0.5000 0.5000 0.3846 0.2193
147 0.0989 0.4559 0.7804 0.7946 0.8744 0.6034 0.2647 0.1176 0.4755 0.6667 0.7304 0.7843 0.5581 0.2272
148 0.0166 0.0564 0.0727 0.1966 0.2062 0.1068 0.0721 0.0000 0.1667 0.1667 0.1667 0.1667 0.1410 0.0626
149 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000
160 0.0407 0.0554 0.0694 0.1049 0.1667 0.0837 0.0377 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000
161 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000
162 0.1429 0.3125 0.5000 0.6250 1.0000 0.5156 0.3055 0.0000 0.0000 0.0000 0.2500 1.0000 0.2308 0.4385
163 0.0000 0.8142 0.8645 0.8776 0.8973 0.7258 0.3239 0.0000 0.7172 0.7869 0.7971 0.8197 0.6551 0.2936
164 0.0360 0.0911 0.1278 0.1676 0.2255 0.1281 0.0581 0.0000 0.1019 0.1111 0.1481 0.3333 0.1339 0.0836
165 0.2000 1.0000 1.0000 1.0000 1.0000 0.9000 0.2517 0.0000 1.0000 1.0000 1.0000 1.0000 0.8462 0.3755
166 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000
167 0.0000 0.2391 0.3067 0.3142 0.3490 0.2482 0.1150 0.0000 0.2000 0.4000 0.4000 0.4000 0.2923 0.1553
168 0.3465 0.3922 0.4598 0.5070 0.7185 0.4721 0.1022 0.3333 0.3889 0.4444 0.5139 0.6667 0.4615 0.0972
169 0.0000 0.0100 0.0262 0.0582 0.0765 0.0314 0.0250 0.0000 0.0000 0.0000 0.0000 0.1667 0.0256 0.0626
170 0.1250 0.8125 1.0000 1.0000 1.0000 0.8109 0.3603 0.0000 0.7500 1.0000 1.0000 1.0000 0.7692 0.4385
171 0.0000 0.0809 0.1114 0.1265 0.1844 0.0973 0.0518 0.0000 0.0526 0.0526 0.1053 0.1579 0.0769 0.0509
172 0.0199 0.0687 0.1108 0.1480 0.1896 0.1052 0.0553 0.0000 0.0000 0.0000 0.1786 0.2857 0.0879 0.1242
173 0.0000 1.0000 1.0000 1.0000 1.0000 0.8462 0.3755 0.0000 1.0000 1.0000 1.0000 1.0000 0.8462 0.3755
174 1.0000 1.0000 1.0000 1.0000 1.0000 1.0000 0.0000 1.0000 1.0000 1.0000 1.0000 1.0000 1.0000 0.0000
175 0.0000 0.6787 0.7492 0.8665 0.9071 0.6573 0.3051 0.0000 0.5357 0.7143 0.8571 0.8571 0.6044 0.2983
176 0.0032 0.9143 0.9330 0.9428 0.9471 0.7902 0.3488 0.0256 0.7949 0.8462 0.8526 0.8718 0.7140 0.3066
177 0.0158 0.3153 0.3724 0.4313 0.6169 0.3681 0.1614 0.0000 0.3462 0.3846 0.4808 0.5385 0.3846 0.1538
178 0.0000 0.0263 0.0875 0.1423 0.2722 0.1010 0.0913 0.0000 0.0000 0.0000 0.2500 0.2500 0.1154 0.1297
179 0.0188 0.0258 0.6407 0.8133 0.8954 0.5012 0.3683 0.0000 0.0341 0.5909 0.7386 0.8636 0.4615 0.3396
190 0.0006 0.2130 0.5424 0.7292 0.7531 0.4555 0.2942 0.0000 0.1944 0.4444 0.6667 0.7778 0.4274 0.2827
191 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000
192 0.0186 0.1604 0.4730 0.5822 0.7722 0.4027 0.2452 0.0000 0.2917 0.3333 0.5417 0.8333 0.3974 0.2504
193 0.2270 0.3874 0.4637 0.5252 0.5371 0.4373 0.1003 0.1429 0.2857 0.4286 0.4286 0.5714 0.3736 0.1242
194 0.0000 0.0153 0.1667 0.2500 1.0000 0.1949 0.2687 0.0000 0.0000 0.0000 0.0000 1.0000 0.0769 0.2774
195 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000
196 0.5000 0.5000 0.5000 1.0000 1.0000 0.6538 0.2402 0.0000 0.0000 0.0000 1.0000 1.0000 0.3077 0.4804
197 0.1193 0.2598 0.7609 0.8422 0.9288 0.5944 0.2950 0.1000 0.2850 0.6800 0.7450 0.8600 0.5415 0.2697
198 0.0096 0.0243 1.0000 1.0000 1.0000 0.5571 0.4985 0.0000 0.0000 1.0000 1.0000 1.0000 0.5385 0.5189
199 0.4867 0.6248 0.6553 0.6804 0.7108 0.6374 0.0622 0.4000 0.5000 0.6000 0.7000 0.7000 0.5846 0.0987
200 0.0162 0.2380 0.3615 0.5911 0.6509 0.3689 0.2199 0.0000 0.3333 0.4444 0.5833 0.6667 0.4274 0.2215
ALL 0.0712 0.3918 0.4289 0.4352 0.4912 0.3762 0.1367 0.0578 0.3657 0.3978 0.4057 0.4572 0.3515 0.1296
29
30
AH-ROBUST-CLEF2006 Track Overview Results and Graphs AH-ROBUST-MONO-ES-CLEF2006
Ad−Hoc Robust Monolingual Spanish track Top 5 Participants − Interpolated Recall vs Average Precision
100%
hummingbird [humES06Rtde; MAP 45.66%; Not Pooled]
reina [reinaEStdtest; MAP 44.01%; Not Pooled]
90% dcu [dcudescsp12075; MAP 42.14%; Not Pooled]
daedalus [esFSesR2S; MAP 40.40%; Not Pooled]
colesir [CoLesIResTst; MAP 40.17%; Not Pooled]
80%
70%
Average Precision
60%
50%
40%
30%
20%
10%
0%
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
Interpolated Recall
Ad−Hoc Robust Monolingual Spanish track Top 5 Participants − Comparison to Median Mean Average Precision by Topic (Topics 041to 085)
1
hummingbird [humES06Rtde; MAP 45.66%; Not Pooled]
reina [reinaEStdtest; MAP 44.01%; Not Pooled]
0.5 dcu [dcudescsp12075; MAP 42.14%; Not Pooled]
Difference
daedalus [esFSesR2S; MAP 40.40%; Not Pooled]
0 colesir [CoLesIResTst; MAP 40.17%; Not Pooled]
−0.5
−1
041 042 043 044 045 046 047 048 049 060 061 062 063 064 065 066 067 068 069 080 081 082 083 084 085
Topic Identifier
Ad−Hoc Robust Monolingual Spanish track Top 5 Participants − Comparison to Median Mean Average Precision by Topic (Topics 086to 130)
1
hummingbird [humES06Rtde; MAP 45.66%; Not Pooled]
reina [reinaEStdtest; MAP 44.01%; Not Pooled]
0.5 dcu [dcudescsp12075; MAP 42.14%; Not Pooled]
Difference
daedalus [esFSesR2S; MAP 40.40%; Not Pooled]
0 colesir [CoLesIResTst; MAP 40.17%; Not Pooled]
−0.5
−1
086 087 088 089 090 091 092 093 094 095 096 097 098 099 110 111 112 113 114 115 116 117 118 119 130
Topic Identifier
Ad−Hoc Robust Monolingual Spanish track Top 5 Participants − Comparison to Median Mean Average Precision by Topic (Topics 131to 165)
1
hummingbird [humES06Rtde; MAP 45.66%; Not Pooled]
reina [reinaEStdtest; MAP 44.01%; Not Pooled]
0.5 dcu [dcudescsp12075; MAP 42.14%; Not Pooled]
Difference
daedalus [esFSesR2S; MAP 40.40%; Not Pooled]
0 colesir [CoLesIResTst; MAP 40.17%; Not Pooled]
−0.5
−1
131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 160 161 162 163 164 165
Topic Identifier
Ad−Hoc Robust Monolingual Spanish track Top 5 Participants − Comparison to Median Mean Average Precision by Topic (Topics 166to 200)
1
hummingbird [humES06Rtde; MAP 45.66%; Not Pooled]
reina [reinaEStdtest; MAP 44.01%; Not Pooled]
0.5 dcu [dcudescsp12075; MAP 42.14%; Not Pooled]
Difference
daedalus [esFSesR2S; MAP 40.40%; Not Pooled]
0 colesir [CoLesIResTst; MAP 40.17%; Not Pooled]
−0.5
−1
166 167 168 169 170 171 172 173 174 175 176 177 178 179 190 191 192 193 194 195 196 197 198 199 200
Topic Identifier
31
AH-ROBUST-CLEF2006 Track Overview Results and Graphs AH-ROBUST-MONO-ES-CLEF2006
Ad−Hoc Robust Monolingual Spanish track − Box Plot of the Topics
esFSesR3S [MAP 45.84%; Not Pooled]
dcu1500narrsp1005.qy1 [MAP 45.81%; Not Pooled]
humES06Rtde [MAP 45.66%; Not Pooled]
esx011esRFW3FS3 [MAP 45.09%; Not Pooled]
esx021esRFW3FS3 [MAP 44.46%; Not Pooled]
Experiments
reinaEStdtest [MAP 44.01%; Not Pooled]
dcudescsp12075 [MAP 42.14%; Not Pooled]
humES06Rtd [MAP 42.00%; Not Pooled]
esFSesR2S [MAP 40.40%; Not Pooled]
CoLesIResTst [MAP 40.17%; Not Pooled]
reinaESttest [MAP 38.93%; Not Pooled]
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
Mean Average Precision
32
AH-ROBUST-CLEF2006 Track Overview Results and Graphs AH-ROBUST-MONO-ES-CLEF2006
Ad−Hoc Robust Monolingual Spanish track − Tukey T test with "top group" highlighted
esFSesR3S
dcu1500narrsp1005.qy1
esx011esRFW3FS3
humES06Rtde
esx021esRFW3FS3
Experiments
reinaEStdtest
dcudescsp12075
humES06Rtd
esFSesR2S
CoLesIResTst
reinaESttest
0.55 0.6 0.65 0.7 0.75 0.8
arcsin(sqrt(Mean average precision))
33
AH-ROBUST-CLEF2006 Track Overview Results and Graphs AH-ROBUST-MONO-ES-CLEF2006
Ad−Hoc Robust Monolingual Spanish track Top 5 Participants − Retrieved documents vs Precision
100%
hummingbird [humES06Rtde; R−Prec 43.99%; Not Pooled]
reina [reinaEStdtest; R−Prec 42.96%; Not Pooled]
90% dcu [dcudescsp12075; R−Prec 40.58%; Not Pooled]
daedalus [esFSesR2S; R−Prec 40.15%; Not Pooled]
colesir [CoLesIResTst; R−Prec 40.03%; Not Pooled]
80%
70%
60%
R−Precision
50%
40%
30%
20%
10%
0%
5 10 15 20 30 100 200 500 1000
Retrieved Documents (logarithmic scale)
Ad−Hoc Robust Monolingual Spanish track Top 5 Participants − Comparison to Median R−Precision by Topic (Topics 041 to 085)
1
hummingbird [humES06Rtde; R−Prec 43.99%; Not Pooled]
reina [reinaEStdtest; R−Prec 42.96%; Not Pooled]
0.5 dcu [dcudescsp12075; R−Prec 40.58%; Not Pooled]
Difference
daedalus [esFSesR2S; R−Prec 40.15%; Not Pooled]
0 colesir [CoLesIResTst; R−Prec 40.03%; Not Pooled]
−0.5
−1
041 042 043 044 045 046 047 048 049 060 061 062 063 064 065 066 067 068 069 080 081 082 083 084 085
Topic Identifier
Ad−Hoc Robust Monolingual Spanish track Top 5 Participants − Comparison to Median R−Precision by Topic (Topics 086 to 130)
1
hummingbird [humES06Rtde; R−Prec 43.99%; Not Pooled]
reina [reinaEStdtest; R−Prec 42.96%; Not Pooled]
0.5 dcu [dcudescsp12075; R−Prec 40.58%; Not Pooled]
Difference
daedalus [esFSesR2S; R−Prec 40.15%; Not Pooled]
0 colesir [CoLesIResTst; R−Prec 40.03%; Not Pooled]
−0.5
−1
086 087 088 089 090 091 092 093 094 095 096 097 098 099 110 111 112 113 114 115 116 117 118 119 130
Topic Identifier
Ad−Hoc Robust Monolingual Spanish track Top 5 Participants − Comparison to Median R−Precision by Topic (Topics 131 to 165)
1
hummingbird [humES06Rtde; R−Prec 43.99%; Not Pooled]
reina [reinaEStdtest; R−Prec 42.96%; Not Pooled]
0.5 dcu [dcudescsp12075; R−Prec 40.58%; Not Pooled]
Difference
daedalus [esFSesR2S; R−Prec 40.15%; Not Pooled]
0 colesir [CoLesIResTst; R−Prec 40.03%; Not Pooled]
−0.5
−1
131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 160 161 162 163 164 165
Topic Identifier
Ad−Hoc Robust Monolingual Spanish track Top 5 Participants − Comparison to Median R−Precision by Topic (Topics 166 to 200)
1
hummingbird [humES06Rtde; R−Prec 43.99%; Not Pooled]
reina [reinaEStdtest; R−Prec 42.96%; Not Pooled]
0.5 dcu [dcudescsp12075; R−Prec 40.58%; Not Pooled]
Difference
daedalus [esFSesR2S; R−Prec 40.15%; Not Pooled]
0 colesir [CoLesIResTst; R−Prec 40.03%; Not Pooled]
−0.5
−1
166 167 168 169 170 171 172 173 174 175 176 177 178 179 190 191 192 193 194 195 196 197 198 199 200
Topic Identifier
34
AH-ROBUST-CLEF2006 Track Overview Results and Graphs AH-ROBUST-MONO-ES-CLEF2006
Ad−Hoc Robust Monolingual Spanish track − Box Plot of the Topics
esFSesR3S [R−Prec 44.77%; Not Pooled]
esx011esRFW3FS3 [R−Prec 44.57%; Not Pooled]
esx021esRFW3FS3 [R−Prec 44.20%; Not Pooled]
humES06Rtde [R−Prec 43.99%; Not Pooled]
dcu1500narrsp1005.qy1 [R−Prec 43.96%; Not Pooled]
Experiments
reinaEStdtest [R−Prec 42.96%; Not Pooled]
humES06Rtd [R−Prec 41.41%; Not Pooled]
dcudescsp12075 [R−Prec 40.58%; Not Pooled]
esFSesR2S [R−Prec 40.15%; Not Pooled]
CoLesIResTst [R−Prec 40.03%; Not Pooled]
reinaESttest [R−Prec 38.82%; Not Pooled]
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
R−Precision
35
AH-ROBUST-CLEF2006 Track Overview Results and Graphs AH-ROBUST-MONO-ES-CLEF2006
Ad−Hoc Robust Monolingual Spanish track − Tukey T test with "top group" highlighted
esFSesR3S
esx011esRFW3FS3
esx021esRFW3FS3
dcu1500narrsp1005.qy1
humES06Rtde
Experiments
reinaEStdtest
humES06Rtd
esFSesR2S
CoLesIResTst
dcudescsp12075
reinaESttest
0.55 0.6 0.65 0.7 0.75 0.8
arcsin(sqrt(R Precision))
36
AH-ROBUST-CLEF2006 Track Overview Results and Graphs AH-ROBUST-MONO-ES-CLEF2006
Average Precision R-Precision
Topic Minimum 1st Q. Median 3rd Q. Maximum Mean Std Minimum 1st Q. Median 3rd Q. Maximum Mean Std
041 0.2119 0.3289 0.3587 0.6256 0.7200 0.4352 0.1859 0.3125 0.3750 0.3750 0.5312 0.6250 0.4432 0.1099
042 0.2214 0.5869 0.6385 0.7111 0.7130 0.6067 0.1469 0.3295 0.5757 0.6130 0.6580 0.6705 0.5869 0.1001
043 0.0142 0.1644 0.3497 0.4127 0.4168 0.2794 0.1497 0.0556 0.2222 0.3889 0.3889 0.4444 0.3030 0.1300
044 0.1362 0.2067 0.2440 0.2874 0.3388 0.2436 0.0571 0.1250 0.2063 0.2500 0.2875 0.3750 0.2455 0.0650
045 0.4512 0.5621 0.6076 0.6347 0.6900 0.6019 0.0670 0.4921 0.5707 0.5864 0.6283 0.6911 0.5945 0.0553
046 0.0144 0.0252 0.0287 0.0391 0.1111 0.0377 0.0260 0.0000 0.0000 0.0000 0.0667 0.1333 0.0364 0.0458
047 0.1043 0.1221 0.1486 0.1896 0.2034 0.1529 0.0361 0.1759 0.1875 0.2222 0.2500 0.2500 0.2180 0.0300
048 0.0104 0.0760 0.1065 0.1656 0.1788 0.1151 0.0581 0.0104 0.1068 0.1562 0.1562 0.2292 0.1383 0.0588
049 0.0984 0.1050 0.1163 0.1784 0.2533 0.1474 0.0539 0.1579 0.1579 0.2105 0.3026 0.3684 0.2249 0.0784
060 0.2266 0.3994 0.4879 0.5977 0.7475 0.4851 0.1607 0.3067 0.4367 0.5067 0.5900 0.6800 0.4994 0.1163
061 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000
062 0.2067 0.2141 0.2405 0.2633 0.3564 0.2475 0.0429 0.1333 0.1750 0.2000 0.2500 0.4333 0.2273 0.0941
063 0.7162 0.7267 0.8102 0.8584 0.8785 0.7992 0.0636 0.6441 0.6568 0.7627 0.7797 0.8136 0.7319 0.0660
064 1.0000 1.0000 1.0000 1.0000 1.0000 1.0000 0.0000 1.0000 1.0000 1.0000 1.0000 1.0000 1.0000 0.0000
065 0.1248 0.1615 0.1796 0.2467 0.2536 0.1915 0.0478 0.2297 0.2736 0.3243 0.3716 0.3784 0.3206 0.0510
066 0.8744 0.8931 0.9334 0.9341 0.9415 0.9145 0.0256 0.7778 0.8207 0.8586 0.8687 0.8687 0.8411 0.0341
067 0.1054 0.1137 0.1289 0.2022 0.3570 0.1682 0.0796 0.0612 0.1429 0.1633 0.2959 0.4286 0.2152 0.1198
068 0.0637 0.0742 0.0904 0.1120 0.2473 0.1055 0.0522 0.0000 0.0000 0.0000 0.1111 0.2222 0.0606 0.0911
069 0.5964 0.7843 0.8750 0.8833 0.8958 0.8175 0.1042 0.6667 0.7083 0.8333 0.8333 0.8333 0.7879 0.0778
080 0.3238 0.4122 0.4554 0.4735 0.4738 0.4364 0.0469 0.3714 0.4612 0.4898 0.4990 0.5306 0.4772 0.0427
081 0.6315 0.7640 0.7872 0.8205 0.8619 0.7770 0.0670 0.6552 0.7356 0.7816 0.7931 0.8046 0.7544 0.0545
082 0.3925 0.7348 0.8242 0.8438 0.9823 0.7628 0.1711 0.4211 0.6908 0.7632 0.7829 0.9474 0.7201 0.1364
083 0.4168 0.4657 0.5234 0.5544 0.6094 0.5151 0.0561 0.4444 0.4444 0.4444 0.4444 0.5556 0.4646 0.0449
084 0.1701 0.2233 0.3115 0.3139 0.3783 0.2851 0.0667 0.1250 0.2812 0.3750 0.3750 0.3750 0.3295 0.0843
085 0.7224 0.7505 0.8022 0.8373 0.8389 0.7908 0.0444 0.7009 0.7360 0.7570 0.7570 0.7850 0.7511 0.0234
086 0.2468 0.2690 0.3012 0.3149 0.3207 0.2916 0.0262 0.2500 0.2902 0.3214 0.3750 0.4107 0.3312 0.0488
087 0.2637 0.3319 0.3912 0.5962 0.7650 0.4531 0.1679 0.2955 0.3750 0.4205 0.5881 0.7159 0.4659 0.1356
088 0.2805 0.3063 0.3424 0.4144 0.4956 0.3629 0.0755 0.1429 0.2857 0.4286 0.4286 0.4286 0.3636 0.0982
089 0.4379 0.4421 0.4483 0.4777 0.6824 0.4768 0.0725 0.5000 0.5000 0.5625 0.5625 0.6250 0.5398 0.0421
090 0.0204 0.0527 0.0750 0.1164 0.1257 0.0809 0.0376 0.0667 0.0667 0.0667 0.1333 0.1333 0.0970 0.0348
091 0.0480 0.3211 0.3410 0.3828 0.4404 0.3331 0.1021 0.0753 0.3226 0.3441 0.3763 0.4194 0.3304 0.0906
092 0.1920 0.2449 0.2706 0.2881 0.3494 0.2677 0.0421 0.1860 0.2587 0.3023 0.3256 0.3837 0.2960 0.0603
093 0.1303 0.1769 0.2620 0.2819 0.3491 0.2376 0.0676 0.2857 0.2857 0.2857 0.3571 0.4286 0.3247 0.0491
094 0.7511 0.8031 0.8686 0.8723 0.8904 0.8428 0.0480 0.8182 0.8182 0.8182 0.8523 0.8636 0.8306 0.0212
095 0.0045 0.0548 0.1005 0.1907 0.2649 0.1200 0.0861 0.0343 0.1456 0.2212 0.3014 0.3988 0.2198 0.1081
096 0.0593 0.0926 0.1338 0.1955 0.2279 0.1431 0.0599 0.0215 0.0645 0.2043 0.2796 0.3011 0.1789 0.1112
097 0.3764 0.6586 0.7088 0.7335 0.7731 0.6677 0.1134 0.4800 0.6800 0.7200 0.7200 0.7600 0.6873 0.0776
098 0.3493 0.4589 0.6727 0.8030 0.9000 0.6401 0.1840 0.4000 0.4000 0.6000 0.7500 0.8000 0.5636 0.1748
099 0.0058 0.1463 0.2670 0.2926 0.3892 0.2292 0.1101 0.0357 0.1964 0.3214 0.3571 0.5357 0.2890 0.1292
110 0.0962 0.6131 0.6642 0.6723 0.7401 0.6065 0.1747 0.1429 0.5714 0.5714 0.5714 0.7143 0.5455 0.1402
111 0.0005 0.0028 0.0046 0.0107 0.0221 0.0080 0.0074 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000
112 0.1176 0.2779 0.3909 0.4245 0.4772 0.3469 0.1098 0.0000 0.2812 0.3750 0.4688 0.5000 0.3523 0.1460
113 0.0887 0.2207 0.2367 0.2781 0.3446 0.2423 0.0695 0.1139 0.2690 0.2785 0.3323 0.3861 0.2854 0.0756
114 0.0140 0.0552 0.0703 0.4795 0.7827 0.2354 0.2677 0.0000 0.0222 0.0667 0.4667 0.6889 0.2101 0.2558
115 0.1421 0.1669 0.2118 0.2920 0.2982 0.2234 0.0605 0.2105 0.2105 0.2632 0.3158 0.3158 0.2584 0.0497
116 0.1102 0.1457 0.2405 0.2732 0.3746 0.2171 0.0807 0.0000 0.1429 0.1429 0.2500 0.2857 0.1558 0.1001
117 0.0772 0.2087 0.2366 0.2435 0.3043 0.2252 0.0574 0.1250 0.2857 0.3214 0.3705 0.3750 0.3052 0.0760
118 0.2577 0.3287 0.4015 0.4280 0.4496 0.3710 0.0680 0.2000 0.2000 0.4000 0.4000 0.4000 0.3273 0.1009
119 0.6469 0.7111 0.7333 0.7501 0.8063 0.7302 0.0401 0.5619 0.6667 0.6667 0.6976 0.7238 0.6667 0.0463
130 0.6425 0.6873 0.7150 0.7605 0.8421 0.7240 0.0566 0.5385 0.5385 0.6154 0.6731 0.6923 0.6084 0.0639
37
AH-ROBUST-CLEF2006 Track Overview Results and Graphs AH-ROBUST-MONO-ES-CLEF2006
Average Precision R-Precision
Topic Minimum 1st Q. Median 3rd Q. Maximum Mean Std Minimum 1st Q. Median 3rd Q. Maximum Mean Std
131 0.3116 0.3297 0.3540 0.3661 0.3716 0.3472 0.0216 0.3545 0.3636 0.3727 0.3818 0.4182 0.3777 0.0183
132 0.5553 0.6048 0.6431 0.6469 0.6512 0.6213 0.0350 0.5000 0.5096 0.5385 0.5769 0.5769 0.5455 0.0336
133 0.0000 0.0056 0.0121 0.1959 0.4640 0.1091 0.1523 0.0000 0.0000 0.0526 0.2632 0.4211 0.1244 0.1584
134 0.0729 0.1572 0.2015 0.8172 0.9167 0.4058 0.3450 0.0000 0.0000 0.0000 0.6875 0.7500 0.2727 0.3438
135 0.3284 0.5170 0.5512 0.6773 0.7541 0.5732 0.1177 0.4423 0.5497 0.5641 0.6394 0.6923 0.5857 0.0718
136 0.4119 0.4289 0.4678 0.5418 0.5714 0.4841 0.0570 0.3750 0.5000 0.6250 0.7500 0.7500 0.5909 0.1489
137 0.1139 0.1621 0.1840 0.2158 0.2526 0.1852 0.0389 0.2000 0.2000 0.2000 0.2750 0.3000 0.2273 0.0467
138 0.0444 0.1023 0.2993 0.7422 1.0000 0.4313 0.3501 0.0000 0.0000 0.1429 0.8214 1.0000 0.3896 0.4290
139 0.0703 0.0878 0.1163 0.1191 0.1257 0.1056 0.0202 0.0115 0.0374 0.1034 0.1135 0.1437 0.0852 0.0477
140 0.0331 0.0709 0.1096 0.1179 0.1506 0.0994 0.0343 0.1092 0.1261 0.1849 0.1849 0.2353 0.1673 0.0404
141 0.8106 0.8495 0.8753 0.9227 0.9757 0.8821 0.0506 0.7692 0.7692 0.8462 0.8462 0.9231 0.8252 0.0605
142 0.7334 0.7350 0.8847 0.9054 0.9256 0.8352 0.0855 0.6786 0.7143 0.8214 0.8214 0.8571 0.7727 0.0644
143 0.2950 0.3619 0.4493 0.5074 0.6338 0.4431 0.0989 0.3403 0.4236 0.4375 0.5417 0.5625 0.4710 0.0724
144 0.1474 0.4261 0.5076 0.5619 0.6169 0.4785 0.1284 0.1818 0.5114 0.5909 0.6364 0.7273 0.5620 0.1441
145 0.3414 0.6272 0.6364 0.6493 0.7251 0.6177 0.0962 0.3191 0.5319 0.5532 0.6064 0.6809 0.5493 0.0907
146 0.0570 0.1044 0.1484 0.5582 0.5625 0.3089 0.2337 0.0000 0.0000 0.0000 0.5000 0.5000 0.2273 0.2611
147 0.0018 0.0916 0.1429 0.1670 0.2077 0.1210 0.0684 0.0000 0.1111 0.1111 0.2222 0.2222 0.1313 0.0834
148 0.1241 0.1330 0.1403 0.1495 0.3222 0.1564 0.0557 0.0909 0.0909 0.0909 0.0909 0.3636 0.1157 0.0822
149 0.5347 0.6761 0.7331 0.8996 0.9343 0.7537 0.1344 0.5385 0.5577 0.6923 0.8077 0.8462 0.6853 0.1214
160 0.2778 0.4500 0.5455 0.6321 0.7000 0.5303 0.1328 0.0000 0.5000 0.5000 0.5000 0.5000 0.4545 0.1508
161 0.7100 0.8025 1.0000 1.0000 1.0000 0.9006 0.1228 0.8000 0.8000 1.0000 1.0000 1.0000 0.9091 0.1044
162 0.6614 0.7665 0.8513 0.8832 0.8932 0.8224 0.0806 0.6434 0.7403 0.7752 0.7907 0.8217 0.7569 0.0608
163 0.4036 0.5898 0.6460 0.6504 0.7232 0.6155 0.0805 0.3000 0.5417 0.6000 0.6667 0.6667 0.5848 0.1079
164 0.0173 0.2047 0.2939 0.3486 0.3656 0.2573 0.1187 0.0278 0.2986 0.3889 0.3889 0.4167 0.3131 0.1300
165 0.5661 0.6538 0.6996 0.7343 0.8645 0.7010 0.0822 0.4444 0.4722 0.6667 0.6667 0.7778 0.6061 0.1151
166 0.0041 0.0181 0.0420 0.0677 0.0722 0.0410 0.0248 0.0000 0.0000 0.0000 0.0000 0.1429 0.0130 0.0431
167 0.4117 0.4402 0.4473 0.5026 0.5888 0.4702 0.0509 0.3000 0.4000 0.4000 0.4750 0.5000 0.4182 0.0603
168 0.0450 0.0993 0.1024 0.1215 0.1702 0.1045 0.0344 0.0000 0.1026 0.1282 0.1538 0.2051 0.1166 0.0600
169 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000
170 0.4754 0.7383 0.8588 0.8597 0.9429 0.7912 0.1541 0.4000 0.6000 0.8000 0.8000 0.8000 0.6909 0.1640
171 0.0875 0.1713 0.3454 0.4265 0.5184 0.3117 0.1530 0.0000 0.2500 0.2500 0.5000 0.5000 0.3182 0.1617
172 0.0407 0.0877 0.1370 0.2275 0.3661 0.1634 0.1019 0.0769 0.0962 0.1538 0.3846 0.4615 0.2168 0.1607
173 0.3202 0.6040 0.6613 0.6693 0.7462 0.6248 0.1101 0.3333 0.5000 0.5000 0.6250 0.6667 0.5303 0.1005
174 0.5260 0.7967 0.8469 0.9030 0.9030 0.8269 0.1100 0.4545 0.8182 0.8182 0.8182 0.8182 0.7851 0.1096
175 1.0000 1.0000 1.0000 1.0000 1.0000 1.0000 0.0000 1.0000 1.0000 1.0000 1.0000 1.0000 1.0000 0.0000
176 0.9245 0.9408 0.9594 0.9761 0.9862 0.9584 0.0203 0.7662 0.8701 0.8961 0.9286 0.9610 0.8890 0.0533
177 0.1129 0.2564 0.2889 0.3129 0.3443 0.2702 0.0663 0.1739 0.2609 0.3043 0.3043 0.3478 0.2806 0.0528
178 0.2292 0.2459 0.3270 0.4014 0.4833 0.3331 0.0941 0.3523 0.3551 0.4205 0.4830 0.5227 0.4236 0.0701
179 0.0697 0.3162 0.5713 0.6481 0.8006 0.4986 0.2316 0.0796 0.3274 0.5752 0.6726 0.7611 0.5052 0.2199
190 0.0240 0.0251 0.0356 0.0448 0.0467 0.0351 0.0093 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000
191 0.1429 0.1667 0.2000 0.2500 1.0000 0.2972 0.2534 0.0000 0.0000 0.0000 0.0000 1.0000 0.0909 0.3015
192 0.1048 0.5200 0.7028 0.7499 0.8857 0.6201 0.2135 0.1591 0.5227 0.6818 0.7045 0.7955 0.6054 0.1764
193 0.0808 0.1375 0.1422 0.1868 0.2232 0.1560 0.0438 0.1176 0.1765 0.1765 0.2279 0.2941 0.1952 0.0547
194 0.0042 0.6615 0.6782 0.7219 0.8433 0.6238 0.2294 0.0000 0.6562 0.7500 0.7500 0.8750 0.6591 0.2378
195 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000
196 0.6279 0.7091 0.7171 0.7326 0.7598 0.7163 0.0332 0.6000 0.6667 0.7333 0.7333 0.7333 0.6970 0.0458
197 0.4407 0.5831 0.6482 0.6980 0.7135 0.6283 0.0894 0.4958 0.5885 0.6208 0.6396 0.6583 0.6042 0.0511
198 0.1980 0.2574 0.3560 0.4242 0.5661 0.3544 0.1107 0.1250 0.1250 0.2500 0.5000 0.6250 0.3295 0.1958
199 0.5314 0.7074 0.7208 0.8345 0.8580 0.7466 0.1058 0.5529 0.6353 0.6588 0.8000 0.8471 0.7102 0.0997
200 0.5915 0.7186 0.8185 0.8665 0.8693 0.7874 0.0897 0.5319 0.6330 0.7234 0.7660 0.7872 0.7002 0.0804
ALL 0.3893 0.4080 0.4401 0.4552 0.4584 0.4314 0.0252 0.3882 0.4026 0.4296 0.4415 0.4477 0.4231 0.0216
38
39
AH-ROBUST-CLEF2006 Track Overview Results and Graphs AH-ROBUST-MONO-FR-CLEF2006
Ad−Hoc Robust Monolingual French track Top 5 Participants − Interpolated Recall vs Average Precision
100%
unine [UniNEfrr1; MAP 47.57%; Not Pooled]
hummingbird [humFR06Rtde; MAP 45.43%; Not Pooled]
90% reina [reinaFRtdtest; MAP 44.58%; Not Pooled]
dcu [dcudescfr12075; MAP 41.08%; Not Pooled]
colesir [CoLesIRfrTst; MAP 39.51%; Not Pooled]
80%
70%
Average Precision
60%
50%
40%
30%
20%
10%
0%
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
Interpolated Recall
Ad−Hoc Robust Monolingual French track Top 5 Participants − Comparison to Median Mean Average Precision by Topic (Topics 041to 085)
1
unine [UniNEfrr1; MAP 47.57%; Not Pooled]
hummingbird [humFR06Rtde; MAP 45.43%; Not Pooled]
0.5 reina [reinaFRtdtest; MAP 44.58%; Not Pooled]
Difference
dcu [dcudescfr12075; MAP 41.08%; Not Pooled]
0 colesir [CoLesIRfrTst; MAP 39.51%; Not Pooled]
−0.5
−1
041 042 043 044 045 046 047 048 049 060 061 062 063 064 065 066 067 068 069 080 081 082 083 084 085
Topic Identifier
Ad−Hoc Robust Monolingual French track Top 5 Participants − Comparison to Median Mean Average Precision by Topic (Topics 086to 130)
1
unine [UniNEfrr1; MAP 47.57%; Not Pooled]
hummingbird [humFR06Rtde; MAP 45.43%; Not Pooled]
0.5 reina [reinaFRtdtest; MAP 44.58%; Not Pooled]
Difference
dcu [dcudescfr12075; MAP 41.08%; Not Pooled]
0 colesir [CoLesIRfrTst; MAP 39.51%; Not Pooled]
−0.5
−1
086 087 088 089 090 091 092 093 094 095 096 097 098 099 110 111 112 113 114 115 116 117 118 119 130
Topic Identifier
Ad−Hoc Robust Monolingual French track Top 5 Participants − Comparison to Median Mean Average Precision by Topic (Topics 131to 165)
1
unine [UniNEfrr1; MAP 47.57%; Not Pooled]
hummingbird [humFR06Rtde; MAP 45.43%; Not Pooled]
0.5 reina [reinaFRtdtest; MAP 44.58%; Not Pooled]
Difference
dcu [dcudescfr12075; MAP 41.08%; Not Pooled]
0 colesir [CoLesIRfrTst; MAP 39.51%; Not Pooled]
−0.5
−1
131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 160 161 162 163 164 165
Topic Identifier
Ad−Hoc Robust Monolingual French track Top 5 Participants − Comparison to Median Mean Average Precision by Topic (Topics 166to 200)
1
unine [UniNEfrr1; MAP 47.57%; Not Pooled]
hummingbird [humFR06Rtde; MAP 45.43%; Not Pooled]
0.5 reina [reinaFRtdtest; MAP 44.58%; Not Pooled]
Difference
dcu [dcudescfr12075; MAP 41.08%; Not Pooled]
0 colesir [CoLesIRfrTst; MAP 39.51%; Not Pooled]
−0.5
−1
166 167 168 169 170 171 172 173 174 175 176 177 178 179 190 191 192 193 194 195 196 197 198 199 200
Topic Identifier
40
AH-ROBUST-CLEF2006 Track Overview Results and Graphs AH-ROBUST-MONO-FR-CLEF2006
Ad−Hoc Robust Monolingual French track − Box Plot of the Topics
UniNEfrr1 [MAP 47.57%; Not Pooled]
dcunarrfr1005 [MAP 45.87%; Not Pooled]
humFR06Rtde [MAP 45.43%; Not Pooled]
reinaFRtdtest [MAP 44.58%; Not Pooled]
reinaFRttest [MAP 44.15%; Not Pooled]
humFR06Rtd [MAP 42.74%; Not Pooled]
frFSfrR3S [MAP 42.54%; Not Pooled]
UniNEfrr3 [MAP 41.92%; Not Pooled]
Experiments
fryfrRFS3456 [MAP 41.54%; Not Pooled]
frx011frRFW3FS3 [MAP 41.41%; Not Pooled]
dcudescfr12075 [MAP 41.08%; Not Pooled]
dcudescfr12075 [MAP 41.08%; Not Pooled]
UniNEfrr2 [MAP 40.00%; Not Pooled]
CoLesIRfrTst [MAP 39.51%; Not Pooled]
frFSfrR2S [MAP 38.49%; Not Pooled]
uhifrmo2 [MAP 29.85%; Not Pooled]
uhifrmo1 [MAP 28.50%; Not Pooled]
uhifrmo1 [MAP 28.50%; Not Pooled]
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
Mean Average Precision
41
AH-ROBUST-CLEF2006 Track Overview Results and Graphs AH-ROBUST-MONO-FR-CLEF2006
Ad−Hoc Robust Monolingual French track − Tukey T test with "top group" highlighted
UniNEfrr1
dcunarrfr1005
humFR06Rtde
reinaFRtdtest
reinaFRttest
humFR06Rtd
frFSfrR3S
fryfrRFS3456
Experiments
frx011frRFW3FS3
UniNEfrr3
dcudescfr12075
dcudescfr12075
UniNEfrr2
CoLesIRfrTst
frFSfrR2S
uhifrmo2
uhifrmo1
uhifrmo1
0.45 0.5 0.55 0.6 0.65 0.7 0.75 0.8 0.85 0.9
arcsin(sqrt(Mean average precision))
42
AH-ROBUST-CLEF2006 Track Overview Results and Graphs AH-ROBUST-MONO-FR-CLEF2006
Ad−Hoc Robust Monolingual French track Top 5 Participants − Retrieved documents vs Precision
100%
unine [UniNEfrr1; R−Prec 45.45%; Not Pooled]
hummingbird [humFR06Rtde; R−Prec 42.77%; Not Pooled]
90% reina [reinaFRtdtest; R−Prec 44.23%; Not Pooled]
dcu [dcudescfr12075; R−Prec 37.93%; Not Pooled]
colesir [CoLesIRfrTst; R−Prec 37.56%; Not Pooled]
80%
70%
60%
R−Precision
50%
40%
30%
20%
10%
0%
5 10 15 20 30 100 200 500 1000
Retrieved Documents (logarithmic scale)
Ad−Hoc Robust Monolingual French track Top 5 Participants − Comparison to Median R−Precision by Topic (Topics 041 to 085)
1
unine [UniNEfrr1; R−Prec 45.45%; Not Pooled]
hummingbird [humFR06Rtde; R−Prec 42.77%; Not Pooled]
0.5 reina [reinaFRtdtest; R−Prec 44.23%; Not Pooled]
Difference
dcu [dcudescfr12075; R−Prec 37.93%; Not Pooled]
0 colesir [CoLesIRfrTst; R−Prec 37.56%; Not Pooled]
−0.5
−1
041 042 043 044 045 046 047 048 049 060 061 062 063 064 065 066 067 068 069 080 081 082 083 084 085
Topic Identifier
Ad−Hoc Robust Monolingual French track Top 5 Participants − Comparison to Median R−Precision by Topic (Topics 086 to 130)
1
unine [UniNEfrr1; R−Prec 45.45%; Not Pooled]
hummingbird [humFR06Rtde; R−Prec 42.77%; Not Pooled]
0.5 reina [reinaFRtdtest; R−Prec 44.23%; Not Pooled]
Difference
dcu [dcudescfr12075; R−Prec 37.93%; Not Pooled]
0 colesir [CoLesIRfrTst; R−Prec 37.56%; Not Pooled]
−0.5
−1
086 087 088 089 090 091 092 093 094 095 096 097 098 099 110 111 112 113 114 115 116 117 118 119 130
Topic Identifier
Ad−Hoc Robust Monolingual French track Top 5 Participants − Comparison to Median R−Precision by Topic (Topics 131 to 165)
1
unine [UniNEfrr1; R−Prec 45.45%; Not Pooled]
hummingbird [humFR06Rtde; R−Prec 42.77%; Not Pooled]
0.5 reina [reinaFRtdtest; R−Prec 44.23%; Not Pooled]
Difference
dcu [dcudescfr12075; R−Prec 37.93%; Not Pooled]
0 colesir [CoLesIRfrTst; R−Prec 37.56%; Not Pooled]
−0.5
−1
131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 160 161 162 163 164 165
Topic Identifier
Ad−Hoc Robust Monolingual French track Top 5 Participants − Comparison to Median R−Precision by Topic (Topics 166 to 200)
1
unine [UniNEfrr1; R−Prec 45.45%; Not Pooled]
hummingbird [humFR06Rtde; R−Prec 42.77%; Not Pooled]
0.5 reina [reinaFRtdtest; R−Prec 44.23%; Not Pooled]
Difference
dcu [dcudescfr12075; R−Prec 37.93%; Not Pooled]
0 colesir [CoLesIRfrTst; R−Prec 37.56%; Not Pooled]
−0.5
−1
166 167 168 169 170 171 172 173 174 175 176 177 178 179 190 191 192 193 194 195 196 197 198 199 200
Topic Identifier
43
AH-ROBUST-CLEF2006 Track Overview Results and Graphs AH-ROBUST-MONO-FR-CLEF2006
Ad−Hoc Robust Monolingual French track − Box Plot of the Topics
UniNEfrr1 [R−Prec 45.45%; Not Pooled]
dcunarrfr1005 [R−Prec 44.55%; Not Pooled]
reinaFRtdtest [R−Prec 44.23%; Not Pooled]
reinaFRttest [R−Prec 43.52%; Not Pooled]
humFR06Rtde [R−Prec 42.77%; Not Pooled]
humFR06Rtd [R−Prec 41.97%; Not Pooled]
frFSfrR3S [R−Prec 40.96%; Not Pooled]
fryfrRFS3456 [R−Prec 40.06%; Not Pooled]
Experiments
frx011frRFW3FS3 [R−Prec 39.02%; Not Pooled]
UniNEfrr3 [R−Prec 38.77%; Not Pooled]
UniNEfrr2 [R−Prec 38.34%; Not Pooled]
dcudescfr12075 [R−Prec 37.93%; Not Pooled]
dcudescfr12075 [R−Prec 37.93%; Not Pooled]
frFSfrR2S [R−Prec 37.59%; Not Pooled]
CoLesIRfrTst [R−Prec 37.56%; Not Pooled]
uhifrmo2 [R−Prec 28.45%; Not Pooled]
uhifrmo1 [R−Prec 27.68%; Not Pooled]
uhifrmo1 [R−Prec 27.68%; Not Pooled]
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
R−Precision
44
AH-ROBUST-CLEF2006 Track Overview Results and Graphs AH-ROBUST-MONO-FR-CLEF2006
Ad−Hoc Robust Monolingual French track − Tukey T test with "top group" highlighted
UniNEfrr1
dcunarrfr1005
reinaFRtdtest
reinaFRttest
humFR06Rtde
humFR06Rtd
frFSfrR3S
fryfrRFS3456
Experiments
frx011frRFW3FS3
frFSfrR2S
CoLesIRfrTst
UniNEfrr2
UniNEfrr3
dcudescfr12075
dcudescfr12075
uhifrmo2
uhifrmo1
uhifrmo1
0.4 0.45 0.5 0.55 0.6 0.65 0.7 0.75 0.8
arcsin(sqrt(R Precision))
45
AH-ROBUST-CLEF2006 Track Overview Results and Graphs AH-ROBUST-MONO-FR-CLEF2006
Average Precision R-Precision
Topic Minimum 1st Q. Median 3rd Q. Maximum Mean Std Minimum 1st Q. Median 3rd Q. Maximum Mean Std
041 0.0580 1.0000 1.0000 1.0000 1.0000 0.8254 0.3569 0.0000 1.0000 1.0000 1.0000 1.0000 0.8056 0.3888
042 0.1976 0.4501 0.4620 0.5215 0.6829 0.4632 0.1199 0.2895 0.4474 0.5132 0.5263 0.6447 0.4846 0.0916
043 0.0081 0.3333 1.0000 1.0000 1.0000 0.6779 0.4339 0.0000 0.0000 1.0000 1.0000 1.0000 0.6111 0.5016
044 0.0946 0.1848 0.2163 0.2754 0.3585 0.2203 0.0644 0.0000 0.0000 0.1667 0.1667 0.3333 0.1204 0.1253
045 0.2409 0.2735 0.4547 0.5622 0.6329 0.4223 0.1380 0.2340 0.3191 0.4362 0.5106 0.5319 0.4090 0.0994
046 0.0125 0.0503 0.0532 0.0850 0.1367 0.0652 0.0308 0.0000 0.0000 0.0714 0.1429 0.1429 0.0714 0.0693
047 0.1274 0.1654 0.2022 0.2322 0.2922 0.1962 0.0451 0.0962 0.2115 0.2404 0.2885 0.3269 0.2393 0.0537
048 0.0032 0.0087 0.0140 0.0163 0.0787 0.0167 0.0162 0.0000 0.0000 0.0000 0.0000 0.1429 0.0079 0.0337
049 0.0046 0.1313 0.1815 0.2165 0.2504 0.1600 0.0811 0.0000 0.1667 0.1667 0.2500 0.3333 0.1852 0.1127
060 0.0000 0.1748 0.2692 0.3208 0.3318 0.2328 0.1057 0.0000 0.2667 0.3056 0.4000 0.4444 0.3012 0.1282
061 0.0331 0.0510 0.0824 0.1920 0.5254 0.1322 0.1200 0.0000 0.0833 0.0833 0.0833 0.4167 0.1019 0.1013
062 0.1914 0.3606 0.4147 0.5611 0.6926 0.4578 0.1434 0.2500 0.3750 0.5000 0.6250 0.6250 0.5000 0.1286
063 0.2228 0.2871 0.4001 0.5140 0.5805 0.3965 0.1211 0.1765 0.2941 0.4706 0.5294 0.5882 0.4085 0.1419
064 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000
065 0.0026 0.2000 0.2717 0.2801 0.3792 0.2250 0.1112 0.0000 0.1538 0.2308 0.2308 0.4615 0.2009 0.1154
066 0.8805 0.8946 0.9039 0.9216 0.9471 0.9091 0.0216 0.7647 0.8235 0.8431 0.8627 0.8824 0.8410 0.0307
067 0.3229 0.4221 0.4529 0.5046 0.5820 0.4551 0.0733 0.3684 0.4211 0.4737 0.5263 0.6316 0.4825 0.0750
068 0.1455 0.3403 0.4458 0.4980 0.5581 0.4032 0.1195 0.2000 0.4000 0.4600 0.5200 0.6000 0.4444 0.1163
069 0.2707 0.3437 0.3828 0.4524 0.5525 0.4024 0.0883 0.3750 0.4375 0.4375 0.5000 0.5000 0.4410 0.0453
080 0.0561 0.1375 0.1468 0.1581 0.1634 0.1320 0.0366 0.0323 0.0968 0.1452 0.1935 0.2258 0.1434 0.0577
081 0.4648 0.6002 0.6399 0.6606 0.6770 0.6228 0.0538 0.4706 0.5294 0.5588 0.5882 0.7059 0.5621 0.0503
082 0.0604 0.1660 0.3539 0.7773 0.9228 0.4616 0.3189 0.0625 0.2500 0.4688 0.7500 0.8750 0.4965 0.2774
083 0.1838 0.4167 0.6250 1.0000 1.0000 0.6883 0.2965 0.0000 0.0000 0.5000 1.0000 1.0000 0.5556 0.4162
084 0.1111 0.5000 0.5000 1.0000 1.0000 0.6728 0.3159 0.0000 0.0000 0.0000 1.0000 1.0000 0.4444 0.5113
085 0.1801 0.2447 0.2788 0.3089 0.3777 0.2752 0.0502 0.1667 0.2667 0.3000 0.3333 0.4333 0.2889 0.0686
086 0.0836 0.1533 0.1851 0.2040 0.2240 0.1723 0.0424 0.0833 0.1667 0.1667 0.1944 0.2778 0.1713 0.0439
087 0.0025 0.4038 0.4263 0.4549 0.5473 0.3653 0.1716 0.0000 0.3684 0.4737 0.5263 0.5789 0.4006 0.1929
088 0.4790 0.5956 0.6392 0.6766 0.7612 0.6340 0.0627 0.4839 0.5484 0.5484 0.5806 0.6774 0.5645 0.0522
089 0.3740 0.4593 0.4985 0.6022 0.7769 0.5249 0.1033 0.3913 0.5217 0.5217 0.5217 0.6522 0.5242 0.0796
090 0.0035 0.0896 0.1040 0.1175 0.1334 0.0907 0.0423 0.0000 0.1176 0.1176 0.1765 0.2353 0.1176 0.0669
091 0.0008 0.0358 0.0895 0.1996 0.2308 0.1018 0.0812 0.0000 0.0000 0.0769 0.2308 0.3846 0.1154 0.1158
092 0.1802 0.2107 0.2598 0.3015 0.3962 0.2692 0.0708 0.2500 0.2708 0.3125 0.3333 0.4583 0.3113 0.0487
093 0.8500 0.9667 0.9833 1.0000 1.0000 0.9667 0.0482 0.8000 0.8000 0.9000 1.0000 1.0000 0.9000 0.1029
094 0.0611 0.5538 0.5822 0.6513 0.7656 0.5378 0.1894 0.0000 0.5000 0.5625 0.6250 0.7500 0.5174 0.2043
095 0.1137 0.2449 0.2567 0.2796 0.3050 0.2475 0.0536 0.1864 0.3220 0.3475 0.3785 0.4068 0.3387 0.0563
096 0.1238 0.2702 0.3099 0.3732 0.3966 0.2985 0.0890 0.0000 0.2143 0.2857 0.3571 0.5000 0.2540 0.1369
097 0.1969 0.2510 0.3263 0.4200 0.4706 0.3326 0.0901 0.1429 0.2857 0.2857 0.4286 0.5714 0.3333 0.1470
098 0.0366 0.6047 0.8921 0.9460 0.9658 0.7004 0.3396 0.1111 0.5556 0.7778 0.8889 0.8889 0.6543 0.2772
099 0.0475 0.1297 0.2352 0.2464 0.3162 0.1995 0.0814 0.0588 0.1765 0.3529 0.4118 0.4706 0.3170 0.1443
110 0.5324 0.5633 0.5967 0.6506 0.6882 0.6025 0.0485 0.6154 0.6154 0.6667 0.6923 0.7179 0.6595 0.0361
111 0.0074 0.0266 0.0546 0.1763 0.2374 0.0924 0.0790 0.0000 0.0000 0.0000 0.1667 0.3333 0.0648 0.1013
112 0.3217 0.3835 0.4166 0.4423 0.4837 0.4113 0.0478 0.3333 0.3333 0.4000 0.4667 0.5333 0.4148 0.0629
113 0.0196 0.0280 0.0636 0.0728 0.1544 0.0647 0.0434 0.0000 0.0000 0.0000 0.1111 0.2222 0.0494 0.0871
114 0.1340 0.3333 0.3545 0.4616 0.5998 0.3771 0.1210 0.1667 0.3667 0.3667 0.4667 0.6000 0.3926 0.1020
115 0.3354 0.5264 0.5451 0.6275 0.6927 0.5396 0.1135 0.2000 0.4000 0.4000 0.6000 0.8000 0.5000 0.1572
116 0.2661 0.3242 0.3488 0.4375 0.5391 0.3691 0.0789 0.3000 0.3000 0.4000 0.4000 0.5000 0.3778 0.0647
117 0.0054 0.0474 0.2534 0.3098 0.3539 0.1923 0.1350 0.0000 0.1071 0.2500 0.2857 0.3214 0.1905 0.1213
118 0.0166 0.0323 0.0771 0.0954 0.7137 0.1009 0.1559 0.0000 0.0000 0.0833 0.1667 0.6667 0.1111 0.1617
119 0.1046 0.7250 0.7643 0.8149 0.8642 0.6995 0.1953 0.1375 0.6875 0.7313 0.7750 0.8000 0.6681 0.1699
130 0.5658 0.5836 0.6222 0.6331 0.7196 0.6175 0.0388 0.5385 0.6154 0.6154 0.6923 0.6923 0.6325 0.0563
46
AH-ROBUST-CLEF2006 Track Overview Results and Graphs AH-ROBUST-MONO-FR-CLEF2006
Average Precision R-Precision
Topic Minimum 1st Q. Median 3rd Q. Maximum Mean Std Minimum 1st Q. Median 3rd Q. Maximum Mean Std
131 0.0959 0.1866 0.3157 0.3538 0.3870 0.2660 0.1017 0.1341 0.2317 0.3659 0.4024 0.4268 0.3164 0.1016
132 0.3811 0.4713 0.4933 0.5437 0.6667 0.5084 0.0780 0.2500 0.2500 0.2500 0.5000 0.7500 0.3472 0.1745
133 0.0127 0.0177 0.0951 0.2008 0.3302 0.1250 0.1005 0.0000 0.0625 0.1875 0.2188 0.3750 0.1701 0.1099
134 0.1641 0.8333 0.9167 0.9167 1.0000 0.7735 0.2722 0.0000 0.6667 0.6667 0.6667 1.0000 0.6111 0.2061
135 0.0112 0.1168 0.3529 0.6290 0.8347 0.4012 0.2832 0.0000 0.1923 0.4135 0.6538 0.7885 0.4199 0.2586
136 0.5833 1.0000 1.0000 1.0000 1.0000 0.9769 0.0982 0.5000 1.0000 1.0000 1.0000 1.0000 0.9722 0.1179
137 0.0006 0.0597 0.1380 0.1959 0.3714 0.1409 0.1099 0.0000 0.0000 0.3333 0.3333 0.3333 0.2037 0.1672
138 0.1070 0.2216 0.4014 0.5693 0.7043 0.3993 0.2114 0.1765 0.2549 0.4216 0.5490 0.6275 0.4139 0.1545
139 0.0449 0.0895 0.1604 0.1939 0.2572 0.1471 0.0666 0.0222 0.0889 0.1333 0.2667 0.3111 0.1691 0.0962
140 0.1623 0.5126 0.5822 0.6157 0.7264 0.5133 0.1824 0.2500 0.5391 0.5703 0.6250 0.6562 0.5195 0.1456
141 0.1000 0.5000 1.0000 1.0000 1.0000 0.7694 0.3465 0.0000 0.0000 1.0000 1.0000 1.0000 0.6667 0.4851
142 0.5862 0.9856 0.9897 1.0000 1.0000 0.9058 0.1700 0.6154 0.9231 0.9231 1.0000 1.0000 0.8761 0.1472
143 0.0750 0.3079 0.3453 0.4673 0.6010 0.3552 0.1490 0.1429 0.2500 0.3214 0.5000 0.6071 0.3472 0.1428
144 0.2939 0.6389 0.9048 1.0000 1.0000 0.8046 0.2325 0.3333 0.6667 0.8333 1.0000 1.0000 0.7593 0.2755
145 0.1307 0.2839 0.4145 0.4588 0.5110 0.3708 0.1144 0.1429 0.2143 0.3571 0.4286 0.5000 0.3492 0.1146
146 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000
147 0.0270 0.3689 0.4199 0.4808 0.6751 0.3790 0.1756 0.0000 0.3750 0.5000 0.5000 0.6250 0.4097 0.1705
148 0.0207 0.0278 0.0296 0.0433 0.0927 0.0392 0.0197 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000
149 0.4693 0.7862 0.8762 0.9267 1.0000 0.8348 0.1284 0.6000 0.6000 0.6000 0.8000 1.0000 0.7000 0.1237
160 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000
161 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000
162 0.0724 0.3780 0.7028 0.7998 0.8786 0.5619 0.3028 0.0400 0.4000 0.7000 0.7600 0.8000 0.5267 0.3022
163 0.1571 0.2428 0.3774 0.4533 0.5406 0.3642 0.1234 0.0909 0.2727 0.3636 0.4545 0.6364 0.3737 0.1425
164 0.1972 0.2507 0.4065 0.4819 0.5976 0.3835 0.1262 0.1685 0.3034 0.4607 0.4944 0.5730 0.4101 0.1187
165 0.2667 0.3250 0.4167 0.4500 0.4500 0.3918 0.0606 0.0000 0.0000 0.5000 0.5000 0.5000 0.3333 0.2425
166 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000
167 0.1016 0.5000 0.6429 0.8333 1.0000 0.6201 0.2706 0.0000 0.5000 0.5000 0.5000 1.0000 0.4444 0.2357
168 0.0924 0.1721 0.1985 0.2686 0.3940 0.2222 0.0942 0.0769 0.1538 0.2308 0.3077 0.3846 0.2479 0.1041
169 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000
170 0.0057 0.0500 0.1458 0.3333 0.5000 0.1786 0.1464 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000
171 0.1537 0.1873 0.2956 0.3626 0.4400 0.2842 0.0997 0.0000 0.3333 0.3333 0.3333 0.3333 0.3148 0.0786
172 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000
173 1.0000 1.0000 1.0000 1.0000 1.0000 1.0000 0.0000 1.0000 1.0000 1.0000 1.0000 1.0000 1.0000 0.0000
174 0.9683 1.0000 1.0000 1.0000 1.0000 0.9972 0.0084 0.8571 1.0000 1.0000 1.0000 1.0000 0.9841 0.0462
175 0.5000 1.0000 1.0000 1.0000 1.0000 0.9444 0.1617 0.0000 1.0000 1.0000 1.0000 1.0000 0.8889 0.3234
176 0.8281 0.9258 0.9656 0.9769 0.9950 0.9463 0.0491 0.7895 0.8421 0.8947 0.8947 0.9474 0.8801 0.0352
177 0.1677 0.3154 0.3789 0.5597 0.6698 0.3978 0.1427 0.0000 0.2857 0.2857 0.4286 0.5714 0.3254 0.1454
178 0.0625 0.1296 0.2974 0.4167 0.6711 0.3100 0.1992 0.0000 0.0000 0.3333 0.3333 0.6667 0.2593 0.2695
179 0.0000 0.2204 0.2877 0.3419 0.4638 0.2586 0.1355 0.0000 0.1250 0.2500 0.2500 0.5000 0.2153 0.1343
190 0.0982 0.2241 0.4106 0.4801 0.6522 0.3650 0.1644 0.1176 0.2941 0.4118 0.4706 0.6471 0.3856 0.1386
191 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000
192 0.3748 0.5055 0.6696 0.8390 0.9284 0.6662 0.1787 0.4737 0.5263 0.6579 0.7368 0.8421 0.6433 0.1218
193 0.0496 0.3678 0.4203 0.4844 0.5171 0.3539 0.1712 0.0244 0.2927 0.3780 0.4390 0.4634 0.3144 0.1591
194 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000
195 0.0122 0.1396 0.2892 0.3867 0.5100 0.2672 0.1575 0.0000 0.1579 0.3158 0.3684 0.4737 0.2690 0.1465
196 1.0000 1.0000 1.0000 1.0000 1.0000 1.0000 0.0000 1.0000 1.0000 1.0000 1.0000 1.0000 1.0000 0.0000
197 0.8053 0.8358 0.8552 0.8689 0.8970 0.8530 0.0289 0.7176 0.7710 0.7901 0.8092 0.8397 0.7901 0.0309
198 0.0673 0.2266 0.3837 0.4715 0.6952 0.3623 0.1823 0.0000 0.3333 0.3333 0.3333 0.6667 0.2963 0.1943
199 0.5159 0.6233 0.7435 0.7920 0.8389 0.7131 0.0965 0.4848 0.5152 0.6212 0.6667 0.7273 0.6077 0.0822
200 0.0005 0.0166 0.0443 0.1389 0.2761 0.0900 0.0958 0.0000 0.0000 0.0278 0.2222 0.3333 0.0926 0.1220
ALL 0.2850 0.3951 0.4147 0.4415 0.4757 0.4026 0.0570 0.2768 0.3759 0.3890 0.4277 0.4545 0.3858 0.0553
47
48
AH-ROBUST-CLEF2006 Track Overview Results and Graphs AH-ROBUST-MONO-IT-CLEF2006
Ad−Hoc Robust Monolingual Italian track Top 5 Participants − Interpolated Recall vs Average Precision
100%
hummingbird [humIT06Rtde; MAP 41.94%; Not Pooled]
reina [reinaITtdtest; MAP 38.45%; Not Pooled]
90% dcu [dcudescit1005; MAP 37.73%; Not Pooled]
daedalus [itFSitR2S; MAP 35.11%; Not Pooled]
colesir [CoLesIRitTst; MAP 32.23%; Not Pooled]
80%
70%
Average Precision
60%
50%
40%
30%
20%
10%
0%
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
Interpolated Recall
Ad−Hoc Robust Monolingual Italian track Top 5 Participants − Comparison to Median Mean Average Precision by Topic (Topics 041to 085)
1
hummingbird [humIT06Rtde; MAP 41.94%; Not Pooled]
reina [reinaITtdtest; MAP 38.45%; Not Pooled]
0.5 dcu [dcudescit1005; MAP 37.73%; Not Pooled]
Difference
daedalus [itFSitR2S; MAP 35.11%; Not Pooled]
0 colesir [CoLesIRitTst; MAP 32.23%; Not Pooled]
−0.5
−1
041 042 043 044 045 046 047 048 049 060 061 062 063 064 065 066 067 068 069 080 081 082 083 084 085
Topic Identifier
Ad−Hoc Robust Monolingual Italian track Top 5 Participants − Comparison to Median Mean Average Precision by Topic (Topics 086to 130)
1
hummingbird [humIT06Rtde; MAP 41.94%; Not Pooled]
reina [reinaITtdtest; MAP 38.45%; Not Pooled]
0.5 dcu [dcudescit1005; MAP 37.73%; Not Pooled]
Difference
daedalus [itFSitR2S; MAP 35.11%; Not Pooled]
0 colesir [CoLesIRitTst; MAP 32.23%; Not Pooled]
−0.5
−1
086 087 088 089 090 091 092 093 094 095 096 097 098 099 110 111 112 113 114 115 116 117 118 119 130
Topic Identifier
Ad−Hoc Robust Monolingual Italian track Top 5 Participants − Comparison to Median Mean Average Precision by Topic (Topics 131to 165)
1
hummingbird [humIT06Rtde; MAP 41.94%; Not Pooled]
reina [reinaITtdtest; MAP 38.45%; Not Pooled]
0.5 dcu [dcudescit1005; MAP 37.73%; Not Pooled]
Difference
daedalus [itFSitR2S; MAP 35.11%; Not Pooled]
0 colesir [CoLesIRitTst; MAP 32.23%; Not Pooled]
−0.5
−1
131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 160 161 162 163 164 165
Topic Identifier
Ad−Hoc Robust Monolingual Italian track Top 5 Participants − Comparison to Median Mean Average Precision by Topic (Topics 166to 200)
1
hummingbird [humIT06Rtde; MAP 41.94%; Not Pooled]
reina [reinaITtdtest; MAP 38.45%; Not Pooled]
0.5 dcu [dcudescit1005; MAP 37.73%; Not Pooled]
Difference
daedalus [itFSitR2S; MAP 35.11%; Not Pooled]
0 colesir [CoLesIRitTst; MAP 32.23%; Not Pooled]
−0.5
−1
166 167 168 169 170 171 172 173 174 175 176 177 178 179 190 191 192 193 194 195 196 197 198 199 200
Topic Identifier
49
AH-ROBUST-CLEF2006 Track Overview Results and Graphs AH-ROBUST-MONO-IT-CLEF2006
Ad−Hoc Robust Monolingual Italian track − Box Plot of the Topics
humIT06Rtde [MAP 41.94%; Not Pooled]
dcu1500narrit1005.qy1 [MAP 40.94%; Not Pooled]
reinaITtdtest [MAP 38.45%; Not Pooled]
itFSitR6S [MAP 37.82%; Not Pooled]
ityitRFS3456 [MAP 37.80%; Not Pooled]
Experiments
dcudescit1005 [MAP 37.73%; Not Pooled]
itFSitR4S [MAP 37.61%; Not Pooled]
humIT06Rtd [MAP 36.83%; Not Pooled]
reinaITttest [MAP 35.62%; Not Pooled]
itFSitR2S [MAP 35.11%; Not Pooled]
CoLesIRitTst [MAP 32.23%; Not Pooled]
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
Mean Average Precision
50
AH-ROBUST-CLEF2006 Track Overview Results and Graphs AH-ROBUST-MONO-IT-CLEF2006
Ad−Hoc Robust Monolingual Italian track − Tukey T test with "top group" highlighted
humIT06Rtde
dcu1500narrit1005.qy1
reinaITtdtest
itFSitR6S
ityitRFS3456
Experiments
itFSitR4S
dcudescit1005
humIT06Rtd
itFSitR2S
reinaITttest
CoLesIRitTst
0.52 0.54 0.56 0.58 0.6 0.62 0.64 0.66 0.68 0.7 0.72
arcsin(sqrt(Mean average precision))
51
AH-ROBUST-CLEF2006 Track Overview Results and Graphs AH-ROBUST-MONO-IT-CLEF2006
Ad−Hoc Robust Monolingual Italian track Top 5 Participants − Retrieved documents vs Precision
100%
hummingbird [humIT06Rtde; R−Prec 39.70%; Not Pooled]
reina [reinaITtdtest; R−Prec 37.15%; Not Pooled]
90% dcu [dcudescit1005; R−Prec 35.21%; Not Pooled]
daedalus [itFSitR2S; R−Prec 35.72%; Not Pooled]
colesir [CoLesIRitTst; R−Prec 32.06%; Not Pooled]
80%
70%
60%
R−Precision
50%
40%
30%
20%
10%
0%
5 10 15 20 30 100 200 500 1000
Retrieved Documents (logarithmic scale)
Ad−Hoc Robust Monolingual Italian track Top 5 Participants − Comparison to Median R−Precision by Topic (Topics 041 to 085)
1
hummingbird [humIT06Rtde; R−Prec 39.70%; Not Pooled]
reina [reinaITtdtest; R−Prec 37.15%; Not Pooled]
0.5 dcu [dcudescit1005; R−Prec 35.21%; Not Pooled]
Difference
daedalus [itFSitR2S; R−Prec 35.72%; Not Pooled]
0 colesir [CoLesIRitTst; R−Prec 32.06%; Not Pooled]
−0.5
−1
041 042 043 044 045 046 047 048 049 060 061 062 063 064 065 066 067 068 069 080 081 082 083 084 085
Topic Identifier
Ad−Hoc Robust Monolingual Italian track Top 5 Participants − Comparison to Median R−Precision by Topic (Topics 086 to 130)
1
hummingbird [humIT06Rtde; R−Prec 39.70%; Not Pooled]
reina [reinaITtdtest; R−Prec 37.15%; Not Pooled]
0.5 dcu [dcudescit1005; R−Prec 35.21%; Not Pooled]
Difference
daedalus [itFSitR2S; R−Prec 35.72%; Not Pooled]
0 colesir [CoLesIRitTst; R−Prec 32.06%; Not Pooled]
−0.5
−1
086 087 088 089 090 091 092 093 094 095 096 097 098 099 110 111 112 113 114 115 116 117 118 119 130
Topic Identifier
Ad−Hoc Robust Monolingual Italian track Top 5 Participants − Comparison to Median R−Precision by Topic (Topics 131 to 165)
1
hummingbird [humIT06Rtde; R−Prec 39.70%; Not Pooled]
reina [reinaITtdtest; R−Prec 37.15%; Not Pooled]
0.5 dcu [dcudescit1005; R−Prec 35.21%; Not Pooled]
Difference
daedalus [itFSitR2S; R−Prec 35.72%; Not Pooled]
0 colesir [CoLesIRitTst; R−Prec 32.06%; Not Pooled]
−0.5
−1
131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 160 161 162 163 164 165
Topic Identifier
Ad−Hoc Robust Monolingual Italian track Top 5 Participants − Comparison to Median R−Precision by Topic (Topics 166 to 200)
1
hummingbird [humIT06Rtde; R−Prec 39.70%; Not Pooled]
reina [reinaITtdtest; R−Prec 37.15%; Not Pooled]
0.5 dcu [dcudescit1005; R−Prec 35.21%; Not Pooled]
Difference
daedalus [itFSitR2S; R−Prec 35.72%; Not Pooled]
0 colesir [CoLesIRitTst; R−Prec 32.06%; Not Pooled]
−0.5
−1
166 167 168 169 170 171 172 173 174 175 176 177 178 179 190 191 192 193 194 195 196 197 198 199 200
Topic Identifier
52
AH-ROBUST-CLEF2006 Track Overview Results and Graphs AH-ROBUST-MONO-IT-CLEF2006
Ad−Hoc Robust Monolingual Italian track − Box Plot of the Topics
humIT06Rtde [R−Prec 39.70%; Not Pooled]
dcu1500narrit1005.qy1 [R−Prec 38.86%; Not Pooled]
reinaITtdtest [R−Prec 37.15%; Not Pooled]
ityitRFS3456 [R−Prec 36.89%; Not Pooled]
itFSitR6S [R−Prec 36.49%; Not Pooled]
Experiments
humIT06Rtd [R−Prec 36.34%; Not Pooled]
itFSitR4S [R−Prec 36.03%; Not Pooled]
itFSitR2S [R−Prec 35.72%; Not Pooled]
reinaITttest [R−Prec 35.63%; Not Pooled]
dcudescit1005 [R−Prec 35.21%; Not Pooled]
CoLesIRitTst [R−Prec 32.06%; Not Pooled]
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
R−Precision
53
AH-ROBUST-CLEF2006 Track Overview Results and Graphs AH-ROBUST-MONO-IT-CLEF2006
Ad−Hoc Robust Monolingual Italian track − Tukey T test with "top group" highlighted
humIT06Rtde
dcu1500narrit1005.qy1
reinaITtdtest
ityitRFS3456
itFSitR6S
Experiments
itFSitR4S
itFSitR2S
humIT06Rtd
reinaITttest
dcudescit1005
CoLesIRitTst
0.5 0.52 0.54 0.56 0.58 0.6 0.62 0.64 0.66 0.68
arcsin(sqrt(R Precision))
54
AH-ROBUST-CLEF2006 Track Overview Results and Graphs AH-ROBUST-MONO-IT-CLEF2006
Average Precision R-Precision
Topic Minimum 1st Q. Median 3rd Q. Maximum Mean Std Minimum 1st Q. Median 3rd Q. Maximum Mean Std
041 0.3799 0.6306 0.6491 0.7008 0.7497 0.6365 0.0990 0.4286 0.5714 0.5714 0.6786 0.7143 0.5974 0.0861
042 0.0499 0.2858 0.3170 0.4134 0.4902 0.3198 0.1229 0.0361 0.3313 0.3494 0.4518 0.5301 0.3625 0.1317
043 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000
044 0.0893 0.1341 0.1465 0.2130 0.4167 0.1806 0.0921 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000
045 0.1225 0.2052 0.2178 0.2418 0.3219 0.2245 0.0496 0.1190 0.1905 0.2143 0.2560 0.3333 0.2273 0.0633
046 0.0817 0.1054 0.1172 0.1247 0.1784 0.1177 0.0260 0.0625 0.0781 0.1250 0.1875 0.2500 0.1364 0.0614
047 0.0254 0.0308 0.0579 0.0944 0.1096 0.0614 0.0319 0.0000 0.0385 0.0769 0.1154 0.1154 0.0734 0.0470
048 0.0305 0.0791 0.1104 0.1210 0.1297 0.0942 0.0359 0.0800 0.0900 0.1600 0.1600 0.2000 0.1345 0.0411
049 0.1615 0.2366 0.2545 0.2900 0.3231 0.2578 0.0471 0.2414 0.3103 0.3103 0.3448 0.3793 0.3135 0.0421
060 0.1639 0.1957 0.2081 0.2625 0.5457 0.2664 0.1289 0.2535 0.2817 0.2958 0.3873 0.5634 0.3444 0.1129
061 0.2468 0.2730 0.2867 0.3423 0.4598 0.3126 0.0613 0.3125 0.3125 0.3750 0.4375 0.5000 0.3920 0.0744
062 0.3754 0.4310 0.4689 0.5298 0.6416 0.4818 0.0782 0.3846 0.3846 0.3846 0.4615 0.5385 0.4196 0.0529
063 0.4537 0.5459 0.5696 0.6563 0.7645 0.5920 0.0894 0.3750 0.4062 0.5000 0.5000 0.6250 0.4886 0.0876
064 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000
065 0.1185 0.1675 0.2333 0.3069 0.3154 0.2357 0.0742 0.1333 0.2667 0.3333 0.3833 0.4667 0.3091 0.0956
066 0.6651 0.7046 0.7928 0.8052 0.8066 0.7616 0.0567 0.5946 0.7027 0.7027 0.7297 0.7568 0.7052 0.0475
067 0.2275 0.3608 0.3867 0.4246 0.4565 0.3737 0.0687 0.2941 0.3529 0.3824 0.4412 0.4412 0.3850 0.0534
068 0.0692 0.2237 0.3657 0.4185 0.5321 0.3249 0.1377 0.1667 0.2583 0.3667 0.4000 0.5000 0.3455 0.1057
069 0.0245 0.0407 0.0456 0.0698 0.0988 0.0545 0.0218 0.0000 0.0000 0.0000 0.1667 0.1667 0.0758 0.0870
080 0.1760 0.3208 0.3908 0.4134 0.4202 0.3610 0.0772 0.2262 0.3393 0.4286 0.4405 0.4524 0.3918 0.0717
081 0.2484 0.3320 0.3404 0.3737 0.4557 0.3493 0.0533 0.3333 0.3704 0.3704 0.4074 0.4815 0.3906 0.0449
082 0.5626 0.5895 0.6312 0.7766 0.9275 0.6908 0.1203 0.4848 0.5227 0.5758 0.6591 0.8182 0.5950 0.0989
083 0.1833 0.5308 0.5956 0.7167 0.7917 0.5895 0.1683 0.0000 0.3333 0.3333 0.6667 0.6667 0.4545 0.2247
084 0.0000 0.0967 0.1188 0.1268 0.1473 0.1049 0.0390 0.0000 0.0000 0.0000 0.0000 0.2500 0.0455 0.1011
085 0.3703 0.5166 0.5411 0.6003 0.6116 0.5397 0.0685 0.3973 0.4932 0.5205 0.5582 0.5753 0.5156 0.0532
086 0.1217 0.1646 0.1847 0.1921 0.2554 0.1794 0.0336 0.1935 0.1935 0.2258 0.2581 0.2903 0.2258 0.0353
087 0.3920 0.4100 0.4429 0.5119 0.6580 0.4781 0.0879 0.3000 0.4000 0.5000 0.5000 0.6000 0.4636 0.0924
088 0.3474 0.3741 0.4173 0.4539 0.4819 0.4137 0.0454 0.3333 0.3333 0.4000 0.4000 0.4667 0.3879 0.0501
089 0.7196 0.7418 0.7444 0.7938 0.9021 0.7714 0.0521 0.7308 0.7692 0.7692 0.7981 0.8462 0.7762 0.0336
090 0.0064 0.0348 0.0861 0.1168 0.1198 0.0738 0.0431 0.0000 0.0000 0.0000 0.3333 0.3333 0.1212 0.1682
091 0.0010 0.0530 0.0632 0.4127 0.5103 0.2060 0.2040 0.0000 0.0000 0.0000 0.4375 0.5000 0.1818 0.2261
092 0.1457 0.2152 0.2328 0.2656 0.2966 0.2350 0.0434 0.1944 0.2222 0.2222 0.2500 0.3333 0.2374 0.0400
093 0.5895 0.6436 0.6571 0.7836 0.8709 0.7034 0.0939 0.6667 0.7778 0.7778 0.7778 0.7778 0.7677 0.0335
094 0.3356 0.3568 0.4223 0.4724 0.5498 0.4243 0.0708 0.4000 0.4125 0.4500 0.4875 0.5500 0.4545 0.0472
095 0.0159 0.0275 0.0322 0.0413 0.0802 0.0384 0.0189 0.0286 0.0571 0.0714 0.1107 0.1571 0.0818 0.0414
096 0.1189 0.1975 0.2445 0.2683 0.3609 0.2365 0.0727 0.0000 0.2500 0.2500 0.3750 0.5000 0.2614 0.1526
097 0.3704 0.5668 0.6326 0.6543 0.6951 0.5964 0.1043 0.5000 0.5000 0.5000 0.5000 0.7500 0.5227 0.0754
098 0.1394 0.1406 0.4854 0.7076 0.8189 0.4504 0.2921 0.1667 0.1875 0.5000 0.5833 0.7500 0.4242 0.2250
099 0.2350 0.3876 0.4260 0.4781 0.6068 0.4262 0.1168 0.2821 0.3654 0.4359 0.4615 0.5641 0.4172 0.0812
110 0.5470 0.7793 0.8310 0.8426 0.8646 0.7947 0.0891 0.5652 0.6957 0.7391 0.7717 0.7826 0.7194 0.0655
111 0.0640 0.1325 0.1453 0.1600 0.3351 0.1554 0.0659 0.0000 0.2727 0.2727 0.2727 0.4545 0.2479 0.1156
112 0.3909 0.3937 0.4608 0.5385 0.6314 0.4813 0.0921 0.3333 0.3333 0.4167 0.5000 0.5833 0.4318 0.0899
113 0.0108 0.0727 0.1012 0.1272 0.1685 0.0987 0.0420 0.0000 0.0556 0.1667 0.1667 0.2778 0.1313 0.0834
114 0.1271 0.3014 0.3808 0.4893 0.5133 0.3741 0.1178 0.0769 0.3846 0.3846 0.3846 0.4615 0.3636 0.1037
115 0.2789 0.3951 0.4162 0.4223 0.5164 0.4137 0.0611 0.4074 0.4444 0.4815 0.5185 0.5185 0.4747 0.0432
116 0.2616 0.3042 0.3599 0.3965 0.4185 0.3517 0.0521 0.3000 0.3000 0.4000 0.4000 0.4000 0.3545 0.0522
117 0.1498 0.2155 0.3549 0.3986 0.4599 0.3210 0.1057 0.1765 0.2500 0.3529 0.5000 0.5882 0.3690 0.1419
118 0.0347 0.1053 0.2171 0.2618 0.4333 0.1940 0.1200 0.0000 0.2500 0.2500 0.2500 0.2500 0.2045 0.1011
119 0.2926 0.5509 0.5985 0.6636 0.7653 0.5918 0.1229 0.3667 0.5000 0.6000 0.6458 0.7167 0.5667 0.1011
130 0.3003 0.3093 0.3609 0.3791 0.4451 0.3553 0.0451 0.3000 0.3500 0.4000 0.4000 0.4500 0.3818 0.0462
55
AH-ROBUST-CLEF2006 Track Overview Results and Graphs AH-ROBUST-MONO-IT-CLEF2006
Average Precision R-Precision
Topic Minimum 1st Q. Median 3rd Q. Maximum Mean Std Minimum 1st Q. Median 3rd Q. Maximum Mean Std
131 0.0211 0.0701 0.1138 0.2705 0.3288 0.1576 0.1118 0.0488 0.1280 0.1951 0.4268 0.4634 0.2506 0.1524
132 0.2578 0.4187 0.4542 0.5053 0.5812 0.4428 0.0992 0.3333 0.3333 0.3333 0.3333 0.6667 0.3636 0.1005
133 0.1978 0.3586 0.3816 0.4645 0.5993 0.3996 0.1036 0.3125 0.3750 0.4375 0.4844 0.5625 0.4318 0.0813
134 0.4939 0.7731 0.7857 0.8797 0.9667 0.7886 0.1257 0.4000 0.6000 0.6000 0.8000 0.8000 0.6545 0.1572
135 0.0625 0.0665 0.4543 0.6199 0.7413 0.3906 0.2727 0.1558 0.1558 0.4935 0.6006 0.6883 0.4144 0.2148
136 0.5381 0.5837 0.6298 0.6748 0.6965 0.6284 0.0527 0.4167 0.5000 0.5833 0.5833 0.6667 0.5530 0.0770
137 0.2003 0.2961 0.3571 0.4029 0.6042 0.3594 0.1046 0.2500 0.2500 0.5000 0.5000 0.7500 0.4318 0.1617
138 0.1119 0.3182 0.4434 0.6784 0.7867 0.4818 0.2096 0.1538 0.3077 0.4615 0.5962 0.6923 0.4266 0.1766
139 0.0468 0.0566 0.0730 0.1145 0.1178 0.0822 0.0278 0.0000 0.0000 0.0000 0.0000 0.0625 0.0057 0.0188
140 0.1245 0.1456 0.1988 0.3123 0.4821 0.2344 0.1131 0.1600 0.2050 0.2200 0.3550 0.4000 0.2636 0.0876
141 0.3299 0.5547 0.7560 0.9315 1.0000 0.7227 0.2169 0.2857 0.4643 0.5714 0.8571 1.0000 0.6364 0.2246
142 0.7582 0.7582 0.9032 0.9722 0.9952 0.8770 0.1050 0.7857 0.7857 0.8571 0.9286 0.9286 0.8636 0.0674
143 0.1790 0.2672 0.3091 0.3308 0.4327 0.3004 0.0687 0.1923 0.2788 0.3462 0.3846 0.4615 0.3357 0.0771
144 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000
145 1.0000 1.0000 1.0000 1.0000 1.0000 1.0000 0.0000 1.0000 1.0000 1.0000 1.0000 1.0000 1.0000 0.0000
146 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000
147 0.0190 0.0865 0.2392 0.2638 0.3358 0.1936 0.1060 0.0000 0.2000 0.2000 0.4000 0.4000 0.2727 0.1348
148 0.0612 0.0938 0.1027 0.1556 0.5400 0.1650 0.1405 0.0000 0.0000 0.0000 0.0000 0.5000 0.0909 0.2023
149 0.2158 0.2842 0.3278 0.3631 0.4788 0.3378 0.0809 0.0000 0.0000 0.3333 0.3333 0.3333 0.1818 0.1741
160 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000
161 0.0000 0.0000 0.0000 1.0000 1.0000 0.3648 0.5036 0.0000 0.0000 0.0000 1.0000 1.0000 0.3636 0.5045
162 0.3973 0.5465 0.6043 0.6572 0.7080 0.5956 0.0878 0.4286 0.5000 0.5714 0.6429 0.7143 0.5779 0.0982
163 0.1647 0.4656 0.4989 0.5777 0.6835 0.4837 0.1634 0.1739 0.4891 0.5652 0.6087 0.6957 0.5059 0.1642
164 0.0466 0.1757 0.2333 0.2892 0.3576 0.2285 0.0853 0.0968 0.2339 0.2903 0.3226 0.4194 0.2757 0.0882
165 0.5000 1.0000 1.0000 1.0000 1.0000 0.9545 0.1508 0.0000 1.0000 1.0000 1.0000 1.0000 0.9091 0.3015
166 0.0015 0.0019 0.0036 0.0051 0.0075 0.0038 0.0020 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000
167 0.5833 1.0000 1.0000 1.0000 1.0000 0.9470 0.1306 0.5000 1.0000 1.0000 1.0000 1.0000 0.9091 0.2023
168 0.0759 0.2364 0.2692 0.3158 0.4604 0.2728 0.1060 0.0938 0.2578 0.3125 0.3984 0.4688 0.3153 0.1041
169 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000
170 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000
171 0.2344 0.3597 0.4253 0.4600 0.4923 0.3980 0.0818 0.2857 0.4286 0.4286 0.4286 0.4286 0.4026 0.0578
172 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000
173 0.2917 0.5208 0.6429 0.7500 0.8333 0.6251 0.1787 0.0000 0.5000 0.5000 0.5000 0.5000 0.4091 0.2023
174 0.6912 0.7460 0.7528 0.7758 0.8887 0.7654 0.0487 0.6250 0.6250 0.6250 0.6250 0.7500 0.6477 0.0506
175 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000
176 0.5063 0.5212 0.5369 0.6221 0.6480 0.5655 0.0560 0.4375 0.5625 0.5625 0.6094 0.6250 0.5625 0.0559
177 0.2665 0.4565 0.4781 0.5610 0.6458 0.4925 0.1065 0.3333 0.3750 0.5000 0.5000 0.5000 0.4545 0.0778
178 0.0791 0.2232 0.2952 0.3113 0.4219 0.2766 0.0916 0.0000 0.2500 0.2500 0.4375 0.5000 0.2955 0.1508
179 0.2199 0.5476 0.5792 0.6833 0.7701 0.5868 0.1438 0.2750 0.5500 0.5500 0.5938 0.6500 0.5500 0.0975
190 0.3425 0.3630 0.4639 0.4897 0.6329 0.4479 0.0899 0.3333 0.4167 0.4167 0.5000 0.5833 0.4545 0.0683
191 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000
192 0.2493 0.3636 0.4667 0.5038 0.8093 0.4648 0.1438 0.2903 0.4274 0.4839 0.4839 0.7097 0.4633 0.1033
193 0.1708 0.2032 0.2059 0.4229 0.5860 0.3152 0.1496 0.0000 0.0000 0.2500 0.4375 0.5000 0.2045 0.2185
194 0.0021 0.2988 0.3941 0.5041 0.5225 0.3764 0.1548 0.0000 0.2500 0.3077 0.3846 0.5385 0.3217 0.1411
195 0.4532 0.5344 0.5730 0.6406 0.7728 0.5911 0.0978 0.4167 0.4375 0.5833 0.5833 0.6667 0.5379 0.0863
196 1.0000 1.0000 1.0000 1.0000 1.0000 1.0000 0.0000 1.0000 1.0000 1.0000 1.0000 1.0000 1.0000 0.0000
197 0.4573 0.6353 0.6652 0.7078 0.7135 0.6521 0.0733 0.4455 0.6000 0.6273 0.6523 0.6727 0.6124 0.0613
198 0.0666 0.1299 0.2681 0.3020 0.3955 0.2319 0.1061 0.0000 0.1429 0.2857 0.2857 0.2857 0.2208 0.0982
199 0.4255 0.4395 0.4693 0.5682 0.7222 0.5164 0.0985 0.3611 0.3681 0.5000 0.5278 0.6111 0.4672 0.0886
200 0.1389 0.3719 0.4547 0.5843 0.6109 0.4476 0.1449 0.1714 0.4071 0.4286 0.5643 0.6571 0.4597 0.1349
ALL 0.3223 0.3592 0.3773 0.3829 0.4194 0.3746 0.0265 0.3206 0.3565 0.3634 0.3708 0.3970 0.3637 0.0198
56
57
AH-ROBUST-CLEF2006 Track Overview Results and Graphs AH-ROBUST-MONO-NL-CLEF2006
Ad−Hoc Robust Monolingual Dutch track Top 5 Participants − Interpolated Recall vs Average Precision
100%
hummingbird [humNL06Rtde; MAP 51.06%; Not Pooled]
daedalus [nlFSnlR2S; MAP 42.39%; Not Pooled]
90%
colesir [CoLesIRnlTst; MAP 41.60%; Not Pooled]
80%
70%
Average Precision
60%
50%
40%
30%
20%
10%
0%
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
Interpolated Recall
Ad−Hoc Robust Monolingual Dutch track Top 5 Participants − Comparison to Median Mean Average Precision by Topic (Topics 041ot 085)
1
hummingbird [humNL06Rtde; MAP 51.06%; Not Pooled]
daedalus [nlFSnlR2S; MAP 42.39%; Not Pooled]
0.5 colesir [CoLesIRnlTst; MAP 41.60%; Not Pooled]
Difference
0
−0.5
−1
041 042 043 044 045 046 047 048 049 060 061 062 063 064 065 066 067 068 069 080 081 082 083 084 085
Topic Identifier
Ad−Hoc Robust Monolingual Dutch track Top 5 Participants − Comparison to Median Mean Average Precision by Topic (Topics 086ot 130)
1
hummingbird [humNL06Rtde; MAP 51.06%; Not Pooled]
daedalus [nlFSnlR2S; MAP 42.39%; Not Pooled]
0.5 colesir [CoLesIRnlTst; MAP 41.60%; Not Pooled]
Difference
0
−0.5
−1
086 087 088 089 090 091 092 093 094 095 096 097 098 099 110 111 112 113 114 115 116 117 118 119 130
Topic Identifier
Ad−Hoc Robust Monolingual Dutch track Top 5 Participants − Comparison to Median Mean Average Precision by Topic (Topics 131ot 165)
1
hummingbird [humNL06Rtde; MAP 51.06%; Not Pooled]
daedalus [nlFSnlR2S; MAP 42.39%; Not Pooled]
0.5 colesir [CoLesIRnlTst; MAP 41.60%; Not Pooled]
Difference
0
−0.5
−1
131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 160 161 162 163 164 165
Topic Identifier
Ad−Hoc Robust Monolingual Dutch track Top 5 Participants − Comparison to Median Mean Average Precision by Topic (Topics 166ot 200)
1
hummingbird [humNL06Rtde; MAP 51.06%; Not Pooled]
daedalus [nlFSnlR2S; MAP 42.39%; Not Pooled]
0.5 colesir [CoLesIRnlTst; MAP 41.60%; Not Pooled]
Difference
0
−0.5
−1
166 167 168 169 170 171 172 173 174 175 176 177 178 179 190 191 192 193 194 195 196 197 198 199 200
Topic Identifier
58
AH-ROBUST-CLEF2006 Track Overview Results and Graphs AH-ROBUST-MONO-NL-CLEF2006
Ad−Hoc Robust Monolingual Dutch track − Box Plot of the Topics
humNL06Rtde [MAP 51.06%; Not Pooled]
humNL06Rtd [MAP 46.47%; Not Pooled]
nlFSnlR4S [MAP 45.02%; Not Pooled]
Experiments
nlynlRFS3456 [MAP 44.99%; Not Pooled]
nlx011nlRFW4FS4 [MAP 44.88%; Not Pooled]
nlFSnlR2S [MAP 42.39%; Not Pooled]
CoLesIRnlTst [MAP 41.60%; Not Pooled]
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
Mean Average Precision
59
AH-ROBUST-CLEF2006 Track Overview Results and Graphs AH-ROBUST-MONO-NL-CLEF2006
Ad−Hoc Robust Monolingual Dutch track − Tukey T test with "top group" highlighted
humNL06Rtde
humNL06Rtd
nlFSnlR4S
Experiments
nlynlRFS3456
nlx011nlRFW4FS4
CoLesIRnlTst
nlFSnlR2S
0.65 0.7 0.75 0.8 0.85 0.9
arcsin(sqrt(Mean average precision))
60
AH-ROBUST-CLEF2006 Track Overview Results and Graphs AH-ROBUST-MONO-NL-CLEF2006
Ad−Hoc Robust Monolingual Dutch track Top 5 Participants − Retrieved documents vs Precision
100%
hummingbird [humNL06Rtde; R−Prec 48.92%; Not Pooled]
daedalus [nlFSnlR2S; R−Prec 39.51%; Not Pooled]
90%
colesir [CoLesIRnlTst; R−Prec 39.85%; Not Pooled]
80%
70%
60%
R−Precision
50%
40%
30%
20%
10%
0%
5 10 15 20 30 100 200 500 1000
Retrieved Documents (logarithmic scale)
Ad−Hoc Robust Monolingual Dutch track Top 5 Participants − Comparison to Median R−Precision by Topic (Topics 041 to 085)
1
hummingbird [humNL06Rtde; R−Prec 48.92%; Not Pooled]
daedalus [nlFSnlR2S; R−Prec 39.51%; Not Pooled]
0.5 colesir [CoLesIRnlTst; R−Prec 39.85%; Not Pooled]
Difference
0
−0.5
−1
041 042 043 044 045 046 047 048 049 060 061 062 063 064 065 066 067 068 069 080 081 082 083 084 085
Topic Identifier
Ad−Hoc Robust Monolingual Dutch track Top 5 Participants − Comparison to Median R−Precision by Topic (Topics 086 to 130)
1
hummingbird [humNL06Rtde; R−Prec 48.92%; Not Pooled]
daedalus [nlFSnlR2S; R−Prec 39.51%; Not Pooled]
0.5 colesir [CoLesIRnlTst; R−Prec 39.85%; Not Pooled]
Difference
0
−0.5
−1
086 087 088 089 090 091 092 093 094 095 096 097 098 099 110 111 112 113 114 115 116 117 118 119 130
Topic Identifier
Ad−Hoc Robust Monolingual Dutch track Top 5 Participants − Comparison to Median R−Precision by Topic (Topics 131 to 165)
1
hummingbird [humNL06Rtde; R−Prec 48.92%; Not Pooled]
daedalus [nlFSnlR2S; R−Prec 39.51%; Not Pooled]
0.5 colesir [CoLesIRnlTst; R−Prec 39.85%; Not Pooled]
Difference
0
−0.5
−1
131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 160 161 162 163 164 165
Topic Identifier
Ad−Hoc Robust Monolingual Dutch track Top 5 Participants − Comparison to Median R−Precision by Topic (Topics 166 to 200)
1
hummingbird [humNL06Rtde; R−Prec 48.92%; Not Pooled]
daedalus [nlFSnlR2S; R−Prec 39.51%; Not Pooled]
0.5 colesir [CoLesIRnlTst; R−Prec 39.85%; Not Pooled]
Difference
0
−0.5
−1
166 167 168 169 170 171 172 173 174 175 176 177 178 179 190 191 192 193 194 195 196 197 198 199 200
Topic Identifier
61
AH-ROBUST-CLEF2006 Track Overview Results and Graphs AH-ROBUST-MONO-NL-CLEF2006
Ad−Hoc Robust Monolingual Dutch track − Box Plot of the Topics
humNL06Rtde [R−Prec 48.92%; Not Pooled]
humNL06Rtd [R−Prec 44.15%; Not Pooled]
nlFSnlR4S [R−Prec 42.89%; Not Pooled]
Experiments
nlynlRFS3456 [R−Prec 42.85%; Not Pooled]
nlx011nlRFW4FS4 [R−Prec 42.85%; Not Pooled]
CoLesIRnlTst [R−Prec 39.85%; Not Pooled]
nlFSnlR2S [R−Prec 39.51%; Not Pooled]
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
R−Precision
62
AH-ROBUST-CLEF2006 Track Overview Results and Graphs AH-ROBUST-MONO-NL-CLEF2006
Ad−Hoc Robust Monolingual Dutch track − Tukey T test with "top group" highlighted
humNL06Rtde
humNL06Rtd
nlFSnlR4S
Experiments
nlynlRFS3456
nlx011nlRFW4FS4
CoLesIRnlTst
nlFSnlR2S
0.62 0.64 0.66 0.68 0.7 0.72 0.74 0.76 0.78 0.8
arcsin(sqrt(R Precision))
63
AH-ROBUST-CLEF2006 Track Overview Results and Graphs AH-ROBUST-MONO-NL-CLEF2006
Average Precision R-Precision
Topic Minimum 1st Q. Median 3rd Q. Maximum Mean Std Minimum 1st Q. Median 3rd Q. Maximum Mean Std
041 0.1507 0.1818 0.1826 0.2244 0.2707 0.2018 0.0395 0.1739 0.2283 0.2609 0.2609 0.3043 0.2484 0.0414
042 0.1516 0.1683 0.1695 0.1753 0.1781 0.1691 0.0087 0.1250 0.1667 0.1667 0.2083 0.2083 0.1786 0.0315
043 0.0763 0.4497 0.4595 0.6027 0.7042 0.4653 0.2012 0.2500 0.2500 0.2500 0.4375 0.7500 0.3571 0.1967
044 0.2354 0.2576 0.2747 0.2862 0.2870 0.2696 0.0193 0.2000 0.2667 0.2667 0.2667 0.3333 0.2667 0.0385
045 0.3897 0.3952 0.3979 0.4081 0.4827 0.4104 0.0326 0.3500 0.3500 0.3500 0.4688 0.5250 0.4000 0.0777
046 0.1387 0.1396 0.1641 0.2281 0.2563 0.1789 0.0517 0.2500 0.2500 0.2500 0.3385 0.3542 0.2857 0.0492
047 0.2523 0.2669 0.3089 0.3249 0.3254 0.2956 0.0322 0.2581 0.3065 0.3226 0.3548 0.3548 0.3226 0.0361
048 0.1646 0.1752 0.2063 0.2527 0.3161 0.2170 0.0553 0.1636 0.1636 0.2545 0.2955 0.3455 0.2364 0.0750
049 0.0285 0.0286 0.0488 0.1346 0.1744 0.0763 0.0629 0.0000 0.0000 0.0714 0.1429 0.2143 0.0816 0.0868
060 0.0827 0.0831 0.1005 0.2259 0.3547 0.1546 0.1093 0.0769 0.1154 0.1154 0.2308 0.2692 0.1538 0.0801
061 0.3507 0.3729 0.3778 0.6075 0.6637 0.4764 0.1362 0.3478 0.3913 0.3913 0.5978 0.6522 0.4783 0.1255
062 0.5391 0.5549 0.5791 0.5915 0.5927 0.5742 0.0222 0.5000 0.5000 0.5000 0.5375 0.5500 0.5143 0.0244
063 0.6133 0.6616 0.6653 0.8307 0.8576 0.7306 0.1015 0.5333 0.5333 0.5333 0.7333 0.8000 0.6286 0.1208
064 0.0641 0.1570 0.2929 0.3164 0.3688 0.2481 0.1117 0.1250 0.1562 0.2500 0.3750 0.3750 0.2679 0.1125
065 0.0359 0.0589 0.0963 0.1453 0.1457 0.1010 0.0466 0.0580 0.1268 0.2754 0.3043 0.3043 0.2257 0.1071
066 0.6097 0.6646 0.6896 0.6914 0.7079 0.6747 0.0323 0.6071 0.6250 0.6786 0.7143 0.7143 0.6735 0.0480
067 0.0968 0.1231 0.1535 0.1719 0.2317 0.1531 0.0437 0.1500 0.2000 0.2500 0.2500 0.3000 0.2286 0.0488
068 0.4102 0.4758 0.5402 0.5435 0.6307 0.5221 0.0704 0.3043 0.4457 0.4783 0.5109 0.5652 0.4658 0.0822
069 0.0170 0.0669 0.0849 0.0861 0.1127 0.0747 0.0295 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000
080 0.6855 0.8029 0.8130 0.8367 0.8419 0.8021 0.0538 0.6989 0.7903 0.8172 0.8172 0.8280 0.7957 0.0448
081 0.5482 0.6374 0.6670 0.6672 0.8255 0.6649 0.0824 0.5926 0.6019 0.6296 0.6296 0.7037 0.6296 0.0370
082 0.0979 0.2181 0.2278 0.4086 0.5333 0.2918 0.1495 0.1000 0.2000 0.2000 0.3750 0.7000 0.3000 0.2000
083 0.0614 0.0684 0.0834 0.1206 0.2091 0.1022 0.0527 0.0000 0.0000 0.0000 0.0000 0.1667 0.0238 0.0630
084 0.1654 0.5060 0.5123 0.5162 0.5898 0.4737 0.1390 0.3000 0.4000 0.5000 0.5000 0.5000 0.4429 0.0787
085 0.5245 0.5785 0.5795 0.6291 0.6609 0.5957 0.0446 0.6111 0.6667 0.6667 0.7083 0.7222 0.6746 0.0383
086 0.4892 0.5212 0.5384 0.5429 0.6258 0.5418 0.0419 0.4600 0.4850 0.5600 0.5600 0.5800 0.5286 0.0474
087 0.6350 0.6616 0.7262 0.7263 0.7711 0.7050 0.0481 0.4615 0.4615 0.5385 0.5962 0.6154 0.5275 0.0692
088 0.0676 0.1740 0.2096 0.2446 0.2798 0.1991 0.0692 0.0000 0.2500 0.2500 0.5000 0.5000 0.3214 0.1890
089 0.8495 0.8630 0.9105 0.9112 0.9420 0.8935 0.0337 0.7674 0.7965 0.8605 0.8605 0.9070 0.8372 0.0484
090 0.0067 0.1981 0.2396 0.4083 0.4878 0.2636 0.1660 0.0000 0.2404 0.3846 0.4712 0.5385 0.3407 0.1864
091 0.0523 0.0566 0.0933 0.1402 0.1687 0.1013 0.0465 0.1111 0.1111 0.1111 0.1111 0.2222 0.1270 0.0420
092 0.3403 0.4258 0.4341 0.4392 0.4521 0.4225 0.0373 0.3947 0.4013 0.5000 0.5461 0.5526 0.4774 0.0720
093 0.7264 0.7609 0.7707 0.7816 0.7965 0.7686 0.0222 0.7500 0.7625 0.8000 0.8000 0.8500 0.7929 0.0345
094 0.7527 0.7588 0.7683 0.7720 0.7856 0.7677 0.0110 0.7368 0.7895 0.7895 0.7895 0.7895 0.7820 0.0199
095 0.5354 0.5652 0.6565 0.6712 0.6767 0.6242 0.0602 0.5249 0.5407 0.6146 0.6196 0.6279 0.5880 0.0449
096 0.3414 0.3635 0.4228 0.4495 0.4517 0.4070 0.0465 0.3684 0.3684 0.4211 0.4211 0.4211 0.3985 0.0281
097 0.4553 0.5791 0.6844 0.6910 0.6938 0.6299 0.0911 0.4375 0.5156 0.6875 0.7500 0.7500 0.6339 0.1322
098 0.8486 0.8819 0.8819 0.8819 0.8819 0.8772 0.0126 0.6667 0.6667 0.6667 0.6667 0.6667 0.6667 0.0000
099 0.1722 0.2000 0.2566 0.2595 0.2997 0.2360 0.0446 0.1667 0.1667 0.2500 0.3333 0.3333 0.2500 0.0833
110 1.0000 1.0000 1.0000 1.0000 1.0000 1.0000 0.0000 1.0000 1.0000 1.0000 1.0000 1.0000 1.0000 0.0000
111 0.0009 0.0012 0.0012 0.1780 0.2154 0.0733 0.0998 0.0000 0.0000 0.0000 0.2500 0.3571 0.1122 0.1535
112 0.4335 0.4831 0.5633 0.5825 0.5882 0.5364 0.0635 0.4000 0.5250 0.6000 0.6000 0.6000 0.5571 0.0787
113 0.0045 0.0049 0.0064 0.0064 0.0154 0.0071 0.0038 0.0152 0.0189 0.0303 0.0303 0.0606 0.0303 0.0152
114 0.2364 0.2685 0.4178 0.4257 0.5564 0.3784 0.1135 0.2745 0.3284 0.4314 0.4314 0.5098 0.3950 0.0804
115 0.1542 0.2812 0.3649 0.4089 0.4226 0.3338 0.0970 0.2308 0.4231 0.4615 0.4615 0.5000 0.4231 0.0888
116 0.4624 0.5404 0.6898 0.6954 0.8368 0.6466 0.1267 0.4828 0.6466 0.7241 0.7500 0.8276 0.6946 0.1116
117 0.0693 0.0882 0.1092 0.1209 0.1698 0.1103 0.0323 0.1111 0.1111 0.2222 0.2222 0.2222 0.1746 0.0594
118 0.1509 0.1569 0.1736 0.2048 0.2664 0.1855 0.0414 0.1818 0.1932 0.2273 0.2273 0.2727 0.2208 0.0314
119 0.4910 0.6375 0.6455 0.6528 0.7625 0.6393 0.0791 0.4516 0.5242 0.5806 0.5806 0.6452 0.5576 0.0610
130 0.3704 0.5884 0.7117 0.7196 0.7201 0.6315 0.1299 0.3333 0.4444 0.5556 0.5556 0.6667 0.5079 0.1084
64
AH-ROBUST-CLEF2006 Track Overview Results and Graphs AH-ROBUST-MONO-NL-CLEF2006
Average Precision R-Precision
Topic Minimum 1st Q. Median 3rd Q. Maximum Mean Std Minimum 1st Q. Median 3rd Q. Maximum Mean Std
131 0.1726 0.2145 0.2153 0.2885 0.3659 0.2492 0.0652 0.2250 0.2312 0.2500 0.3500 0.3500 0.2857 0.0610
132 0.5695 0.5772 0.6105 0.6135 0.7797 0.6223 0.0720 0.3846 0.4615 0.6154 0.6154 0.6154 0.5385 0.0993
133 0.0867 0.1827 0.2082 0.2097 0.4634 0.2211 0.1154 0.1071 0.1786 0.1786 0.1786 0.4643 0.2092 0.1156
134 1.0000 1.0000 1.0000 1.0000 1.0000 1.0000 0.0000 1.0000 1.0000 1.0000 1.0000 1.0000 1.0000 0.0000
135 0.6602 0.6783 0.7396 0.7447 0.8134 0.7263 0.0526 0.6452 0.6552 0.6694 0.6694 0.7419 0.6728 0.0319
136 0.9429 0.9667 0.9667 0.9917 1.0000 0.9728 0.0205 0.8000 0.8000 0.8000 0.9500 1.0000 0.8571 0.0976
137 0.0002 0.0212 0.0271 0.0383 0.0469 0.0280 0.0154 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000
138 0.0028 0.1450 0.2398 0.4099 0.4125 0.2582 0.1604 0.0000 0.1389 0.3333 0.3333 0.3333 0.2381 0.1350
139 0.0723 0.0785 0.0785 0.3180 0.4827 0.1934 0.1654 0.1522 0.1957 0.1957 0.3370 0.4348 0.2547 0.1055
140 0.7383 0.7502 0.7506 0.7831 0.8053 0.7633 0.0250 0.6736 0.6736 0.6736 0.6944 0.7153 0.6855 0.0164
141 0.3179 0.4279 0.6071 0.6071 0.6500 0.5270 0.1251 0.2500 0.3125 0.5000 0.5000 0.5000 0.4286 0.1220
142 0.7232 0.7232 0.7237 0.9958 0.9995 0.8397 0.1452 0.7273 0.7273 0.7273 0.9716 0.9773 0.8312 0.1298
143 0.7389 0.7392 0.7532 0.8270 0.8757 0.7829 0.0553 0.6897 0.6897 0.7241 0.7931 0.7931 0.7389 0.0521
144 1.0000 1.0000 1.0000 1.0000 1.0000 1.0000 0.0000 1.0000 1.0000 1.0000 1.0000 1.0000 1.0000 0.0000
145 0.1188 0.3373 0.3616 0.3644 0.4468 0.3337 0.1014 0.2143 0.2857 0.2857 0.3393 0.3571 0.2959 0.0493
146 0.2917 0.2961 0.3333 0.6167 0.7500 0.4347 0.2005 0.0000 0.0000 0.0000 0.3750 0.5000 0.1429 0.2440
147 0.1638 0.1835 0.2409 0.2523 0.3576 0.2374 0.0646 0.1667 0.2083 0.3333 0.3333 0.3333 0.2857 0.0813
148 0.5783 0.6279 0.6487 0.6526 0.6779 0.6389 0.0313 0.5952 0.6012 0.6190 0.6369 0.6667 0.6224 0.0255
149 0.8218 0.8351 0.8819 0.8819 1.0000 0.8781 0.0595 0.6667 0.6667 0.6667 0.6667 1.0000 0.7143 0.1260
160 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000
161 1.0000 1.0000 1.0000 1.0000 1.0000 1.0000 0.0000 1.0000 1.0000 1.0000 1.0000 1.0000 1.0000 0.0000
162 0.8410 0.8490 0.8505 0.8624 0.8909 0.8569 0.0167 0.6970 0.7273 0.7273 0.7273 0.7879 0.7316 0.0273
163 0.1376 0.1388 0.1692 0.2410 0.2580 0.1864 0.0530 0.1429 0.1429 0.2143 0.2857 0.4286 0.2347 0.1069
164 0.0305 0.0619 0.0684 0.0809 0.1116 0.0705 0.0245 0.0600 0.1050 0.1800 0.1800 0.1800 0.1429 0.0496
165 0.5435 0.5553 0.7000 0.9583 1.0000 0.7445 0.2020 0.5000 0.5000 0.5000 0.8750 1.0000 0.6429 0.2440
166 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000
167 0.3667 0.3667 0.3667 0.8125 1.0000 0.5786 0.2743 0.0000 0.0000 0.0000 0.5000 1.0000 0.2857 0.3934
168 0.0655 0.4341 0.4571 0.4633 0.6071 0.4194 0.1668 0.1212 0.4318 0.5152 0.5152 0.5758 0.4459 0.1512
169 0.0004 0.0112 0.0237 0.0245 0.0268 0.0179 0.0098 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000
170 0.5510 0.7410 0.8030 0.8046 0.8533 0.7624 0.1009 0.5000 0.6071 0.7143 0.7679 0.8571 0.6939 0.1217
171 0.0319 0.1150 0.1168 0.3647 0.3843 0.2122 0.1497 0.0556 0.1667 0.1667 0.3194 0.4444 0.2302 0.1300
172 0.1298 0.1730 0.2065 0.2095 0.2213 0.1900 0.0318 0.1429 0.1429 0.2857 0.2857 0.2857 0.2245 0.0764
173 0.5833 0.5833 0.8333 0.9583 1.0000 0.7738 0.1907 0.5000 0.5000 0.5000 0.8750 1.0000 0.6429 0.2440
174 0.6986 0.9276 0.9452 0.9479 0.9620 0.9093 0.0936 0.6087 0.8804 0.9130 0.9130 0.9130 0.8634 0.1135
175 0.0023 0.0024 0.0042 0.0073 0.0098 0.0049 0.0029 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000
176 0.9089 0.9120 0.9122 0.9448 0.9600 0.9269 0.0208 0.8000 0.8000 0.8000 0.8375 0.8500 0.8143 0.0244
177 0.0908 0.1396 0.5000 0.5000 0.5000 0.3463 0.1959 0.0000 0.2083 0.5000 0.5000 0.5000 0.3571 0.2025
178 0.4353 0.4680 0.5133 0.5538 0.5629 0.5084 0.0489 0.4286 0.4286 0.4286 0.4286 0.4286 0.4286 0.0000
179 0.2635 0.4479 0.4782 0.4839 0.7377 0.4804 0.1385 0.2903 0.4556 0.4677 0.4677 0.6774 0.4700 0.1123
190 0.3343 0.4075 0.5447 0.5552 0.5586 0.4846 0.0917 0.3871 0.4355 0.6129 0.6129 0.6129 0.5346 0.1018
191 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000
192 0.2569 0.3173 0.3661 0.4873 0.6480 0.4053 0.1356 0.4000 0.4333 0.5333 0.5833 0.6667 0.5238 0.0976
193 0.5362 0.6477 0.6498 0.6861 0.7680 0.6576 0.0690 0.5789 0.5789 0.5789 0.6316 0.7368 0.6165 0.0586
194 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000
195 0.0052 0.0164 0.0769 0.0769 0.0833 0.0510 0.0348 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000
196 0.6110 0.7399 0.8157 0.8401 0.8623 0.7797 0.0881 0.5556 0.6667 0.6667 0.6667 0.6667 0.6508 0.0420
197 0.6638 0.8510 0.9087 0.9098 0.9111 0.8613 0.0909 0.5714 0.7584 0.8067 0.8067 0.8235 0.7647 0.0885
198 0.2111 0.3667 0.4167 0.4167 0.7000 0.4135 0.1458 0.0000 0.0000 0.0000 0.3750 0.5000 0.1429 0.2440
199 0.6999 0.7982 0.8420 0.8455 0.8585 0.8130 0.0551 0.6875 0.7188 0.7500 0.7812 0.7812 0.7455 0.0380
200 0.0322 0.0904 0.1086 0.2101 0.2129 0.1359 0.0726 0.0357 0.0446 0.1429 0.3482 0.3571 0.1888 0.1511
ALL 0.4160 0.4302 0.4499 0.4611 0.5106 0.4520 0.0308 0.3951 0.4060 0.4285 0.4383 0.4892 0.4300 0.0312
65
66
AH-ROBUST-CLEF2006 Track Overview Results and Graphs AH-ROBUST-MULTI-CLEF2006
Ad−Hoc Robust Multilingual track Top 5 Participants − Interpolated Recall vs Average Precision
100%
jaen [ujamlrsv2; MAP 27.85%; Not Pooled]
daedalus [mlRSFSen2S; MAP 22.67%; Not Pooled]
90% colesir [CoLesIRmultTst; MAP 22.63%; Not Pooled]
reina [reinaES2mtdtest; MAP 19.96%; Not Pooled]
80%
70%
Average Precision
60%
50%
40%
30%
20%
10%
0%
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
Interpolated Recall
Ad−Hoc Robust Multilingual track Top 5 Participants − Comparison to Median Mean Average Precision by Topic (Topics 041 to 085
)
1
jaen [ujamlrsv2; MAP 27.85%; Not Pooled]
daedalus [mlRSFSen2S; MAP 22.67%; Not Pooled]
0.5 colesir [CoLesIRmultTst; MAP 22.63%; Not Pooled]
Difference
reina [reinaES2mtdtest; MAP 19.96%; Not Pooled]
0
−0.5
−1
041 042 043 044 045 046 047 048 049 060 061 062 063 064 065 066 067 068 069 080 081 082 083 084 085
Topic Identifier
Ad−Hoc Robust Multilingual track Top 5 Participants − Comparison to Median Mean Average Precision by Topic (Topics 086 to 130
)
1
jaen [ujamlrsv2; MAP 27.85%; Not Pooled]
daedalus [mlRSFSen2S; MAP 22.67%; Not Pooled]
0.5 colesir [CoLesIRmultTst; MAP 22.63%; Not Pooled]
Difference
reina [reinaES2mtdtest; MAP 19.96%; Not Pooled]
0
−0.5
−1
086 087 088 089 090 091 092 093 094 095 096 097 098 099 110 111 112 113 114 115 116 117 118 119 130
Topic Identifier
Ad−Hoc Robust Multilingual track Top 5 Participants − Comparison to Median Mean Average Precision by Topic (Topics 131 to 165
)
1
jaen [ujamlrsv2; MAP 27.85%; Not Pooled]
daedalus [mlRSFSen2S; MAP 22.67%; Not Pooled]
0.5 colesir [CoLesIRmultTst; MAP 22.63%; Not Pooled]
Difference
reina [reinaES2mtdtest; MAP 19.96%; Not Pooled]
0
−0.5
−1
131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 160 161 162 163 164 165
Topic Identifier
Ad−Hoc Robust Multilingual track Top 5 Participants − Comparison to Median Mean Average Precision by Topic (Topics 166 to 200
)
1
jaen [ujamlrsv2; MAP 27.85%; Not Pooled]
daedalus [mlRSFSen2S; MAP 22.67%; Not Pooled]
0.5 colesir [CoLesIRmultTst; MAP 22.63%; Not Pooled]
Difference
reina [reinaES2mtdtest; MAP 19.96%; Not Pooled]
0
−0.5
−1
166 167 168 169 170 171 172 173 174 175 176 177 178 179 190 191 192 193 194 195 196 197 198 199 200
Topic Identifier
67
AH-ROBUST-CLEF2006 Track Overview Results and Graphs AH-ROBUST-MULTI-CLEF2006
Ad−Hoc Robust Multilingual track − Box Plot of the Topics
ujamlrsv2 [MAP 27.85%; Not Pooled]
ujamllr [MAP 27.11%; Not Pooled]
ujamlblr [MAP 27.01%; Not Pooled]
ml6XRSFSen4S [MAP 25.67%; Not Pooled]
ml5XRSFSen4S [MAP 25.67%; Not Pooled]
Experiments
ml4XRSFSen4S [MAP 25.65%; Not Pooled]
mlRSFSen2S [MAP 22.67%; Not Pooled]
CoLesIRmultTst [MAP 22.63%; Not Pooled]
reinaES2mtdtest [MAP 19.96%; Not Pooled]
reinaES2mttest [MAP 17.31%; Not Pooled]
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
Mean Average Precision
68
AH-ROBUST-CLEF2006 Track Overview Results and Graphs AH-ROBUST-MULTI-CLEF2006
Ad−Hoc Robust Multilingual track − Tukey T test with "top group" highlighted
ujamlrsv2
ujamllr
ujamlblr
ml5XRSFSen4S
Experiments
ml6XRSFSen4S
ml4XRSFSen4S
mlRSFSen2S
CoLesIRmultTst
reinaES2mtdtest
reinaES2mttest
0.35 0.4 0.45 0.5 0.55 0.6 0.65 0.7
arcsin(sqrt(Mean average precision))
69
AH-ROBUST-CLEF2006 Track Overview Results and Graphs AH-ROBUST-MULTI-CLEF2006
Ad−Hoc Robust Multilingual track Top 5 Participants − Retrieved documents vs Precision
100%
jaen [ujamlrsv2; R−Prec 32.70%; Not Pooled]
daedalus [mlRSFSen2S; R−Prec 28.75%; Not Pooled]
90% colesir [CoLesIRmultTst; R−Prec 29.02%; Not Pooled]
reina [reinaES2mtdtest; R−Prec 27.60%; Not Pooled]
80%
70%
60%
R−Precision
50%
40%
30%
20%
10%
0%
5 10 15 20 30 100 200 500 1000
Retrieved Documents (logarithmic scale)
Ad−Hoc Robust Multilingual track Top 5 Participants − Comparison to Median R−Precision by Topic (Topics 041 to 085)
1
jaen [ujamlrsv2; R−Prec 32.70%; Not Pooled]
daedalus [mlRSFSen2S; R−Prec 28.75%; Not Pooled]
0.5 colesir [CoLesIRmultTst; R−Prec 29.02%; Not Pooled]
Difference
reina [reinaES2mtdtest; R−Prec 27.60%; Not Pooled]
0
−0.5
−1
041 042 043 044 045 046 047 048 049 060 061 062 063 064 065 066 067 068 069 080 081 082 083 084 085
Topic Identifier
Ad−Hoc Robust Multilingual track Top 5 Participants − Comparison to Median R−Precision by Topic (Topics 086 to 130)
1
jaen [ujamlrsv2; R−Prec 32.70%; Not Pooled]
daedalus [mlRSFSen2S; R−Prec 28.75%; Not Pooled]
0.5 colesir [CoLesIRmultTst; R−Prec 29.02%; Not Pooled]
Difference
reina [reinaES2mtdtest; R−Prec 27.60%; Not Pooled]
0
−0.5
−1
086 087 088 089 090 091 092 093 094 095 096 097 098 099 110 111 112 113 114 115 116 117 118 119 130
Topic Identifier
Ad−Hoc Robust Multilingual track Top 5 Participants − Comparison to Median R−Precision by Topic (Topics 131 to 165)
1
jaen [ujamlrsv2; R−Prec 32.70%; Not Pooled]
daedalus [mlRSFSen2S; R−Prec 28.75%; Not Pooled]
0.5 colesir [CoLesIRmultTst; R−Prec 29.02%; Not Pooled]
Difference
reina [reinaES2mtdtest; R−Prec 27.60%; Not Pooled]
0
−0.5
−1
131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 160 161 162 163 164 165
Topic Identifier
Ad−Hoc Robust Multilingual track Top 5 Participants − Comparison to Median R−Precision by Topic (Topics 166 to 200)
1
jaen [ujamlrsv2; R−Prec 32.70%; Not Pooled]
daedalus [mlRSFSen2S; R−Prec 28.75%; Not Pooled]
0.5 colesir [CoLesIRmultTst; R−Prec 29.02%; Not Pooled]
Difference
reina [reinaES2mtdtest; R−Prec 27.60%; Not Pooled]
0
−0.5
−1
166 167 168 169 170 171 172 173 174 175 176 177 178 179 190 191 192 193 194 195 196 197 198 199 200
Topic Identifier
70
AH-ROBUST-CLEF2006 Track Overview Results and Graphs AH-ROBUST-MULTI-CLEF2006
Ad−Hoc Robust Multilingual track − Box Plot of the Topics
ujamlrsv2 [R−Prec 32.70%; Not Pooled]
ujamllr [R−Prec 32.42%; Not Pooled]
ujamlblr [R−Prec 32.34%; Not Pooled]
ml4XRSFSen4S [R−Prec 30.82%; Not Pooled]
ml5XRSFSen4S [R−Prec 30.80%; Not Pooled]
Experiments
ml6XRSFSen4S [R−Prec 30.76%; Not Pooled]
CoLesIRmultTst [R−Prec 29.02%; Not Pooled]
mlRSFSen2S [R−Prec 28.75%; Not Pooled]
reinaES2mtdtest [R−Prec 27.60%; Not Pooled]
reinaES2mttest [R−Prec 25.14%; Not Pooled]
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
R−Precision
71
AH-ROBUST-CLEF2006 Track Overview Results and Graphs AH-ROBUST-MULTI-CLEF2006
Ad−Hoc Robust Multilingual track − Tukey T test with "top group" highlighted
ujamlrsv2
ujamlblr
ujamllr
ml4XRSFSen4S
Experiments
ml5XRSFSen4S
ml6XRSFSen4S
CoLesIRmultTst
mlRSFSen2S
reinaES2mtdtest
reinaES2mttest
0.46 0.48 0.5 0.52 0.54 0.56 0.58 0.6 0.62
arcsin(sqrt(R Precision))
72
AH-ROBUST-CLEF2006 Track Overview Results and Graphs AH-ROBUST-MULTI-CLEF2006
Average Precision R-Precision
Topic Minimum 1st Q. Median 3rd Q. Maximum Mean Std Minimum 1st Q. Median 3rd Q. Maximum Mean Std
041 0.0977 0.1465 0.1548 0.1599 0.1640 0.1463 0.0206 0.1310 0.2024 0.2083 0.2143 0.2381 0.2036 0.0289
042 0.0000 0.0715 0.1310 0.3001 0.3591 0.1727 0.1304 0.0014 0.2405 0.3076 0.4314 0.4639 0.3100 0.1430
043 0.0489 0.1396 0.4005 0.5654 0.5746 0.3650 0.1927 0.1111 0.3056 0.5278 0.5278 0.5556 0.4278 0.1626
044 0.0701 0.0739 0.0779 0.1056 0.1237 0.0878 0.0186 0.0462 0.0462 0.1077 0.1538 0.1692 0.1046 0.0485
045 0.2647 0.2801 0.3505 0.3681 0.3701 0.3297 0.0432 0.3834 0.4096 0.4444 0.4749 0.4967 0.4431 0.0397
046 0.0176 0.0234 0.0479 0.1013 0.1064 0.0593 0.0392 0.0446 0.0446 0.0759 0.1786 0.1875 0.1071 0.0648
047 0.0724 0.0767 0.0994 0.1214 0.1302 0.0982 0.0219 0.1691 0.1780 0.1869 0.1958 0.2166 0.1890 0.0139
048 0.0068 0.0248 0.0421 0.0512 0.0559 0.0366 0.0181 0.0480 0.0959 0.1310 0.1476 0.1550 0.1181 0.0383
049 0.0397 0.0473 0.0614 0.0674 0.0699 0.0575 0.0118 0.0943 0.1038 0.1321 0.1415 0.1698 0.1292 0.0281
060 0.1002 0.1494 0.1496 0.1859 0.1909 0.1548 0.0293 0.2100 0.2414 0.2821 0.2978 0.3009 0.2734 0.0314
061 0.0229 0.0471 0.0822 0.1464 0.1798 0.0942 0.0545 0.0606 0.0909 0.1313 0.1919 0.2626 0.1485 0.0644
062 0.1445 0.2427 0.2789 0.2923 0.3025 0.2581 0.0488 0.2963 0.3086 0.3333 0.3333 0.3457 0.3259 0.0186
063 0.2425 0.4466 0.4737 0.5095 0.5347 0.4494 0.0923 0.3333 0.4638 0.4819 0.5072 0.5580 0.4775 0.0614
064 0.1812 0.2368 0.3397 0.4183 0.4183 0.3240 0.0946 0.1500 0.3500 0.4000 0.5000 0.5000 0.3900 0.1150
065 0.0261 0.0319 0.0370 0.0372 0.0653 0.0395 0.0125 0.0637 0.0717 0.1036 0.1155 0.1474 0.0988 0.0284
066 0.4174 0.4811 0.5225 0.7197 0.7534 0.5638 0.1287 0.4657 0.4657 0.5397 0.6679 0.7040 0.5610 0.0987
067 0.0233 0.0532 0.0728 0.1625 0.1683 0.0877 0.0564 0.0515 0.1340 0.1856 0.2732 0.2835 0.1830 0.0800
068 0.0177 0.0656 0.1107 0.1243 0.1354 0.0940 0.0401 0.0652 0.1304 0.1812 0.2029 0.2391 0.1659 0.0563
069 0.0263 0.0327 0.1210 0.1392 0.2668 0.1096 0.0759 0.0652 0.0652 0.2174 0.2174 0.3261 0.1761 0.0896
080 0.0426 0.2015 0.2132 0.2202 0.2377 0.1962 0.0557 0.1685 0.3704 0.3768 0.4118 0.4229 0.3690 0.0736
081 0.1958 0.3328 0.4077 0.4805 0.4806 0.3938 0.0988 0.3380 0.4306 0.4931 0.5741 0.5741 0.4926 0.0805
082 0.3047 0.3971 0.4393 0.5664 0.6081 0.4642 0.1055 0.3582 0.4104 0.4440 0.5746 0.5896 0.4813 0.0870
083 0.1282 0.1615 0.1757 0.2539 0.2830 0.1962 0.0546 0.1765 0.2059 0.2353 0.3235 0.3235 0.2500 0.0559
084 0.0833 0.2601 0.2961 0.3186 0.3186 0.2694 0.0727 0.1860 0.3256 0.3488 0.3721 0.3721 0.3372 0.0562
085 0.1876 0.1894 0.2126 0.2397 0.4058 0.2395 0.0727 0.2874 0.2904 0.3204 0.3713 0.4790 0.3431 0.0666
086 0.0719 0.1165 0.1266 0.1658 0.1666 0.1313 0.0299 0.1749 0.2295 0.2432 0.2459 0.2514 0.2347 0.0224
087 0.2880 0.2984 0.3874 0.3977 0.4540 0.3648 0.0632 0.3522 0.3648 0.4151 0.4465 0.5535 0.4201 0.0669
088 0.1716 0.1856 0.2517 0.3090 0.3756 0.2536 0.0753 0.3103 0.3276 0.3621 0.3621 0.4397 0.3552 0.0365
089 0.1058 0.2494 0.3884 0.4110 0.4178 0.3253 0.1250 0.2218 0.3930 0.4630 0.4669 0.5058 0.4125 0.1024
090 0.0507 0.1187 0.1200 0.1475 0.1934 0.1228 0.0426 0.1348 0.2022 0.2135 0.2247 0.2360 0.2034 0.0320
091 0.0492 0.0742 0.0863 0.1711 0.2080 0.1153 0.0584 0.0823 0.1139 0.1234 0.3038 0.3038 0.1873 0.0982
092 0.1371 0.1572 0.1770 0.2233 0.2272 0.1858 0.0334 0.2379 0.2714 0.2862 0.2900 0.3123 0.2788 0.0217
093 0.1057 0.1197 0.3189 0.4879 0.5194 0.3081 0.1971 0.1667 0.2024 0.4286 0.5952 0.5952 0.3988 0.2087
094 0.0899 0.1327 0.1574 0.1743 0.5742 0.1894 0.1377 0.1983 0.2231 0.2273 0.2479 0.5868 0.2686 0.1147
095 0.0221 0.0673 0.0780 0.0842 0.0895 0.0717 0.0203 0.1340 0.2177 0.2381 0.2437 0.2493 0.2250 0.0343
096 0.0466 0.0976 0.1085 0.1159 0.1182 0.0981 0.0256 0.1304 0.1925 0.1988 0.2050 0.2050 0.1876 0.0260
097 0.0628 0.1048 0.1754 0.2131 0.2453 0.1603 0.0627 0.1471 0.1471 0.3015 0.3088 0.3529 0.2515 0.0918
098 0.3485 0.4098 0.4414 0.4543 0.5459 0.4334 0.0567 0.4310 0.5172 0.5431 0.5517 0.5517 0.5190 0.0484
099 0.0158 0.0395 0.0606 0.0805 0.0810 0.0570 0.0225 0.0885 0.1094 0.1458 0.1719 0.1719 0.1380 0.0308
110 0.3083 0.5170 0.5238 0.5568 0.5946 0.5018 0.0882 0.4536 0.5567 0.5670 0.6392 0.6495 0.5784 0.0621
111 0.0301 0.0903 0.0924 0.1040 0.1132 0.0891 0.0240 0.1184 0.1579 0.1711 0.1711 0.2105 0.1658 0.0242
112 0.1338 0.2004 0.2486 0.3434 0.3773 0.2640 0.0832 0.2319 0.2319 0.3188 0.3913 0.4638 0.3188 0.0826
113 0.0002 0.0004 0.0010 0.0014 0.0217 0.0034 0.0066 0.0035 0.0070 0.0123 0.0211 0.0880 0.0229 0.0275
114 0.0470 0.1059 0.1302 0.2693 0.2702 0.1622 0.0821 0.1034 0.1667 0.1868 0.3046 0.3046 0.2167 0.0748
115 0.1132 0.1253 0.1453 0.1458 0.1737 0.1397 0.0168 0.2143 0.2302 0.2460 0.2540 0.2857 0.2452 0.0203
116 0.0941 0.3263 0.3579 0.3772 0.4249 0.3121 0.1144 0.2449 0.3776 0.3980 0.4082 0.4388 0.3714 0.0687
117 0.0242 0.0348 0.0504 0.0901 0.1034 0.0604 0.0308 0.0516 0.0516 0.1355 0.1548 0.1677 0.1148 0.0498
118 0.0115 0.0225 0.0265 0.0496 0.0682 0.0325 0.0196 0.0435 0.0652 0.0761 0.1304 0.1522 0.0891 0.0362
119 0.2602 0.4657 0.5224 0.5471 0.6141 0.4993 0.0942 0.3762 0.5210 0.5304 0.5748 0.6051 0.5308 0.0618
130 0.2574 0.3268 0.3509 0.4277 0.4280 0.3609 0.0596 0.3759 0.4135 0.4361 0.4511 0.4887 0.4323 0.0358
73
AH-ROBUST-CLEF2006 Track Overview Results and Graphs AH-ROBUST-MULTI-CLEF2006
Average Precision R-Precision
Topic Minimum 1st Q. Median 3rd Q. Maximum Mean Std Minimum 1st Q. Median 3rd Q. Maximum Mean Std
131 0.1065 0.1466 0.1579 0.1698 0.2146 0.1585 0.0277 0.2322 0.2612 0.2955 0.3087 0.3404 0.2876 0.0336
132 0.0192 0.4303 0.5533 0.5643 0.6122 0.4459 0.2239 0.0135 0.4865 0.5000 0.5676 0.5811 0.4365 0.2053
133 0.0050 0.0063 0.0190 0.0204 0.0216 0.0155 0.0069 0.0419 0.0558 0.0721 0.0791 0.0930 0.0679 0.0166
134 0.1732 0.5924 0.6296 0.7789 0.8997 0.6255 0.2137 0.2963 0.5926 0.6296 0.7037 0.8519 0.6111 0.1640
135 0.0502 0.0524 0.2075 0.2603 0.3525 0.1787 0.1158 0.1297 0.1297 0.3253 0.4032 0.4172 0.2826 0.1201
136 0.1742 0.2298 0.3415 0.4246 0.4535 0.3277 0.0971 0.2571 0.3143 0.4000 0.4286 0.4286 0.3743 0.0638
137 0.0386 0.0469 0.1157 0.1296 0.1586 0.0958 0.0471 0.0800 0.1200 0.1800 0.2000 0.2400 0.1640 0.0515
138 0.0101 0.0186 0.0927 0.3396 0.3397 0.1433 0.1435 0.0420 0.0840 0.1723 0.3697 0.3697 0.1950 0.1297
139 0.0074 0.0155 0.0230 0.0855 0.0998 0.0412 0.0376 0.0481 0.0673 0.1090 0.1474 0.1699 0.1087 0.0400
140 0.1102 0.1376 0.1890 0.1976 0.2036 0.1698 0.0345 0.2517 0.2703 0.3454 0.3936 0.3953 0.3331 0.0585
141 0.4024 0.4680 0.7123 0.7454 0.8033 0.6499 0.1472 0.4706 0.5294 0.7059 0.7941 0.7941 0.6706 0.1246
142 0.3403 0.5669 0.5678 0.8447 0.9182 0.6376 0.2083 0.3547 0.5988 0.5988 0.8023 0.8488 0.6291 0.1745
143 0.1841 0.2437 0.2590 0.2725 0.2862 0.2527 0.0311 0.3436 0.3492 0.3743 0.3911 0.4497 0.3774 0.0323
144 0.1838 0.1969 0.3684 0.4898 0.5243 0.3497 0.1439 0.1667 0.2000 0.4667 0.5000 0.5667 0.3767 0.1626
145 0.0993 0.3876 0.4644 0.4846 0.4994 0.4139 0.1196 0.1452 0.3952 0.5000 0.5161 0.5161 0.4387 0.1166
146 0.0891 0.1710 0.3042 0.3415 0.4300 0.2708 0.1078 0.1667 0.3333 0.3333 0.3333 0.5000 0.3167 0.0946
147 0.0173 0.0599 0.1041 0.4017 0.4335 0.1903 0.1707 0.0648 0.0648 0.1991 0.4722 0.5000 0.2454 0.1885
148 0.0247 0.0292 0.0441 0.0472 0.0700 0.0417 0.0132 0.0800 0.0800 0.0933 0.1067 0.1067 0.0933 0.0109
149 0.2135 0.4992 0.6763 0.6811 0.7262 0.5613 0.1978 0.2821 0.5385 0.6282 0.6410 0.7179 0.5590 0.1494
160 0.0657 0.0657 0.2196 0.2446 0.2688 0.1799 0.0844 0.0000 0.0000 0.2000 0.2000 0.2000 0.1400 0.0966
161 0.0000 0.0003 0.0011 0.0350 0.2815 0.0504 0.0993 0.0000 0.0000 0.0000 0.1111 0.2222 0.0556 0.0944
162 0.2255 0.2900 0.3016 0.4293 0.4846 0.3390 0.0901 0.2832 0.3230 0.3739 0.5044 0.5354 0.4049 0.0967
163 0.2739 0.4020 0.4956 0.6713 0.7021 0.5088 0.1442 0.3522 0.4980 0.5466 0.6235 0.6680 0.5409 0.0951
164 0.0005 0.0066 0.0099 0.0121 0.0764 0.0185 0.0236 0.0147 0.0616 0.0660 0.0733 0.1848 0.0804 0.0482
165 0.0852 0.1083 0.2485 0.3133 0.3487 0.2157 0.1029 0.1053 0.1053 0.3158 0.3684 0.4211 0.2632 0.1289
166 0.0000 0.0017 0.0025 0.0029 0.0045 0.0024 0.0013 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000
167 0.3028 0.3617 0.3822 0.4200 0.4596 0.3859 0.0459 0.3200 0.3600 0.4200 0.4400 0.4400 0.4000 0.0499
168 0.0888 0.1063 0.1237 0.1714 0.1743 0.1322 0.0328 0.1842 0.2105 0.2270 0.2763 0.2763 0.2342 0.0323
169 0.0002 0.0057 0.0064 0.0126 0.0146 0.0081 0.0045 0.0000 0.0000 0.0000 0.0000 0.0526 0.0105 0.0222
170 0.0902 0.1760 0.2701 0.5139 0.5139 0.3025 0.1638 0.1905 0.3333 0.3333 0.4762 0.4762 0.3571 0.1082
171 0.0665 0.0704 0.0779 0.0846 0.1214 0.0818 0.0168 0.0847 0.1017 0.1271 0.1356 0.2203 0.1288 0.0393
172 0.0068 0.0111 0.0540 0.0782 0.1195 0.0552 0.0405 0.0000 0.0357 0.1071 0.1786 0.2143 0.1107 0.0703
173 0.1880 0.4175 0.6827 0.6924 0.7348 0.5692 0.1838 0.1600 0.4400 0.6600 0.6800 0.6800 0.5560 0.1743
174 0.1244 0.4697 0.5605 0.7805 0.8356 0.5640 0.2396 0.2069 0.4713 0.5920 0.7241 0.7816 0.5609 0.1965
175 0.1369 0.2215 0.3303 0.4348 0.5472 0.3356 0.1282 0.1818 0.3182 0.3409 0.4545 0.5909 0.3682 0.1221
176 0.5362 0.7432 0.8195 0.8657 0.8773 0.7695 0.1284 0.6000 0.7510 0.7816 0.7959 0.7959 0.7478 0.0769
177 0.0331 0.1587 0.1706 0.1786 0.2064 0.1615 0.0484 0.0968 0.2097 0.2258 0.2258 0.2419 0.2113 0.0413
178 0.0033 0.0060 0.0309 0.0519 0.1542 0.0487 0.0552 0.0315 0.0394 0.1181 0.1417 0.2992 0.1260 0.0937
179 0.1238 0.1433 0.2646 0.2671 0.2807 0.2161 0.0685 0.1871 0.2109 0.3537 0.3844 0.3844 0.3034 0.0888
190 0.0035 0.0577 0.1085 0.1622 0.1633 0.0992 0.0596 0.0229 0.1374 0.2137 0.2595 0.2595 0.1832 0.0867
191 0.0204 0.0204 0.1625 1.0000 1.0000 0.3928 0.4429 0.0000 0.0000 0.0000 1.0000 1.0000 0.3000 0.4830
192 0.0604 0.1027 0.1620 0.2905 0.3133 0.1893 0.0969 0.1206 0.1773 0.2057 0.3546 0.3901 0.2518 0.1039
193 0.0796 0.2095 0.2475 0.2603 0.2615 0.2195 0.0611 0.1439 0.2734 0.2806 0.2950 0.3309 0.2705 0.0503
194 0.0018 0.0459 0.0548 0.2150 0.2385 0.1090 0.0928 0.0000 0.0833 0.1250 0.2917 0.2917 0.1625 0.1084
195 0.0177 0.0792 0.1164 0.1730 0.1927 0.1155 0.0633 0.0328 0.1639 0.1885 0.2295 0.3279 0.1951 0.0922
196 0.3072 0.5268 0.5524 0.5620 0.6175 0.5149 0.1017 0.3636 0.5758 0.6061 0.6061 0.6364 0.5636 0.0917
197 0.2632 0.3982 0.4456 0.5326 0.5389 0.4456 0.0865 0.3627 0.5142 0.5551 0.5997 0.6062 0.5407 0.0715
198 0.0282 0.0588 0.0959 0.2237 0.2238 0.1193 0.0793 0.0909 0.0909 0.1212 0.2121 0.2121 0.1455 0.0586
199 0.4257 0.6278 0.7116 0.7255 0.7257 0.6547 0.1137 0.5615 0.6352 0.6803 0.6803 0.7090 0.6582 0.0486
200 0.1097 0.2047 0.2773 0.3119 0.3123 0.2489 0.0696 0.2463 0.3005 0.3374 0.3399 0.3744 0.3236 0.0389
ALL 0.1731 0.2263 0.2566 0.2701 0.2785 0.2415 0.0344 0.2514 0.2875 0.3078 0.3234 0.3270 0.3003 0.0241
74
75
AH-ROBUST-CLEF2006 Track Overview Results and Graphs AH-ROBUST-BILI-X2DE-CLEF2006
Ad−Hoc Robust Bilingual German track Top 5 Participants − Interpolated Recall vs Average Precision
100%
daedalus [deFSdeRSen2S; MAP 29.16%; Not Pooled]
colesir [CoLesIRendeTst; MAP 25.24%; Not Pooled]
90%
80%
70%
Average Precision
60%
50%
40%
30%
20%
10%
0%
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
Interpolated Recall
Ad−Hoc Robust Bilingual German track Top 5 Participants − Comparison to Median Mean Average Precision by Topic (Topics 041 to085)
1
daedalus [deFSdeRSen2S; MAP 29.16%; Not Pooled]
colesir [CoLesIRendeTst; MAP 25.24%; Not Pooled]
0.5
Difference
0
−0.5
−1
041 042 043 044 045 046 047 048 049 060 061 062 063 064 065 066 067 068 069 080 081 082 083 084 085
Topic Identifier
Ad−Hoc Robust Bilingual German track Top 5 Participants − Comparison to Median Mean Average Precision by Topic (Topics 086 to130)
1
daedalus [deFSdeRSen2S; MAP 29.16%; Not Pooled]
colesir [CoLesIRendeTst; MAP 25.24%; Not Pooled]
0.5
Difference
0
−0.5
−1
086 087 088 089 090 091 092 093 094 095 096 097 098 099 110 111 112 113 114 115 116 117 118 119 130
Topic Identifier
Ad−Hoc Robust Bilingual German track Top 5 Participants − Comparison to Median Mean Average Precision by Topic (Topics 131 to165)
1
daedalus [deFSdeRSen2S; MAP 29.16%; Not Pooled]
colesir [CoLesIRendeTst; MAP 25.24%; Not Pooled]
0.5
Difference
0
−0.5
−1
131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 160 161 162 163 164 165
Topic Identifier
Ad−Hoc Robust Bilingual German track Top 5 Participants − Comparison to Median Mean Average Precision by Topic (Topics 166 to200)
1
daedalus [deFSdeRSen2S; MAP 29.16%; Not Pooled]
colesir [CoLesIRendeTst; MAP 25.24%; Not Pooled]
0.5
Difference
0
−0.5
−1
166 167 168 169 170 171 172 173 174 175 176 177 178 179 190 191 192 193 194 195 196 197 198 199 200
Topic Identifier
76
AH-ROBUST-CLEF2006 Track Overview Results and Graphs AH-ROBUST-BILI-X2DE-CLEF2006
Ad−Hoc Robust Bilingual German track − Box Plot of the Topics
deFSdeRSen3S [MAP 32.44%; Not Pooled]
deFSdeRSen4S [MAP 32.22%; Not Pooled]
Experiments
deydeRSenFS3456 [MAP 32.13%; Not Pooled]
deFSdeRSen2S [MAP 29.16%; Not Pooled]
CoLesIRendeTst [MAP 25.24%; Not Pooled]
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
Mean Average Precision
77
AH-ROBUST-CLEF2006 Track Overview Results and Graphs AH-ROBUST-BILI-X2DE-CLEF2006
Ad−Hoc Robust Bilingual German track − Tukey T test with "top group" highlighted
deFSdeRSen3S
deFSdeRSen4S
Experiments
deydeRSenFS3456
deFSdeRSen2S
CoLesIRendeTst
0.42 0.44 0.46 0.48 0.5 0.52 0.54 0.56 0.58
arcsin(sqrt(Mean average precision))
78
AH-ROBUST-CLEF2006 Track Overview Results and Graphs AH-ROBUST-BILI-X2DE-CLEF2006
Ad−Hoc Robust Bilingual German track Top 5 Participants − Retrieved documents vs Precision
100%
daedalus [deFSdeRSen2S; R−Prec 30.15%; Not Pooled]
colesir [CoLesIRendeTst; R−Prec 26.40%; Not Pooled]
90%
80%
70%
60%
R−Precision
50%
40%
30%
20%
10%
0%
5 10 15 20 30 100 200 500 1000
Retrieved Documents (logarithmic scale)
Ad−Hoc Robust Bilingual German track Top 5 Participants − Comparison to Median R−Precision by Topic (Topics 041 to 085)
1
daedalus [deFSdeRSen2S; R−Prec 30.15%; Not Pooled]
colesir [CoLesIRendeTst; R−Prec 26.40%; Not Pooled]
0.5
Difference
0
−0.5
−1
041 042 043 044 045 046 047 048 049 060 061 062 063 064 065 066 067 068 069 080 081 082 083 084 085
Topic Identifier
Ad−Hoc Robust Bilingual German track Top 5 Participants − Comparison to Median R−Precision by Topic (Topics 086 to 130)
1
daedalus [deFSdeRSen2S; R−Prec 30.15%; Not Pooled]
colesir [CoLesIRendeTst; R−Prec 26.40%; Not Pooled]
0.5
Difference
0
−0.5
−1
086 087 088 089 090 091 092 093 094 095 096 097 098 099 110 111 112 113 114 115 116 117 118 119 130
Topic Identifier
Ad−Hoc Robust Bilingual German track Top 5 Participants − Comparison to Median R−Precision by Topic (Topics 131 to 165)
1
daedalus [deFSdeRSen2S; R−Prec 30.15%; Not Pooled]
colesir [CoLesIRendeTst; R−Prec 26.40%; Not Pooled]
0.5
Difference
0
−0.5
−1
131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 160 161 162 163 164 165
Topic Identifier
Ad−Hoc Robust Bilingual German track Top 5 Participants − Comparison to Median R−Precision by Topic (Topics 166 to 200)
1
daedalus [deFSdeRSen2S; R−Prec 30.15%; Not Pooled]
colesir [CoLesIRendeTst; R−Prec 26.40%; Not Pooled]
0.5
Difference
0
−0.5
−1
166 167 168 169 170 171 172 173 174 175 176 177 178 179 190 191 192 193 194 195 196 197 198 199 200
Topic Identifier
79
AH-ROBUST-CLEF2006 Track Overview Results and Graphs AH-ROBUST-BILI-X2DE-CLEF2006
Ad−Hoc Robust Bilingual German track − Box Plot of the Topics
deFSdeRSen4S [R−Prec 33.20%; Not Pooled]
deFSdeRSen3S [R−Prec 33.16%; Not Pooled]
Experiments
deydeRSenFS3456 [R−Prec 33.15%; Not Pooled]
deFSdeRSen2S [R−Prec 30.15%; Not Pooled]
CoLesIRendeTst [R−Prec 26.40%; Not Pooled]
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
R−Precision
80
AH-ROBUST-CLEF2006 Track Overview Results and Graphs AH-ROBUST-BILI-X2DE-CLEF2006
Ad−Hoc Robust Bilingual German track − Tukey T test with "top group" highlighted
deFSdeRSen3S
deFSdeRSen4S
Experiments
deydeRSenFS3456
deFSdeRSen2S
CoLesIRendeTst
0.42 0.44 0.46 0.48 0.5 0.52 0.54 0.56 0.58
arcsin(sqrt(R Precision))
81
AH-ROBUST-CLEF2006 Track Overview Results and Graphs AH-ROBUST-BILI-X2DE-CLEF2006
Average Precision R-Precision
Topic Minimum 1st Q. Median 3rd Q. Maximum Mean Std Minimum 1st Q. Median 3rd Q. Maximum Mean Std
041 0.0085 0.0085 0.0090 0.0312 0.0643 0.0221 0.0241 0.0357 0.0625 0.0714 0.0714 0.0714 0.0643 0.0160
042 0.5768 0.6906 0.7409 0.7428 0.7447 0.7066 0.0728 0.6179 0.6533 0.6887 0.6922 0.7028 0.6726 0.0335
043 0.0008 0.0538 0.1021 0.1297 0.1370 0.0877 0.0548 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000
044 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000
045 0.5402 0.5696 0.5995 0.6025 0.6038 0.5850 0.0269 0.5377 0.5731 0.5943 0.5991 0.6132 0.5849 0.0283
046 0.0348 0.1508 0.2133 0.2254 0.2576 0.1820 0.0859 0.0000 0.1500 0.2000 0.2250 0.3000 0.1800 0.1095
047 0.0055 0.0083 0.0423 0.0453 0.0478 0.0299 0.0207 0.0208 0.0208 0.0417 0.0469 0.0625 0.0375 0.0174
048 0.0037 0.0080 0.0102 0.0105 0.0107 0.0089 0.0029 0.0000 0.0273 0.0364 0.0409 0.0545 0.0327 0.0199
049 0.0307 0.0390 0.0419 0.0435 0.0460 0.0406 0.0058 0.0556 0.0556 0.0556 0.0833 0.1667 0.0778 0.0497
060 0.1800 0.1821 0.1833 0.2294 0.2829 0.2081 0.0437 0.2456 0.2588 0.2807 0.3377 0.4035 0.3018 0.0625
061 0.0396 0.0808 0.1273 0.1541 0.1670 0.1156 0.0504 0.0465 0.0988 0.2326 0.2791 0.2791 0.1907 0.1045
062 0.3373 0.3523 0.3859 0.4509 0.5167 0.4052 0.0712 0.3333 0.3333 0.3333 0.4444 0.4444 0.3778 0.0609
063 0.4083 0.4916 0.5371 0.5531 0.5650 0.5158 0.0624 0.4211 0.5197 0.5789 0.5921 0.6316 0.5526 0.0789
064 0.5000 0.8750 1.0000 1.0000 1.0000 0.9000 0.2236 0.0000 0.7500 1.0000 1.0000 1.0000 0.8000 0.4472
065 0.0000 0.0004 0.0007 0.0147 0.0187 0.0067 0.0088 0.0000 0.0000 0.0000 0.0692 0.0923 0.0308 0.0435
066 0.6287 0.7333 0.7718 0.7746 0.7799 0.7443 0.0647 0.6667 0.6667 0.6863 0.6912 0.7059 0.6824 0.0164
067 0.0180 0.0610 0.0927 0.0937 0.0942 0.0748 0.0327 0.0200 0.1250 0.1800 0.1850 0.2000 0.1480 0.0729
068 0.1516 0.1597 0.1724 0.1916 0.1932 0.1741 0.0180 0.1818 0.1989 0.2045 0.2102 0.2273 0.2045 0.0161
069 0.1167 0.1261 0.1871 0.3051 0.3769 0.2182 0.1099 0.1429 0.2500 0.4286 0.4286 0.4286 0.3429 0.1278
080 0.1519 0.3990 0.5001 0.5094 0.5153 0.4312 0.1566 0.2250 0.4563 0.5333 0.5417 0.5417 0.4750 0.1398
081 0.3510 0.4691 0.5715 0.5818 0.5836 0.5191 0.0989 0.4000 0.5050 0.6000 0.6400 0.6400 0.5640 0.1004
082 0.2562 0.5715 0.7388 0.8038 0.8123 0.6569 0.2305 0.2593 0.5926 0.7037 0.7778 0.7778 0.6444 0.2185
083 0.1695 0.1744 0.1772 0.2129 0.2199 0.1906 0.0229 0.1111 0.1111 0.3333 0.3333 0.3333 0.2444 0.1217
084 0.0000 0.0000 0.0000 0.1000 0.1000 0.0400 0.0548 0.0000 0.0000 0.0000 0.2000 0.2000 0.0800 0.1095
085 0.4048 0.5551 0.6808 0.6974 0.7318 0.6217 0.1295 0.4595 0.5405 0.6216 0.6351 0.6757 0.5892 0.0820
086 0.1650 0.1735 0.1925 0.1982 0.2037 0.1868 0.0157 0.2432 0.2432 0.2432 0.2725 0.3063 0.2595 0.0273
087 0.1487 0.2298 0.2808 0.2831 0.2855 0.2508 0.0582 0.1765 0.2206 0.2353 0.2353 0.2353 0.2235 0.0263
088 0.4172 0.4434 0.4905 0.5049 0.5356 0.4780 0.0451 0.5345 0.5474 0.5690 0.5905 0.6034 0.5690 0.0273
089 0.3388 0.3570 0.3917 0.3939 0.3946 0.3764 0.0248 0.4207 0.4310 0.4621 0.4655 0.4759 0.4510 0.0227
090 0.1047 0.1151 0.1268 0.1281 0.1281 0.1212 0.0100 0.1875 0.1875 0.1875 0.1875 0.1875 0.1875 0.0000
091 0.0007 0.0008 0.0013 0.0145 0.0531 0.0115 0.0232 0.0000 0.0000 0.0312 0.0391 0.0625 0.0250 0.0261
092 0.1311 0.1537 0.1657 0.1719 0.1797 0.1614 0.0183 0.1471 0.1471 0.1765 0.2132 0.2353 0.1824 0.0383
093 0.6549 0.6882 0.7377 0.7470 0.7551 0.7183 0.0412 0.7222 0.7431 0.7500 0.7569 0.7778 0.7500 0.0196
094 0.0601 0.0642 0.0742 0.2035 0.4965 0.1604 0.1887 0.1250 0.1484 0.1562 0.2656 0.5000 0.2250 0.1553
095 0.0179 0.0226 0.0278 0.0862 0.0933 0.0494 0.0361 0.0455 0.0523 0.0545 0.1295 0.1364 0.0836 0.0443
096 0.0948 0.1009 0.1084 0.1116 0.1173 0.1066 0.0084 0.1111 0.1111 0.1852 0.1852 0.1852 0.1556 0.0406
097 0.0567 0.0858 0.0969 0.1255 0.1723 0.1063 0.0419 0.1333 0.1333 0.2000 0.2167 0.2667 0.1867 0.0558
098 0.4061 0.4264 0.4357 0.4465 0.4724 0.4370 0.0236 0.4400 0.5300 0.6000 0.6100 0.6400 0.5680 0.0769
099 0.0008 0.0057 0.0098 0.0105 0.0112 0.0079 0.0042 0.0000 0.0000 0.0112 0.0197 0.0449 0.0135 0.0185
110 0.3677 0.4544 0.4899 0.5202 0.5278 0.4773 0.0640 0.5000 0.5313 0.5417 0.5521 0.5833 0.5417 0.0295
111 0.0143 0.0222 0.0423 0.0446 0.0460 0.0343 0.0140 0.0000 0.0000 0.0000 0.1250 0.1250 0.0500 0.0685
112 0.5643 0.6276 0.6495 0.6650 0.6721 0.6395 0.0431 0.5000 0.5536 0.6429 0.6429 0.6429 0.6000 0.0639
113 0.0044 0.0055 0.0156 0.0371 0.0916 0.0273 0.0365 0.0000 0.0000 0.0400 0.0700 0.1600 0.0480 0.0657
114 0.0890 0.1462 0.1803 0.2067 0.2662 0.1775 0.0632 0.1071 0.1875 0.2143 0.2143 0.2143 0.1929 0.0479
115 0.0459 0.0583 0.2164 0.2767 0.2799 0.1761 0.1142 0.1250 0.1250 0.3333 0.3333 0.3333 0.2500 0.1141
116 0.4938 0.5237 0.6598 0.6650 0.6712 0.6043 0.0839 0.5484 0.5484 0.5806 0.6210 0.6452 0.5871 0.0421
117 0.0803 0.0849 0.1424 0.1444 0.1486 0.1202 0.0337 0.1111 0.1611 0.2000 0.2111 0.2444 0.1867 0.0487
118 0.0022 0.0141 0.0188 0.0323 0.0592 0.0243 0.0211 0.0000 0.0000 0.0000 0.0278 0.1111 0.0222 0.0497
119 0.4462 0.4571 0.4626 0.5011 0.6126 0.4892 0.0694 0.4792 0.4792 0.4792 0.4974 0.5521 0.4937 0.0326
130 0.0001 0.1295 0.5970 0.6204 0.6408 0.4048 0.2974 0.0000 0.2000 0.6000 0.6000 0.6000 0.4133 0.2724
82
AH-ROBUST-CLEF2006 Track Overview Results and Graphs AH-ROBUST-BILI-X2DE-CLEF2006
Average Precision R-Precision
Topic Minimum 1st Q. Median 3rd Q. Maximum Mean Std Minimum 1st Q. Median 3rd Q. Maximum Mean Std
131 0.0350 0.1772 0.2325 0.2415 0.2579 0.1972 0.0915 0.1212 0.2689 0.3333 0.3523 0.3636 0.2970 0.0997
132 0.6156 0.6215 0.6565 0.6877 0.7399 0.6611 0.0495 0.5357 0.6161 0.6429 0.7232 0.7500 0.6571 0.0822
133 0.0004 0.0039 0.0187 0.0287 0.0548 0.0198 0.0213 0.0091 0.0295 0.0727 0.0818 0.1091 0.0600 0.0384
134 0.6469 0.6469 0.6633 0.7401 0.8638 0.7039 0.0918 0.5714 0.5714 0.5714 0.7500 0.8571 0.6571 0.1278
135 0.0057 0.0075 0.0192 0.0277 0.0484 0.0205 0.0170 0.0000 0.0000 0.0345 0.0632 0.0805 0.0345 0.0354
136 0.0000 0.3901 0.6226 0.6226 0.6226 0.4776 0.2706 0.0000 0.3214 0.5714 0.5714 0.5714 0.4286 0.2474
137 0.1111 0.2778 0.3333 0.3750 0.5000 0.3222 0.1383 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000
138 0.1450 0.2937 0.5838 0.6250 0.7039 0.4749 0.2268 0.1613 0.2581 0.6452 0.6855 0.7097 0.4968 0.2526
139 0.0014 0.0019 0.0025 0.0030 0.0031 0.0024 0.0007 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000
140 0.0012 0.0012 0.0019 0.0130 0.0464 0.0105 0.0200 0.0000 0.0000 0.0247 0.0525 0.1358 0.0370 0.0566
141 0.5386 0.6971 0.7500 0.7500 0.7500 0.7077 0.0945 0.6250 0.7188 0.7500 0.7500 0.7500 0.7250 0.0559
142 0.3629 0.3630 0.3642 0.4512 0.7114 0.4332 0.1555 0.4000 0.4000 0.4000 0.4923 0.7692 0.4738 0.1651
143 0.1488 0.1518 0.1550 0.2038 0.2264 0.1759 0.0342 0.2222 0.2341 0.2540 0.2579 0.2698 0.2476 0.0181
144 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000
145 0.1946 0.4871 0.6088 0.6939 0.7029 0.5563 0.2086 0.2000 0.4250 0.6000 0.7000 0.7000 0.5400 0.2074
146 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000
147 0.0336 0.0630 0.1178 0.1661 0.1761 0.1126 0.0600 0.0690 0.1207 0.1724 0.2500 0.2759 0.1793 0.0823
148 0.0025 0.0026 0.0026 0.0105 0.0340 0.0089 0.0141 0.0000 0.0000 0.0000 0.0208 0.0833 0.0167 0.0373
149 0.7845 0.8337 0.8507 0.9310 0.9764 0.8755 0.0731 0.7500 0.8125 0.8333 0.8542 0.9167 0.8333 0.0589
160 0.3333 0.8333 1.0000 1.0000 1.0000 0.8667 0.2981 0.0000 0.7500 1.0000 1.0000 1.0000 0.8000 0.4472
161 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000
162 0.3350 0.4287 0.4760 0.4818 0.4838 0.4472 0.0634 0.3333 0.4896 0.5417 0.5937 0.6250 0.5250 0.1126
163 0.1290 0.1460 0.1877 0.2458 0.3600 0.2072 0.0907 0.1915 0.2234 0.2553 0.3298 0.4255 0.2809 0.0895
164 0.0003 0.0019 0.0025 0.0027 0.0031 0.0022 0.0011 0.0000 0.0104 0.0139 0.0139 0.0139 0.0111 0.0062
165 0.1728 0.2014 0.3519 0.4181 0.5357 0.3300 0.1450 0.2500 0.2500 0.2500 0.5000 0.5000 0.3500 0.1369
166 0.0024 0.0041 0.0050 0.0062 0.0094 0.0053 0.0025 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000
167 0.7000 0.7250 0.7333 0.7776 0.8875 0.7590 0.0735 0.5000 0.5000 0.5000 0.5625 0.7500 0.5500 0.1118
168 0.0561 0.0698 0.0771 0.0852 0.0875 0.0759 0.0123 0.0588 0.0588 0.0588 0.1176 0.1176 0.0824 0.0322
169 0.0000 0.0000 0.0000 0.0002 0.0007 0.0001 0.0003 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000
170 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000
171 0.0393 0.0730 0.0860 0.1252 0.1905 0.1007 0.0555 0.0000 0.0938 0.1250 0.2812 0.3750 0.1750 0.1425
172 0.0000 0.0000 0.0000 0.0023 0.0090 0.0018 0.0040 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000
173 0.6703 0.8318 0.9319 0.9336 0.9387 0.8717 0.1146 0.7000 0.7750 0.8000 0.8000 0.8000 0.7800 0.0447
174 0.6787 0.8148 0.8630 0.8898 0.9267 0.8412 0.0947 0.6389 0.7639 0.8056 0.8125 0.8333 0.7778 0.0786
175 0.5058 0.7467 0.8390 0.8612 0.8613 0.7789 0.1534 0.5455 0.6818 0.7273 0.8182 0.8182 0.7273 0.1113
176 0.7703 0.8515 0.9194 0.9252 0.9363 0.8852 0.0677 0.7297 0.7703 0.8784 0.8784 0.8784 0.8297 0.0693
177 0.0102 0.0262 0.1456 0.1468 0.1489 0.0965 0.0695 0.0000 0.0000 0.1429 0.1429 0.1429 0.0857 0.0782
178 0.0000 0.0008 0.0016 0.0018 0.0022 0.0013 0.0008 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000
179 0.0941 0.2948 0.4101 0.4161 0.4293 0.3414 0.1405 0.0816 0.3112 0.4286 0.4490 0.4490 0.3592 0.1572
190 0.1676 0.4267 0.5322 0.5437 0.5439 0.4601 0.1640 0.2807 0.4912 0.5614 0.5789 0.5789 0.5123 0.1298
191 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000
192 0.0049 0.0595 0.0809 0.1082 0.1452 0.0809 0.0503 0.0000 0.1154 0.1923 0.2019 0.2308 0.1538 0.0902
193 0.4397 0.4859 0.5056 0.5285 0.5370 0.5018 0.0377 0.4706 0.4706 0.5294 0.5368 0.5588 0.5118 0.0395
194 0.5500 0.5807 0.7000 0.7333 0.8333 0.6748 0.1107 0.5000 0.5000 0.5000 0.5000 0.5000 0.5000 0.0000
195 0.0010 0.0676 0.1489 0.2238 0.2317 0.1385 0.0960 0.0000 0.2069 0.3103 0.3103 0.3103 0.2414 0.1358
196 0.5300 0.7158 0.7908 0.8906 0.9583 0.7850 0.1597 0.5000 0.6250 0.6667 0.8333 0.8333 0.7000 0.1394
197 0.5549 0.5550 0.5625 0.5799 0.5831 0.5669 0.0133 0.5082 0.5512 0.5656 0.5799 0.5984 0.5623 0.0331
198 0.0001 0.0001 0.0003 0.0011 0.0017 0.0006 0.0007 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000
199 0.7056 0.7189 0.7542 0.7570 0.7618 0.7401 0.0244 0.7083 0.7396 0.7500 0.7552 0.7708 0.7458 0.0228
200 0.0093 0.0314 0.0985 0.1329 0.1589 0.0860 0.0613 0.0152 0.0265 0.1818 0.1894 0.2121 0.1242 0.0936
ALL 0.2524 0.2818 0.3213 0.3228 0.3244 0.3024 0.0310 0.2640 0.2921 0.3315 0.3317 0.3320 0.3121 0.0299
83
84
AH-ROBUST-CLEF2006 Track Overview Results and Graphs AH-ROBUST-BILI-X2ES-CLEF2006
Ad−Hoc Robust Bilingual Spanish track Top 5 Participants − Interpolated Recall vs Average Precision
100%
reina [reinaIT2EStdtest; MAP 36.93%; Not Pooled]
dcu [dcuitqydescsp12075; MAP 33.22%; Not Pooled]
90%
daedalus [esFSesRLit2S; MAP 26.89%; Not Pooled]
80%
70%
Average Precision
60%
50%
40%
30%
20%
10%
0%
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
Interpolated Recall
Ad−Hoc Robust Bilingual Spanish track Top 5 Participants − Comparison to Median Mean Average Precision by Topic (Topics 041ot 085)
1
reina [reinaIT2EStdtest; MAP 36.93%; Not Pooled]
dcu [dcuitqydescsp12075; MAP 33.22%; Not Pooled]
0.5 daedalus [esFSesRLit2S; MAP 26.89%; Not Pooled]
Difference
0
−0.5
−1
041 042 043 044 045 046 047 048 049 060 061 062 063 064 065 066 067 068 069 080 081 082 083 084 085
Topic Identifier
Ad−Hoc Robust Bilingual Spanish track Top 5 Participants − Comparison to Median Mean Average Precision by Topic (Topics 086ot 130)
1
reina [reinaIT2EStdtest; MAP 36.93%; Not Pooled]
dcu [dcuitqydescsp12075; MAP 33.22%; Not Pooled]
0.5 daedalus [esFSesRLit2S; MAP 26.89%; Not Pooled]
Difference
0
−0.5
−1
086 087 088 089 090 091 092 093 094 095 096 097 098 099 110 111 112 113 114 115 116 117 118 119 130
Topic Identifier
Ad−Hoc Robust Bilingual Spanish track Top 5 Participants − Comparison to Median Mean Average Precision by Topic (Topics 131ot 165)
1
reina [reinaIT2EStdtest; MAP 36.93%; Not Pooled]
dcu [dcuitqydescsp12075; MAP 33.22%; Not Pooled]
0.5 daedalus [esFSesRLit2S; MAP 26.89%; Not Pooled]
Difference
0
−0.5
−1
131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 160 161 162 163 164 165
Topic Identifier
Ad−Hoc Robust Bilingual Spanish track Top 5 Participants − Comparison to Median Mean Average Precision by Topic (Topics 166ot 200)
1
reina [reinaIT2EStdtest; MAP 36.93%; Not Pooled]
dcu [dcuitqydescsp12075; MAP 33.22%; Not Pooled]
0.5 daedalus [esFSesRLit2S; MAP 26.89%; Not Pooled]
Difference
0
−0.5
−1
166 167 168 169 170 171 172 173 174 175 176 177 178 179 190 191 192 193 194 195 196 197 198 199 200
Topic Identifier
85
AH-ROBUST-CLEF2006 Track Overview Results and Graphs AH-ROBUST-BILI-X2ES-CLEF2006
Ad−Hoc Robust Bilingual Spanish track − Box Plot of the Topics
reinaIT2EStdtest [MAP 36.93%; Not Pooled]
dcuitqynarrsp1005 [MAP 34.65%; Not Pooled]
dcuitqydescsp12075 [MAP 33.22%; Not Pooled]
esx011esRLitFW3FS3 [MAP 30.68%; Not Pooled]
Experiments
esx021esRLitFW3FS3 [MAP 30.42%; Not Pooled]
reinaIT2ESttest [MAP 30.39%; Not Pooled]
esFSesRLit3S [MAP 30.37%; Not Pooled]
esFSesRLit2S [MAP 26.89%; Not Pooled]
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
Mean Average Precision
86
AH-ROBUST-CLEF2006 Track Overview Results and Graphs AH-ROBUST-BILI-X2ES-CLEF2006
Ad−Hoc Robust Bilingual Spanish track − Tukey T test with "top group" highlighted
reinaIT2EStdtest
dcuitqynarrsp1005
dcuitqydescsp12075
Experiments
esx011esRLitFW3FS3
esx021esRLitFW3FS3
esFSesRLit3S
reinaIT2ESttest
esFSesRLit2S
0.4 0.45 0.5 0.55 0.6 0.65 0.7 0.75
arcsin(sqrt(Mean average precision))
87
AH-ROBUST-CLEF2006 Track Overview Results and Graphs AH-ROBUST-BILI-X2ES-CLEF2006
Ad−Hoc Robust Bilingual Spanish track Top 5 Participants − Retrieved documents vs Precision
100%
reina [reinaIT2EStdtest; R−Prec 36.73%; Not Pooled]
dcu [dcuitqydescsp12075; R−Prec 33.55%; Not Pooled]
90%
daedalus [esFSesRLit2S; R−Prec 28.75%; Not Pooled]
80%
70%
60%
R−Precision
50%
40%
30%
20%
10%
0%
5 10 15 20 30 100 200 500 1000
Retrieved Documents (logarithmic scale)
Ad−Hoc Robust Bilingual Spanish track Top 5 Participants − Comparison to Median R−Precision by Topic (Topics 041 to 085)
1
reina [reinaIT2EStdtest; R−Prec 36.73%; Not Pooled]
dcu [dcuitqydescsp12075; R−Prec 33.55%; Not Pooled]
0.5 daedalus [esFSesRLit2S; R−Prec 28.75%; Not Pooled]
Difference
0
−0.5
−1
041 042 043 044 045 046 047 048 049 060 061 062 063 064 065 066 067 068 069 080 081 082 083 084 085
Topic Identifier
Ad−Hoc Robust Bilingual Spanish track Top 5 Participants − Comparison to Median R−Precision by Topic (Topics 086 to 130)
1
reina [reinaIT2EStdtest; R−Prec 36.73%; Not Pooled]
dcu [dcuitqydescsp12075; R−Prec 33.55%; Not Pooled]
0.5 daedalus [esFSesRLit2S; R−Prec 28.75%; Not Pooled]
Difference
0
−0.5
−1
086 087 088 089 090 091 092 093 094 095 096 097 098 099 110 111 112 113 114 115 116 117 118 119 130
Topic Identifier
Ad−Hoc Robust Bilingual Spanish track Top 5 Participants − Comparison to Median R−Precision by Topic (Topics 131 to 165)
1
reina [reinaIT2EStdtest; R−Prec 36.73%; Not Pooled]
dcu [dcuitqydescsp12075; R−Prec 33.55%; Not Pooled]
0.5 daedalus [esFSesRLit2S; R−Prec 28.75%; Not Pooled]
Difference
0
−0.5
−1
131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 160 161 162 163 164 165
Topic Identifier
Ad−Hoc Robust Bilingual Spanish track Top 5 Participants − Comparison to Median R−Precision by Topic (Topics 166 to 200)
1
reina [reinaIT2EStdtest; R−Prec 36.73%; Not Pooled]
dcu [dcuitqydescsp12075; R−Prec 33.55%; Not Pooled]
0.5 daedalus [esFSesRLit2S; R−Prec 28.75%; Not Pooled]
Difference
0
−0.5
−1
166 167 168 169 170 171 172 173 174 175 176 177 178 179 190 191 192 193 194 195 196 197 198 199 200
Topic Identifier
88
AH-ROBUST-CLEF2006 Track Overview Results and Graphs AH-ROBUST-BILI-X2ES-CLEF2006
Ad−Hoc Robust Bilingual Spanish track − Box Plot of the Topics
reinaIT2EStdtest [R−Prec 36.73%; Not Pooled]
dcuitqynarrsp1005 [R−Prec 34.17%; Not Pooled]
dcuitqydescsp12075 [R−Prec 33.55%; Not Pooled]
reinaIT2ESttest [R−Prec 31.64%; Not Pooled]
Experiments
esx011esRLitFW3FS3 [R−Prec 31.50%; Not Pooled]
esx021esRLitFW3FS3 [R−Prec 31.39%; Not Pooled]
esFSesRLit3S [R−Prec 31.00%; Not Pooled]
esFSesRLit2S [R−Prec 28.75%; Not Pooled]
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
R−Precision
89
AH-ROBUST-CLEF2006 Track Overview Results and Graphs AH-ROBUST-BILI-X2ES-CLEF2006
Ad−Hoc Robust Bilingual Spanish track − Tukey T test with "top group" highlighted
reinaIT2EStdtest
dcuitqynarrsp1005
dcuitqydescsp12075
Experiments
esx011esRLitFW3FS3
esx021esRLitFW3FS3
esFSesRLit3S
reinaIT2ESttest
esFSesRLit2S
0.46 0.48 0.5 0.52 0.54 0.56 0.58 0.6 0.62 0.64 0.66
arcsin(sqrt(R Precision))
90
AH-ROBUST-CLEF2006 Track Overview Results and Graphs AH-ROBUST-BILI-X2ES-CLEF2006
Average Precision R-Precision
Topic Minimum 1st Q. Median 3rd Q. Maximum Mean Std Minimum 1st Q. Median 3rd Q. Maximum Mean Std
041 0.0010 0.0048 0.0069 0.0499 0.1895 0.0392 0.0655 0.0000 0.0000 0.0312 0.0938 0.3750 0.0781 0.1283
042 0.1829 0.3525 0.3731 0.4370 0.5065 0.3768 0.0956 0.2797 0.3985 0.4215 0.4713 0.4981 0.4200 0.0694
043 0.0428 0.0952 0.2150 0.4588 0.4588 0.2549 0.1823 0.0556 0.0833 0.2222 0.5000 0.5000 0.2708 0.2024
044 0.1054 0.1106 0.1315 0.1755 0.2568 0.1497 0.0514 0.1500 0.2000 0.2250 0.2375 0.2500 0.2156 0.0352
045 0.5068 0.5939 0.6188 0.6564 0.6900 0.6169 0.0569 0.4921 0.5759 0.6126 0.6649 0.6911 0.6113 0.0673
046 0.0013 0.0015 0.0039 0.0297 0.0907 0.0203 0.0312 0.0000 0.0000 0.0000 0.0000 0.2000 0.0250 0.0707
047 0.0040 0.0338 0.0419 0.0831 0.1097 0.0539 0.0371 0.0278 0.0556 0.0787 0.1528 0.1852 0.0984 0.0590
048 0.0104 0.0801 0.0885 0.0988 0.1116 0.0821 0.0316 0.0104 0.1406 0.1562 0.1667 0.1667 0.1380 0.0533
049 0.0280 0.0690 0.0944 0.1209 0.1552 0.0940 0.0402 0.0526 0.1053 0.1053 0.1579 0.2105 0.1250 0.0558
060 0.2098 0.2518 0.3636 0.4299 0.4830 0.3479 0.1051 0.3200 0.3333 0.4333 0.4800 0.5333 0.4183 0.0843
061 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000
062 0.2102 0.2138 0.2496 0.2952 0.3412 0.2586 0.0502 0.1667 0.2000 0.2000 0.3000 0.3333 0.2375 0.0653
063 0.6550 0.7374 0.7475 0.8014 0.8373 0.7581 0.0560 0.6441 0.6949 0.7034 0.7542 0.7797 0.7161 0.0460
064 0.0035 0.1733 1.0000 1.0000 1.0000 0.6688 0.4680 0.0000 0.0000 1.0000 1.0000 1.0000 0.6250 0.5175
065 0.0155 0.1049 0.2108 0.2321 0.2392 0.1688 0.0859 0.0000 0.2162 0.3784 0.4189 0.4189 0.3057 0.1534
066 0.1995 0.2984 0.3193 0.5938 0.8533 0.4345 0.2608 0.2929 0.3535 0.3939 0.5808 0.7879 0.4672 0.1954
067 0.0496 0.0507 0.0666 0.1945 0.4758 0.1436 0.1497 0.0612 0.0816 0.1020 0.2551 0.5510 0.1862 0.1677
068 0.0000 0.0000 0.0003 0.0030 0.0658 0.0091 0.0230 0.0000 0.0000 0.0000 0.0000 0.2222 0.0278 0.0786
069 0.0005 0.0044 0.0150 0.0164 0.0198 0.0115 0.0073 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000
080 0.3640 0.4137 0.4721 0.4897 0.4970 0.4515 0.0516 0.4163 0.4449 0.5163 0.5306 0.5347 0.4918 0.0510
081 0.1349 0.4997 0.7161 0.7487 0.7583 0.6028 0.2274 0.2529 0.4943 0.6609 0.6897 0.7241 0.5833 0.1722
082 0.5471 0.5647 0.7171 0.8259 0.8724 0.7044 0.1381 0.5789 0.5789 0.6842 0.7895 0.8421 0.6908 0.1087
083 0.3079 0.3080 0.3497 0.4621 0.5351 0.3853 0.0962 0.3333 0.3333 0.3889 0.5000 0.5556 0.4167 0.0985
084 0.0000 0.0000 0.0008 0.0685 0.2917 0.0538 0.1066 0.0000 0.0000 0.0000 0.1250 0.3750 0.0781 0.1485
085 0.4621 0.4676 0.4830 0.6672 0.8214 0.5649 0.1532 0.4206 0.4346 0.4393 0.6589 0.7664 0.5315 0.1488
086 0.1136 0.2223 0.2494 0.2630 0.2995 0.2353 0.0551 0.2321 0.2857 0.3214 0.3393 0.3393 0.3080 0.0379
087 0.3690 0.4259 0.4597 0.6109 0.6433 0.5007 0.1065 0.4205 0.4432 0.4659 0.5852 0.6250 0.5043 0.0813
088 0.0000 0.1140 0.2156 0.3063 0.3621 0.2042 0.1312 0.0000 0.0714 0.1429 0.2143 0.2857 0.1429 0.1080
089 0.1862 0.4499 0.4787 0.5059 0.5250 0.4475 0.1094 0.3125 0.5000 0.5625 0.5625 0.5625 0.5156 0.0930
090 0.0062 0.0074 0.0102 0.0723 0.0861 0.0340 0.0358 0.0000 0.0000 0.0000 0.0667 0.0667 0.0250 0.0345
091 0.0000 0.0584 0.1559 0.2301 0.2339 0.1403 0.1002 0.0000 0.1183 0.2097 0.2903 0.2903 0.1909 0.1117
092 0.0769 0.1074 0.1238 0.2100 0.2313 0.1488 0.0596 0.1628 0.1628 0.2035 0.2558 0.2907 0.2122 0.0545
093 0.1574 0.2040 0.2198 0.2485 0.3382 0.2300 0.0533 0.2143 0.2143 0.2857 0.3214 0.4286 0.2857 0.0764
094 0.0000 0.1448 0.3119 0.4464 0.6399 0.3057 0.2239 0.0000 0.1591 0.3182 0.5682 0.7273 0.3523 0.2703
095 0.0263 0.0694 0.0883 0.1061 0.2073 0.0952 0.0529 0.0903 0.1807 0.2056 0.2290 0.3178 0.2048 0.0651
096 0.0078 0.0078 0.0112 0.0218 0.0597 0.0186 0.0178 0.0108 0.0108 0.0269 0.0860 0.1290 0.0484 0.0456
097 0.6699 0.6815 0.6832 0.7100 0.7817 0.7001 0.0382 0.6000 0.6200 0.6600 0.6800 0.7200 0.6550 0.0424
098 0.3541 0.4632 0.4743 0.6502 0.8054 0.5418 0.1571 0.4000 0.4000 0.4000 0.5000 0.8000 0.4750 0.1488
099 0.0062 0.0230 0.1641 0.2654 0.2878 0.1499 0.1235 0.0000 0.0536 0.2143 0.3571 0.3571 0.2009 0.1514
110 0.0007 0.6131 0.6926 0.7149 0.7149 0.5946 0.2440 0.0000 0.5714 0.5714 0.5714 0.5714 0.5000 0.2020
111 0.0027 0.0032 0.0036 0.0153 0.0220 0.0086 0.0080 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000
112 0.1057 0.2989 0.3977 0.4179 0.5438 0.3598 0.1381 0.0000 0.3125 0.3750 0.4375 0.5000 0.3438 0.1602
113 0.0000 0.0002 0.0102 0.0128 0.0173 0.0079 0.0070 0.0000 0.0032 0.0633 0.0633 0.0886 0.0435 0.0354
114 0.0299 0.0806 0.4303 0.5414 0.6509 0.3482 0.2528 0.0222 0.0333 0.4444 0.5000 0.5778 0.3194 0.2454
115 0.1805 0.1814 0.1961 0.2066 0.2188 0.1959 0.0150 0.2105 0.2105 0.2105 0.2632 0.3158 0.2368 0.0398
116 0.1093 0.1300 0.2145 0.3037 0.4522 0.2322 0.1263 0.0000 0.0714 0.1429 0.2857 0.4286 0.1786 0.1664
117 0.0442 0.0442 0.0641 0.1119 0.1841 0.0836 0.0528 0.0714 0.0714 0.0804 0.1786 0.2321 0.1205 0.0713
118 0.0069 0.0155 0.0231 0.3129 0.4286 0.1423 0.1776 0.0000 0.0000 0.0000 0.2000 0.4000 0.1000 0.1512
119 0.0000 0.0000 0.0362 0.1319 0.3607 0.0871 0.1250 0.0000 0.0000 0.0857 0.2190 0.3810 0.1238 0.1424
130 0.1558 0.6762 0.7012 0.7775 0.8099 0.6594 0.2106 0.3077 0.5385 0.5769 0.6154 0.6923 0.5577 0.1145
91
AH-ROBUST-CLEF2006 Track Overview Results and Graphs AH-ROBUST-BILI-X2ES-CLEF2006
Average Precision R-Precision
Topic Minimum 1st Q. Median 3rd Q. Maximum Mean Std Minimum 1st Q. Median 3rd Q. Maximum Mean Std
131 0.0022 0.0064 0.0106 0.0240 0.1585 0.0303 0.0529 0.0091 0.0318 0.0455 0.0818 0.3000 0.0784 0.0943
132 0.0029 0.2333 0.4865 0.5682 0.6230 0.4002 0.2506 0.0000 0.2115 0.5192 0.5769 0.5769 0.3990 0.2517
133 0.0134 0.0562 0.0818 0.1558 0.2960 0.1121 0.0974 0.0000 0.1579 0.1579 0.2632 0.3684 0.1908 0.1123
134 0.8088 0.8788 1.0000 1.0000 1.0000 0.9458 0.0804 0.7500 0.7500 1.0000 1.0000 1.0000 0.9062 0.1294
135 0.0219 0.0232 0.1031 0.4456 0.7109 0.2346 0.2916 0.0897 0.0962 0.1955 0.4808 0.6410 0.2845 0.2378
136 0.3441 0.3679 0.3848 0.4446 0.5545 0.4117 0.0732 0.5000 0.5000 0.5000 0.5000 0.6250 0.5156 0.0442
137 0.0126 0.0562 0.0960 0.1271 0.1615 0.0916 0.0490 0.0000 0.1000 0.1000 0.1500 0.2000 0.1125 0.0641
138 0.6019 0.8735 0.9683 0.9841 1.0000 0.9067 0.1331 0.5714 0.7143 0.8571 0.9286 1.0000 0.8214 0.1479
139 0.0000 0.0001 0.0011 0.0092 0.0185 0.0049 0.0072 0.0057 0.0057 0.0057 0.0115 0.0230 0.0093 0.0061
140 0.0286 0.0952 0.1069 0.1969 0.2667 0.1367 0.0830 0.0840 0.1176 0.1471 0.2773 0.3613 0.1912 0.1074
141 0.5148 0.8582 0.8662 0.9065 0.9457 0.8403 0.1359 0.5385 0.8462 0.8462 0.8462 0.9231 0.8173 0.1158
142 0.6622 0.6839 0.7612 0.8758 0.9025 0.7758 0.1062 0.6786 0.6786 0.7143 0.8036 0.8571 0.7411 0.0733
143 0.1825 0.2248 0.2706 0.4845 0.5853 0.3410 0.1583 0.2431 0.3264 0.3715 0.4757 0.5347 0.3906 0.1039
144 0.0545 0.0621 0.1322 0.3569 0.5986 0.2195 0.2010 0.0909 0.0909 0.1818 0.5227 0.5909 0.2841 0.2263
145 0.3985 0.5403 0.5796 0.5972 0.5972 0.5537 0.0675 0.4043 0.4894 0.5319 0.5319 0.5532 0.5080 0.0475
146 0.0734 0.2240 0.4003 0.5476 0.5476 0.3706 0.1911 0.0000 0.2500 0.5000 0.5000 0.5000 0.3750 0.2315
147 0.0215 0.0226 0.0349 0.0866 0.1289 0.0548 0.0439 0.0000 0.1111 0.1111 0.1111 0.1111 0.0972 0.0393
148 0.1245 0.1367 0.1410 0.2085 0.2855 0.1728 0.0612 0.0909 0.0909 0.0909 0.2273 0.3636 0.1591 0.1059
149 0.2287 0.5612 0.6374 0.6933 0.7940 0.6008 0.1717 0.2308 0.5769 0.6154 0.6538 0.6923 0.5769 0.1482
160 0.0000 0.0010 0.1329 0.2242 0.5082 0.1531 0.1733 0.0000 0.0000 0.0000 0.2500 0.5000 0.1250 0.2315
161 0.0005 0.0044 0.2000 0.2003 0.2008 0.1263 0.1021 0.0000 0.0000 0.2000 0.2000 0.2000 0.1250 0.1035
162 0.7914 0.7914 0.8320 0.8850 0.8938 0.8378 0.0452 0.7442 0.7442 0.7752 0.8062 0.8295 0.7781 0.0336
163 0.1564 0.4009 0.5021 0.6361 0.8097 0.5055 0.2113 0.2333 0.4333 0.4333 0.5667 0.7333 0.4792 0.1501
164 0.0022 0.0022 0.0187 0.1304 0.2791 0.0730 0.1093 0.0000 0.0139 0.0347 0.1528 0.3611 0.0955 0.1345
165 0.2321 0.3237 0.4020 0.6273 0.7376 0.4595 0.1839 0.3333 0.3333 0.4444 0.6111 0.6667 0.4722 0.1543
166 0.0094 0.0125 0.0362 0.0924 0.0961 0.0484 0.0389 0.0000 0.0000 0.0000 0.1429 0.1429 0.0536 0.0739
167 0.4343 0.4343 0.4505 0.4939 0.5857 0.4721 0.0544 0.3000 0.4000 0.4000 0.4500 0.5000 0.4125 0.0641
168 0.1024 0.1309 0.1375 0.1594 0.2328 0.1488 0.0392 0.1282 0.1795 0.1795 0.2179 0.2564 0.1923 0.0388
169 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000
170 0.1128 0.1546 0.2534 0.8369 0.8833 0.4357 0.3500 0.2000 0.2000 0.3000 0.7000 0.8000 0.4250 0.2712
171 0.0567 0.1044 0.1127 0.1536 0.4017 0.1500 0.1063 0.0000 0.1250 0.2500 0.2500 0.5000 0.2188 0.1602
172 0.0219 0.0219 0.0316 0.0557 0.0747 0.0394 0.0215 0.0000 0.0000 0.0769 0.0769 0.1538 0.0577 0.0544
173 0.4123 0.6231 0.6474 0.6802 0.7435 0.6322 0.0980 0.3333 0.3333 0.5000 0.5000 0.6667 0.4583 0.1179
174 0.3431 0.5917 0.7328 0.8852 0.9191 0.7102 0.2099 0.3636 0.5455 0.6818 0.8636 0.9091 0.6818 0.2062
175 1.0000 1.0000 1.0000 1.0000 1.0000 1.0000 0.0000 1.0000 1.0000 1.0000 1.0000 1.0000 1.0000 0.0000
176 0.7043 0.8074 0.8879 0.9796 0.9869 0.8801 0.1104 0.6883 0.7532 0.8377 0.9351 0.9481 0.8360 0.1084
177 0.0000 0.0000 0.0022 0.0112 0.0719 0.0123 0.0246 0.0000 0.0000 0.0000 0.0217 0.1304 0.0217 0.0465
178 0.0093 0.0252 0.0320 0.1030 0.2634 0.0741 0.0926 0.0341 0.0739 0.1023 0.1932 0.3523 0.1406 0.1135
179 0.0149 0.0624 0.1632 0.3338 0.5573 0.2114 0.2053 0.0354 0.0841 0.2212 0.3805 0.5841 0.2489 0.2084
190 0.0024 0.0063 0.0124 0.0140 0.0149 0.0103 0.0051 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000
191 0.0476 0.0476 0.1096 0.4167 1.0000 0.2744 0.3372 0.0000 0.0000 0.0000 0.0000 1.0000 0.1250 0.3536
192 0.2686 0.3172 0.5628 0.5784 0.6458 0.4789 0.1516 0.2955 0.4205 0.5455 0.5568 0.6136 0.4943 0.1071
193 0.0607 0.0744 0.0906 0.1121 0.2232 0.1048 0.0514 0.0882 0.1029 0.1176 0.1618 0.2941 0.1434 0.0711
194 0.0066 0.0694 0.1029 0.1735 0.4480 0.1433 0.1355 0.0000 0.1250 0.1250 0.1875 0.5000 0.1719 0.1485
195 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000
196 0.6279 0.6881 0.7231 0.7422 0.7574 0.7115 0.0433 0.6000 0.6667 0.7000 0.7333 0.7333 0.6917 0.0496
197 0.4407 0.4879 0.5270 0.6442 0.7302 0.5612 0.1026 0.4792 0.5000 0.5042 0.5896 0.6667 0.5417 0.0686
198 0.1091 0.2875 0.4106 0.4785 0.5275 0.3737 0.1442 0.1250 0.3750 0.3750 0.4375 0.5000 0.3750 0.1157
199 0.8334 0.8602 0.8770 0.8792 0.8833 0.8687 0.0177 0.7765 0.8059 0.8235 0.8235 0.8235 0.8132 0.0171
200 0.0002 0.0236 0.3415 0.4487 0.5230 0.2689 0.2196 0.0000 0.0426 0.4468 0.4468 0.5319 0.3005 0.2279
ALL 0.2689 0.3038 0.3055 0.3393 0.3693 0.3169 0.0311 0.2875 0.3119 0.3157 0.3386 0.3673 0.3234 0.0242
92
93
AH-ROBUST-CLEF2006 Track Overview Results and Graphs AH-ROBUST-BILI-X2NL-CLEF2006
Ad−Hoc Robust Bilingual Dutch track Top 5 Participants − Interpolated Recall vs Average Precision
100%
daedalus [nlFSnlRLfr2S; MAP 35.37%; Not Pooled]
90%
80%
70%
Average Precision
60%
50%
40%
30%
20%
10%
0%
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
Interpolated Recall
Ad−Hoc Robust Bilingual Dutch track Top 5 Participants − Comparison to Median Mean Average Precision by Topic (Topics 041 to085)
1
daedalus [nlFSnlRLfr2S; MAP 35.37%; Not Pooled]
0.5
Difference
0
−0.5
−1
041 042 043 044 045 046 047 048 049 060 061 062 063 064 065 066 067 068 069 080 081 082 083 084 085
Topic Identifier
Ad−Hoc Robust Bilingual Dutch track Top 5 Participants − Comparison to Median Mean Average Precision by Topic (Topics 086 to130)
1
daedalus [nlFSnlRLfr2S; MAP 35.37%; Not Pooled]
0.5
Difference
0
−0.5
−1
086 087 088 089 090 091 092 093 094 095 096 097 098 099 110 111 112 113 114 115 116 117 118 119 130
Topic Identifier
Ad−Hoc Robust Bilingual Dutch track Top 5 Participants − Comparison to Median Mean Average Precision by Topic (Topics 131 to165)
1
daedalus [nlFSnlRLfr2S; MAP 35.37%; Not Pooled]
0.5
Difference
0
−0.5
−1
131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 160 161 162 163 164 165
Topic Identifier
Ad−Hoc Robust Bilingual Dutch track Top 5 Participants − Comparison to Median Mean Average Precision by Topic (Topics 166 to200)
1
daedalus [nlFSnlRLfr2S; MAP 35.37%; Not Pooled]
0.5
Difference
0
−0.5
−1
166 167 168 169 170 171 172 173 174 175 176 177 178 179 190 191 192 193 194 195 196 197 198 199 200
Topic Identifier
94
AH-ROBUST-CLEF2006 Track Overview Results and Graphs AH-ROBUST-BILI-X2NL-CLEF2006
Ad−Hoc Robust Bilingual Dutch track − Box Plot of the Topics
nlFSnlRLfr6S [MAP 37.20%; Not Pooled]
nlx011nlRLfrFW4FS6 [MAP 36.99%; Not Pooled]
Experiments
nlx021nlRLfrFW4FS6 [MAP 36.92%; Not Pooled]
nlFSnlRLfr2S [MAP 35.37%; Not Pooled]
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
Mean Average Precision
95
AH-ROBUST-CLEF2006 Track Overview Results and Graphs AH-ROBUST-BILI-X2NL-CLEF2006
Ad−Hoc Robust Bilingual Dutch track − Tukey T test with "top group" highlighted
nlFSnlRLfr6S
nlx011nlRLfrFW4FS6
Experiments
nlx021nlRLfrFW4FS6
nlFSnlRLfr2S
0.58 0.59 0.6 0.61 0.62 0.63 0.64 0.65
arcsin(sqrt(Mean average precision))
96
AH-ROBUST-CLEF2006 Track Overview Results and Graphs AH-ROBUST-BILI-X2NL-CLEF2006
Ad−Hoc Robust Bilingual Dutch track Top 5 Participants − Retrieved documents vs Precision
100%
daedalus [nlFSnlRLfr2S; R−Prec 34.18%; Not Pooled]
90%
80%
70%
60%
R−Precision
50%
40%
30%
20%
10%
0%
5 10 15 20 30 100 200 500 1000
Retrieved Documents (logarithmic scale)
Ad−Hoc Robust Bilingual Dutch track Top 5 Participants − Comparison to Median R−Precision by Topic (Topics 041 to 085)
1
daedalus [nlFSnlRLfr2S; R−Prec 34.18%; Not Pooled]
0.5
Difference
0
−0.5
−1
041 042 043 044 045 046 047 048 049 060 061 062 063 064 065 066 067 068 069 080 081 082 083 084 085
Topic Identifier
Ad−Hoc Robust Bilingual Dutch track Top 5 Participants − Comparison to Median R−Precision by Topic (Topics 086 to 130)
1
daedalus [nlFSnlRLfr2S; R−Prec 34.18%; Not Pooled]
0.5
Difference
0
−0.5
−1
086 087 088 089 090 091 092 093 094 095 096 097 098 099 110 111 112 113 114 115 116 117 118 119 130
Topic Identifier
Ad−Hoc Robust Bilingual Dutch track Top 5 Participants − Comparison to Median R−Precision by Topic (Topics 131 to 165)
1
daedalus [nlFSnlRLfr2S; R−Prec 34.18%; Not Pooled]
0.5
Difference
0
−0.5
−1
131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 160 161 162 163 164 165
Topic Identifier
Ad−Hoc Robust Bilingual Dutch track Top 5 Participants − Comparison to Median R−Precision by Topic (Topics 166 to 200)
1
daedalus [nlFSnlRLfr2S; R−Prec 34.18%; Not Pooled]
0.5
Difference
0
−0.5
−1
166 167 168 169 170 171 172 173 174 175 176 177 178 179 190 191 192 193 194 195 196 197 198 199 200
Topic Identifier
97
AH-ROBUST-CLEF2006 Track Overview Results and Graphs AH-ROBUST-BILI-X2NL-CLEF2006
Ad−Hoc Robust Bilingual Dutch track − Box Plot of the Topics
nlx011nlRLfrFW4FS6 [R−Prec 36.13%; Not Pooled]
nlx021nlRLfrFW4FS6 [R−Prec 36.13%; Not Pooled]
Experiments
nlFSnlRLfr6S [R−Prec 35.63%; Not Pooled]
nlFSnlRLfr2S [R−Prec 34.18%; Not Pooled]
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
R−Precision
98
AH-ROBUST-CLEF2006 Track Overview Results and Graphs AH-ROBUST-BILI-X2NL-CLEF2006
Ad−Hoc Robust Bilingual Dutch track − Tukey T test with "top group" highlighted
nlx021nlRLfrFW4FS6
nlx011nlRLfrFW4FS6
Experiments
nlFSnlRLfr6S
nlFSnlRLfr2S
0.55 0.56 0.57 0.58 0.59 0.6 0.61 0.62 0.63
arcsin(sqrt(R Precision))
99
AH-ROBUST-CLEF2006 Track Overview Results and Graphs AH-ROBUST-BILI-X2NL-CLEF2006
Average Precision R-Precision
Topic Minimum 1st Q. Median 3rd Q. Maximum Mean Std Minimum 1st Q. Median 3rd Q. Maximum Mean Std
041 0.0452 0.0462 0.0473 0.0473 0.0473 0.0468 0.0011 0.0435 0.0435 0.0435 0.0435 0.0435 0.0435 0.0000
042 0.1434 0.1434 0.1434 0.1486 0.1538 0.1460 0.0052 0.2500 0.2500 0.2500 0.2500 0.2500 0.2500 0.0000
043 0.4725 0.4725 0.4725 0.4946 0.5167 0.4836 0.0221 0.2500 0.2500 0.2500 0.2500 0.2500 0.2500 0.0000
044 0.2775 0.2823 0.2870 0.3037 0.3203 0.2930 0.0188 0.3333 0.3667 0.4000 0.4000 0.4000 0.3833 0.0333
045 0.3881 0.3887 0.3935 0.3977 0.3977 0.3932 0.0052 0.3500 0.3500 0.3500 0.3625 0.3750 0.3562 0.0125
046 0.1366 0.1366 0.1366 0.1546 0.1725 0.1456 0.0180 0.1667 0.1667 0.1667 0.1979 0.2292 0.1823 0.0312
047 0.0512 0.0759 0.1007 0.1007 0.1007 0.0883 0.0247 0.0484 0.0968 0.1452 0.1452 0.1452 0.1210 0.0484
048 0.0238 0.0314 0.0390 0.0390 0.0390 0.0352 0.0076 0.1091 0.1091 0.1091 0.1091 0.1091 0.1091 0.0000
049 0.0584 0.0584 0.0584 0.0635 0.0685 0.0609 0.0051 0.0000 0.0000 0.0000 0.0357 0.0714 0.0179 0.0357
060 0.0819 0.0835 0.0851 0.1510 0.2168 0.1172 0.0664 0.1538 0.1538 0.1538 0.2308 0.3077 0.1923 0.0769
061 0.2560 0.2576 0.2665 0.2737 0.2737 0.2657 0.0094 0.2609 0.2826 0.3043 0.3043 0.3043 0.2935 0.0217
062 0.5719 0.5849 0.5979 0.5979 0.5979 0.5914 0.0130 0.5000 0.5250 0.5500 0.5500 0.5500 0.5375 0.0250
063 0.1050 0.1365 0.1681 0.1681 0.1681 0.1523 0.0315 0.2000 0.2000 0.2000 0.2000 0.2000 0.2000 0.0000
064 0.7500 0.7500 0.7500 0.7500 0.7500 0.7500 0.0000 0.7500 0.7500 0.7500 0.7500 0.7500 0.7500 0.0000
065 0.0160 0.0186 0.0212 0.0212 0.0212 0.0199 0.0026 0.0725 0.0797 0.0870 0.0870 0.0870 0.0833 0.0072
066 0.7237 0.7237 0.7326 0.7448 0.7481 0.7342 0.0125 0.6429 0.6429 0.6429 0.6607 0.6786 0.6518 0.0179
067 0.0111 0.0127 0.0142 0.0148 0.0154 0.0137 0.0019 0.0500 0.0500 0.0500 0.0500 0.0500 0.0500 0.0000
068 0.2153 0.2244 0.2426 0.2561 0.2605 0.2402 0.0201 0.3043 0.3043 0.3043 0.3043 0.3043 0.3043 0.0000
069 0.0623 0.0706 0.0788 0.0788 0.0788 0.0747 0.0082 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000
080 0.1927 0.4501 0.7119 0.7267 0.7373 0.5884 0.2641 0.1828 0.4462 0.7097 0.7097 0.7097 0.5780 0.2634
081 0.6670 0.7004 0.7337 0.7337 0.7337 0.7170 0.0334 0.6667 0.7037 0.7407 0.7407 0.7407 0.7222 0.0370
082 0.0032 0.0075 0.0118 0.0118 0.0118 0.0097 0.0043 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000
083 0.0443 0.0465 0.0486 0.0486 0.0486 0.0476 0.0022 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000
084 0.5099 0.5099 0.5099 0.5253 0.5407 0.5176 0.0154 0.6000 0.6000 0.6000 0.6000 0.6000 0.6000 0.0000
085 0.5508 0.5508 0.5508 0.5641 0.5774 0.5574 0.0133 0.6111 0.6111 0.6111 0.6111 0.6111 0.6111 0.0000
086 0.3383 0.3752 0.4121 0.4121 0.4121 0.3936 0.0369 0.4000 0.4200 0.4400 0.4400 0.4400 0.4300 0.0200
087 0.6764 0.6896 0.7027 0.7027 0.7027 0.6962 0.0131 0.4615 0.4615 0.4615 0.4615 0.4615 0.4615 0.0000
088 0.3607 0.3622 0.3637 0.3637 0.3637 0.3630 0.0015 0.2500 0.2500 0.2500 0.2500 0.2500 0.2500 0.0000
089 0.9395 0.9395 0.9395 0.9422 0.9449 0.9409 0.0027 0.8605 0.8721 0.8837 0.8837 0.8837 0.8779 0.0116
090 0.3748 0.4473 0.5198 0.5230 0.5262 0.4852 0.0736 0.3846 0.4231 0.4615 0.4615 0.4615 0.4423 0.0385
091 0.0430 0.0430 0.0522 0.0930 0.1246 0.0680 0.0387 0.1111 0.1111 0.1111 0.1111 0.1111 0.1111 0.0000
092 0.3928 0.3928 0.3994 0.4185 0.4311 0.4057 0.0181 0.4737 0.4737 0.4737 0.4868 0.5000 0.4803 0.0132
093 0.7897 0.7947 0.7998 0.7998 0.7998 0.7972 0.0050 0.8000 0.8000 0.8000 0.8000 0.8000 0.8000 0.0000
094 0.0665 0.0713 0.0849 0.1116 0.1295 0.0914 0.0278 0.1579 0.1579 0.1579 0.2105 0.2632 0.1842 0.0526
095 0.2710 0.2760 0.2809 0.2809 0.2809 0.2785 0.0050 0.3555 0.3738 0.3920 0.3920 0.3920 0.3829 0.0183
096 0.0150 0.0150 0.0150 0.0205 0.0260 0.0178 0.0055 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000
097 0.8488 0.8755 0.9022 0.9022 0.9022 0.8888 0.0267 0.6875 0.7500 0.8125 0.8125 0.8125 0.7812 0.0625
098 0.8819 0.8819 0.8819 0.8819 0.8819 0.8819 0.0000 0.6667 0.6667 0.6667 0.6667 0.6667 0.6667 0.0000
099 0.2740 0.2859 0.3180 0.3381 0.3381 0.3120 0.0317 0.2500 0.2500 0.2500 0.2500 0.2500 0.2500 0.0000
110 1.0000 1.0000 1.0000 1.0000 1.0000 1.0000 0.0000 1.0000 1.0000 1.0000 1.0000 1.0000 1.0000 0.0000
111 0.0801 0.0801 0.0801 0.0817 0.0833 0.0809 0.0016 0.2143 0.2143 0.2143 0.2143 0.2143 0.2143 0.0000
112 0.5011 0.5328 0.5645 0.5645 0.5645 0.5487 0.0317 0.5000 0.5500 0.6000 0.6000 0.6000 0.5750 0.0500
113 0.0075 0.0075 0.0075 0.0079 0.0082 0.0077 0.0003 0.0303 0.0303 0.0303 0.0455 0.0606 0.0379 0.0152
114 0.2555 0.3390 0.4226 0.4226 0.4226 0.3808 0.0835 0.3137 0.3725 0.4314 0.4314 0.4314 0.4020 0.0588
115 0.1888 0.1893 0.1957 0.2113 0.2209 0.2003 0.0149 0.3077 0.3077 0.3077 0.3077 0.3077 0.3077 0.0000
116 0.6157 0.6245 0.6333 0.6333 0.6333 0.6289 0.0088 0.5517 0.5517 0.5517 0.5690 0.5862 0.5603 0.0172
117 0.0215 0.0237 0.0258 0.0258 0.0258 0.0248 0.0021 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000
118 0.2398 0.2776 0.3155 0.3155 0.3155 0.2966 0.0379 0.3636 0.4091 0.4545 0.4545 0.4545 0.4318 0.0455
119 0.5174 0.5174 0.5174 0.5646 0.6118 0.5410 0.0472 0.5484 0.5484 0.5484 0.5645 0.5806 0.5565 0.0161
130 0.5943 0.6584 0.7224 0.7224 0.7224 0.6904 0.0641 0.5556 0.5556 0.5556 0.5556 0.5556 0.5556 0.0000
100
AH-ROBUST-CLEF2006 Track Overview Results and Graphs AH-ROBUST-BILI-X2NL-CLEF2006
Average Precision R-Precision
Topic Minimum 1st Q. Median 3rd Q. Maximum Mean Std Minimum 1st Q. Median 3rd Q. Maximum Mean Std
131 0.2696 0.2762 0.2891 0.2954 0.2954 0.2858 0.0123 0.3500 0.3500 0.3500 0.3625 0.3750 0.3562 0.0125
132 0.6385 0.6536 0.6688 0.6688 0.6688 0.6612 0.0151 0.6154 0.6154 0.6154 0.6154 0.6154 0.6154 0.0000
133 0.0808 0.0998 0.1187 0.1187 0.1187 0.1092 0.0190 0.1429 0.1607 0.1786 0.1786 0.1786 0.1696 0.0179
134 1.0000 1.0000 1.0000 1.0000 1.0000 1.0000 0.0000 1.0000 1.0000 1.0000 1.0000 1.0000 1.0000 0.0000
135 0.6288 0.6288 0.6288 0.6429 0.6571 0.6359 0.0141 0.6371 0.6371 0.6371 0.6492 0.6613 0.6431 0.0121
136 1.0000 1.0000 1.0000 1.0000 1.0000 1.0000 0.0000 1.0000 1.0000 1.0000 1.0000 1.0000 1.0000 0.0000
137 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000
138 0.1203 0.2158 0.3114 0.3114 0.3114 0.2636 0.0956 0.1111 0.2222 0.3333 0.3333 0.3333 0.2778 0.1111
139 0.3924 0.3924 0.3924 0.3929 0.3935 0.3926 0.0006 0.3913 0.3913 0.3913 0.3913 0.3913 0.3913 0.0000
140 0.1770 0.1770 0.1770 0.1775 0.1780 0.1772 0.0005 0.2639 0.2639 0.2639 0.2639 0.2639 0.2639 0.0000
141 0.2518 0.2534 0.2551 0.2551 0.2551 0.2543 0.0016 0.2500 0.2500 0.2500 0.2500 0.2500 0.2500 0.0000
142 0.7089 0.7089 0.7089 0.7096 0.7102 0.7092 0.0007 0.7273 0.7273 0.7273 0.7273 0.7273 0.7273 0.0000
143 0.5377 0.5377 0.5377 0.5856 0.6335 0.5617 0.0479 0.5862 0.5862 0.5862 0.6034 0.6207 0.5948 0.0172
144 1.0000 1.0000 1.0000 1.0000 1.0000 1.0000 0.0000 1.0000 1.0000 1.0000 1.0000 1.0000 1.0000 0.0000
145 0.3031 0.3245 0.3460 0.3460 0.3460 0.3353 0.0215 0.2857 0.2857 0.2857 0.2857 0.2857 0.2857 0.0000
146 0.2917 0.2917 0.3125 0.3917 0.4500 0.3417 0.0748 0.0000 0.0000 0.0000 0.2500 0.5000 0.1250 0.2500
147 0.0057 0.0061 0.0064 0.0064 0.0064 0.0063 0.0004 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000
148 0.6514 0.6581 0.6647 0.6677 0.6707 0.6629 0.0081 0.6190 0.6190 0.6190 0.6190 0.6190 0.6190 0.0000
149 0.8362 0.8526 0.8690 0.8690 0.8690 0.8608 0.0164 0.6667 0.6667 0.6667 0.6667 0.6667 0.6667 0.0000
160 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000
161 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000
162 0.7223 0.7264 0.7306 0.7306 0.7306 0.7285 0.0042 0.6061 0.6061 0.6061 0.6212 0.6364 0.6136 0.0152
163 0.0539 0.0539 0.0551 0.0572 0.0581 0.0555 0.0021 0.2143 0.2143 0.2143 0.2143 0.2143 0.2143 0.0000
164 0.0088 0.0088 0.0088 0.0102 0.0116 0.0095 0.0014 0.0400 0.0400 0.0400 0.0400 0.0400 0.0400 0.0000
165 0.1667 0.1667 0.1667 0.1680 0.1692 0.1673 0.0013 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000
166 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000
167 0.3095 0.3512 0.5179 0.6548 0.6667 0.5030 0.1788 0.0000 0.2500 0.5000 0.5000 0.5000 0.3750 0.2500
168 0.2180 0.2565 0.2950 0.2976 0.3001 0.2771 0.0394 0.3636 0.4091 0.4545 0.4545 0.4545 0.4318 0.0455
169 0.0279 0.0279 0.0279 0.0283 0.0286 0.0281 0.0004 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000
170 0.6585 0.6793 0.7002 0.7002 0.7002 0.6897 0.0208 0.5000 0.5714 0.6429 0.6429 0.6429 0.6071 0.0714
171 0.0128 0.0128 0.0128 0.0154 0.0181 0.0141 0.0027 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000
172 0.0218 0.0229 0.0239 0.0239 0.0239 0.0234 0.0011 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000
173 0.5833 0.5833 0.7083 0.8333 0.8333 0.7083 0.1443 0.5000 0.5000 0.5000 0.5000 0.5000 0.5000 0.0000
174 0.9257 0.9403 0.9550 0.9550 0.9550 0.9477 0.0146 0.9130 0.9130 0.9130 0.9130 0.9130 0.9130 0.0000
175 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000
176 0.9089 0.9110 0.9132 0.9132 0.9132 0.9121 0.0021 0.8000 0.8000 0.8000 0.8000 0.8000 0.8000 0.0000
177 0.0780 0.0805 0.0831 0.0831 0.0831 0.0818 0.0026 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000
178 0.1369 0.1817 0.2385 0.2504 0.2504 0.2160 0.0540 0.2857 0.2857 0.2857 0.2857 0.2857 0.2857 0.0000
179 0.3996 0.4017 0.4037 0.4037 0.4037 0.4027 0.0020 0.4032 0.4032 0.4032 0.4032 0.4032 0.4032 0.0000
190 0.0629 0.0726 0.0904 0.1038 0.1092 0.0882 0.0202 0.1290 0.1290 0.1290 0.1290 0.1290 0.1290 0.0000
191 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000
192 0.2718 0.3053 0.3501 0.3612 0.3612 0.3333 0.0423 0.2667 0.3333 0.4000 0.4000 0.4000 0.3667 0.0667
193 0.6100 0.6100 0.6100 0.6188 0.6277 0.6144 0.0089 0.4737 0.4737 0.4737 0.4737 0.4737 0.4737 0.0000
194 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000
195 0.0046 0.0064 0.0082 0.0082 0.0082 0.0073 0.0018 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000
196 0.4241 0.4241 0.4241 0.4256 0.4270 0.4248 0.0015 0.4444 0.4444 0.4444 0.4444 0.4444 0.4444 0.0000
197 0.9118 0.9118 0.9118 0.9131 0.9143 0.9124 0.0012 0.8235 0.8277 0.8319 0.8319 0.8319 0.8298 0.0042
198 0.0067 0.0422 0.2919 0.5062 0.5062 0.2742 0.2695 0.0000 0.0000 0.2500 0.5000 0.5000 0.2500 0.2887
199 0.8099 0.8164 0.8229 0.8229 0.8229 0.8197 0.0065 0.7500 0.7656 0.7812 0.7812 0.7812 0.7734 0.0156
200 0.0130 0.0147 0.0164 0.0164 0.0164 0.0155 0.0017 0.0357 0.0357 0.0357 0.0357 0.0357 0.0357 0.0000
ALL 0.3537 0.3615 0.3696 0.3710 0.3720 0.3662 0.0084 0.3418 0.3490 0.3588 0.3613 0.3613 0.3552 0.0093
101
102
Individual Experiment Results and Graphs
103
104
colesir CoLesIRdeTst AH-ROBUST-MONO-DE-CLEF2006
Overall statistics for 100 queries : Priority 1
Total number of documents over all queries Query Construction AUTOMATIC
Retrieved 100,000 Source Language German
Relevant 3,641 Topic Fields title, description
Relevant retrieved 3,227 Pooled false
Geometric Mean Average Precision 0.1480 robust monolingual DE - TEST; automatic; TD
Binary Preference (BPREF) 0.4078
Interploated Recall (%) Precision Averages (%)
Ad−Hoc Robust Monolingual German track − Interpolated Recall vs Average Precision
100%
0 65.97
CoLesIRdeTst
10 57.08 90%
20 51.59
80%
30 45.07
40 41.80 70%
50 38.62
Average Precision
60%
60 34.91
70 31.28 50%
80 26.85 40%
90 19.40
30%
100 13.14
Average precision (non-interpolated) for all 20%
relevant documents (averaged over queries)
37.21 10%
0%
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
Interpolated Recall
Mean Average Precision Ad−Hoc Robust Monolingual German track − Box plot of the Topics of the Experiment
Maximum 1.0000
Minimum 0.0000
First Quartile 0.1013
Second Quartile 0.3131
Third Quartile 0.5797
Interquartile range 0.4784
Mean 0.3721
Standard Deviation 0.2999
Lower Outlier Threshold 0.0000
Upper Outlier Threshold 1.0000 0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Mean Average Precision
Mean With No Outliers 0.3721
Std With No Outliers 0.2999
Ad−Hoc Robust Monolingual German track − Distribution of the Topics of the Experiment
15
Number of Topics of the Experiment
10
5
0
0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Mean Average Precision
Precision averages (%) for individual Ad−Hoc Robust Monolingual German track − Comparison to Median Mean Average Precision by Topic (Topics 041 to 085)
queries 1
CoLesIRdeTst
0.5
041 74.66 086 44.19 131 0.47 166 1.30
Difference
0
042 73.81 087 22.92 132 79.07 167 61.04
−0.5
043 2.39 088 53.02 133 62.02 168 5.03
044 0.00 089 53.08 134 95.71 169 0.38 −1
041 042 043 044 045 046 047 048 049 060 061 062 063 064 065 066 067 068 069 080 081 082 083 084 085
Topic Identifier
045 53.99 090 22.16 135 34.75 170 0.00 Ad−Hoc Robust Monolingual German track − Comparison to Median Mean Average Precision by Topic (Topics 086 to 130)
1
046 4.92 091 13.31 136 58.17 171 32.47 CoLesIRdeTst
047 6.74 092 16.54 137 7.69 172 33.33 0.5
Difference
048 12.95 093 72.19 138 40.94 173 91.84 0
049 29.12 094 79.85 139 8.30 174 87.53 −0.5
060 30.17 095 12.94 140 26.04 175 60.36 −1
086 087 088 089 090 091 092 093 094 095 096 097 098 099 110 111 112 113 114 115 116 117 118 119 130
061 21.78 096 32.45 141 85.53 176 97.63 Topic Identifier
Ad−Hoc Robust Monolingual German track − Comparison to Median Mean Average Precision by Topic (Topics 131 to 165)
062 52.26 097 50.65 142 89.22 177 10.26 1
CoLesIRdeTst
063 44.99 098 43.68 143 62.30 178 35.77 0.5
Difference
064 100.00 099 27.83 144 0.00 179 53.68
0
065 6.38 110 50.11 145 14.60 190 7.08
−0.5
066 81.95 111 1.78 146 0.00 191 0.00
−1
067 22.06 112 87.29 147 3.19 192 57.77 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 160 161 162 163 164 165
Topic Identifier
068 15.86 113 8.81 148 9.41 193 54.60 Ad−Hoc Robust Monolingual German track − Comparison to Median Mean Average Precision by Topic (Topics 166 to 200)
1
069 8.28 114 25.52 149 93.38 194 83.33 CoLesIRdeTst
0.5
080 7.07 115 13.66 160 14.29 195 40.80
Difference
081 52.21 116 57.45 161 1.59 196 97.62 0
082 33.65 117 23.44 162 51.99 197 57.72 −0.5
083 16.20 118 18.41 163 12.84 198 10.28 −1
166 167 168 169 170 171 172 173 174 175 176 177 178 179 190 191 192 193 194 195 196 197 198 199 200
084 10.00 119 72.40 164 13.70 199 72.81 Topic Identifier
085 7.25 130 61.35 165 44.83 200 24.74
105
colesir CoLesIRdeTst AH-ROBUST-MONO-DE-CLEF2006
Docs Cutoff Levels Precision at DCL (%)
Ad−Hoc Robust Monolingual German track − Retrieved documents vs Precision
100%
5 docs 44.20
CoLesIRdeTst
10 docs 41.20 90%
15 docs 38.53
80%
20 docs 36.30
30 docs 31.87 70%
100 docs 19.22
60%
200 docs 12.27
R−Precision
500 docs 5.92 50%
1000 docs 3.23 40%
R-Precision (precision after R document retrieved,
where R = Relevant retrieved) 30%
36.49
20%
10%
0%
5 10 15 20 30 100 200 500 1000
Retrieved Documents (logarithmic scale)
Exact R-Precision Ad−Hoc Robust Monolingual German track − Box plot of the Topics of the Experiment
Maximum 1.0000
Minimum 0.0000
First Quartile 0.1314
Second Quartile 0.3190
Third Quartile 0.5813
Interquartile range 0.4499
Mean 0.3649
Standard Deviation 0.2762
Lower Outlier Threshold 0.0000
Upper Outlier Threshold 1.0000 0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Exact R−Precision
Mean With No Outliers 0.3649
Std With No Outliers 0.2762
Ad−Hoc Robust Monolingual German track − Distribution of the Topics of the Experiment
20
Number of Topics of the Experiment
15
10
5
0
0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Exact R−Precision
Precision averages (%) for individual Ad−Hoc Robust Monolingual German track − Comparison to Median Mean Exact R−Precision by Topic (Topics 041 to 085)
queries 1
CoLesIRdeTst
0.5
041 67.86 086 48.65 131 1.52 166 0.00
Difference
0
042 68.40 087 23.53 132 71.43 167 50.00
−0.5
043 0.00 088 55.17 133 59.09 168 5.88
044 0.00 089 58.62 134 85.71 169 0.00 −1
041 042 043 044 045 046 047 048 049 060 061 062 063 064 065 066 067 068 069 080 081 082 083 084 085
Topic Identifier
045 56.60 090 31.25 135 43.68 170 0.00 Ad−Hoc Robust Monolingual German track − Comparison to Median Mean Exact R−Precision by Topic (Topics 086 to 130)
1
046 0.00 091 15.62 136 42.86 171 25.00 CoLesIRdeTst
047 4.17 092 17.65 137 0.00 172 0.00 0.5
Difference
048 21.82 093 72.22 138 35.48 173 80.00 0
049 22.22 094 71.88 139 10.71 174 77.78 −0.5
060 38.60 095 24.55 140 38.27 175 54.55 −1
086 087 088 089 090 091 092 093 094 095 096 097 098 099 110 111 112 113 114 115 116 117 118 119 130
061 32.56 096 44.44 141 75.00 176 91.89 Topic Identifier
Ad−Hoc Robust Monolingual German track − Comparison to Median Mean Exact R−Precision by Topic (Topics 131 to 165)
062 44.44 097 53.33 142 80.00 177 14.29 1
CoLesIRdeTst
063 47.37 098 48.00 143 60.32 178 42.86 0.5
Difference
064 100.00 099 34.83 144 0.00 179 53.06
0
065 16.92 110 58.33 145 10.00 190 17.54
−0.5
066 70.59 111 0.00 146 0.00 191 0.00
−1
067 28.00 112 85.71 147 6.90 192 61.54 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 160 161 162 163 164 165
Topic Identifier
068 20.45 113 12.00 148 8.33 193 52.94 Ad−Hoc Robust Monolingual German track − Comparison to Median Mean Exact R−Precision by Topic (Topics 166 to 200)
1
069 14.29 114 28.57 149 91.67 194 50.00 CoLesIRdeTst
0.5
080 11.67 115 25.00 160 0.00 195 37.93
Difference
081 48.00 116 58.06 161 0.00 196 83.33 0
082 29.63 117 26.67 162 54.17 197 58.20 −0.5
083 22.22 118 22.22 163 17.02 198 8.33 −1
166 167 168 169 170 171 172 173 174 175 176 177 178 179 190 191 192 193 194 195 196 197 198 199 200
084 20.00 119 73.96 164 16.67 199 79.17 Topic Identifier
085 4.05 130 60.00 165 25.00 200 30.30
106
daedalus deFSdeR2S AH-ROBUST-MONO-DE-CLEF2006
Overall statistics for 100 queries : Priority 4
Total number of documents over all queries Query Construction AUTOMATIC
Retrieved 100,000 Source Language German
Relevant 3,641 Topic Fields title, description
Relevant retrieved 2,956 Pooled false
Geometric Mean Average Precision 0.1061 Mandatory TD run
Binary Preference (BPREF) 0.3808
Interploated Recall (%) Precision Averages (%)
Ad−Hoc Robust Monolingual German track − Interpolated Recall vs Average Precision
100%
0 66.74
deFSdeR2S
10 56.04 90%
20 48.83
80%
30 43.01
40 38.78 70%
50 35.11
Average Precision
60%
60 30.60
70 27.17 50%
80 21.64 40%
90 16.82
30%
100 9.24
Average precision (non-interpolated) for all 20%
relevant documents (averaged over queries)
34.06 10%
0%
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
Interpolated Recall
Mean Average Precision Ad−Hoc Robust Monolingual German track − Box plot of the Topics of the Experiment
Maximum 1.0000
Minimum 0.0000
First Quartile 0.0849
Second Quartile 0.2410
Third Quartile 0.5680
Interquartile range 0.4830
Mean 0.3406
Standard Deviation 0.2947
Lower Outlier Threshold 0.0000
Upper Outlier Threshold 1.0000 0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Mean Average Precision
Mean With No Outliers 0.3406
Std With No Outliers 0.2947
Ad−Hoc Robust Monolingual German track − Distribution of the Topics of the Experiment
20
Number of Topics of the Experiment
15
10
5
0
0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Mean Average Precision
Precision averages (%) for individual Ad−Hoc Robust Monolingual German track − Comparison to Median Mean Average Precision by Topic (Topics 041 to 085)
queries 1
deFSdeR2S
0.5
041 63.92 086 44.57 131 0.52 166 4.76
Difference
0
042 72.48 087 37.08 132 79.17 167 57.38
−0.5
043 7.41 088 58.00 133 57.51 168 1.90
044 0.00 089 52.09 134 93.67 169 13.89 −1
041 042 043 044 045 046 047 048 049 060 061 062 063 064 065 066 067 068 069 080 081 082 083 084 085
Topic Identifier
045 66.60 090 14.03 135 6.30 170 0.00 Ad−Hoc Robust Monolingual German track − Comparison to Median Mean Average Precision by Topic (Topics 086 to 130)
1
046 8.66 091 10.20 136 45.81 171 2.71 deFSdeR2S
047 13.62 092 18.65 137 0.00 172 0.00 0.5
Difference
048 8.05 093 83.32 138 44.91 173 98.09 0
049 21.15 094 78.07 139 1.53 174 27.60 −0.5
060 22.04 095 3.32 140 13.69 175 74.66 −1
086 087 088 089 090 091 092 093 094 095 096 097 098 099 110 111 112 113 114 115 116 117 118 119 130
061 4.55 096 21.91 141 75.00 176 97.03 Topic Identifier
Ad−Hoc Robust Monolingual German track − Comparison to Median Mean Average Precision by Topic (Topics 131 to 165)
062 42.78 097 31.52 142 32.51 177 0.07 1
deFSdeR2S
063 41.54 098 36.23 143 50.72 178 24.04 0.5
Difference
064 100.00 099 13.50 144 0.00 179 63.90
0
065 5.53 110 42.42 145 8.33 190 54.93
−0.5
066 76.84 111 0.42 146 0.00 191 0.00
−1
067 16.00 112 71.02 147 5.08 192 34.87 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 160 161 162 163 164 165
Topic Identifier
068 28.72 113 15.38 148 13.45 193 64.58 Ad−Hoc Robust Monolingual German track − Comparison to Median Mean Average Precision by Topic (Topics 166 to 200)
1
069 12.93 114 9.46 149 53.87 194 100.00 deFSdeR2S
0.5
080 48.34 115 13.32 160 10.00 195 21.74
Difference
081 51.71 116 49.04 161 0.68 196 95.83 0
082 62.85 117 17.85 162 32.20 197 64.74 −0.5
083 17.71 118 6.32 163 55.48 198 12.81 −1
166 167 168 169 170 171 172 173 174 175 176 177 178 179 190 191 192 193 194 195 196 197 198 199 200
084 10.00 119 62.97 164 7.06 199 72.29 Topic Identifier
085 24.17 130 56.21 165 46.25 200 10.02
107
daedalus deFSdeR2S AH-ROBUST-MONO-DE-CLEF2006
Docs Cutoff Levels Precision at DCL (%)
Ad−Hoc Robust Monolingual German track − Retrieved documents vs Precision
100%
5 docs 46.40
deFSdeR2S
10 docs 40.90 90%
15 docs 37.73
80%
20 docs 34.90
30 docs 31.10 70%
100 docs 18.31
60%
200 docs 11.57
R−Precision
500 docs 5.52 50%
1000 docs 2.96 40%
R-Precision (precision after R document retrieved,
where R = Relevant retrieved) 30%
34.80
20%
10%
0%
5 10 15 20 30 100 200 500 1000
Retrieved Documents (logarithmic scale)
Exact R-Precision Ad−Hoc Robust Monolingual German track − Box plot of the Topics of the Experiment
Maximum 1.0000
Minimum 0.0000
First Quartile 0.1091
Second Quartile 0.2920
Third Quartile 0.5757
Interquartile range 0.4666
Mean 0.3480
Standard Deviation 0.2760
Lower Outlier Threshold 0.0000
Upper Outlier Threshold 1.0000 0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Exact R−Precision
Mean With No Outliers 0.3480
Std With No Outliers 0.2760
Ad−Hoc Robust Monolingual German track − Distribution of the Topics of the Experiment
20
Number of Topics of the Experiment
15
10
5
0
0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Exact R−Precision
Precision averages (%) for individual Ad−Hoc Robust Monolingual German track − Comparison to Median Mean Exact R−Precision by Topic (Topics 041 to 085)
queries 1
deFSdeR2S
0.5
041 57.14 086 44.14 131 4.55 166 0.00
Difference
0
042 67.92 087 35.29 132 71.43 167 50.00
−0.5
043 0.00 088 55.17 133 63.64 168 0.00
044 0.00 089 49.66 134 85.71 169 20.00 −1
041 042 043 044 045 046 047 048 049 060 061 062 063 064 065 066 067 068 069 080 081 082 083 084 085
Topic Identifier
045 62.26 090 18.75 135 16.09 170 0.00 Ad−Hoc Robust Monolingual German track − Comparison to Median Mean Exact R−Precision by Topic (Topics 086 to 130)
1
046 0.00 091 18.75 136 42.86 171 0.00 deFSdeR2S
047 10.42 092 17.65 137 0.00 172 0.00 0.5
Difference
048 14.55 093 80.56 138 38.71 173 90.00 0
049 22.22 094 75.00 139 3.57 174 30.56 −0.5
060 29.82 095 5.45 140 27.16 175 63.64 −1
086 087 088 089 090 091 092 093 094 095 096 097 098 099 110 111 112 113 114 115 116 117 118 119 130
061 2.33 096 25.93 141 75.00 176 89.19 Topic Identifier
Ad−Hoc Robust Monolingual German track − Comparison to Median Mean Exact R−Precision by Topic (Topics 131 to 165)
062 33.33 097 33.33 142 40.00 177 0.00 1
deFSdeR2S
063 34.21 098 48.00 143 58.73 178 38.10 0.5
Difference
064 100.00 099 25.84 144 0.00 179 59.18
0
065 16.92 110 58.33 145 20.00 190 56.14
−0.5
066 68.63 111 0.00 146 0.00 191 0.00
−1
067 18.00 112 64.29 147 6.90 192 46.15 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 160 161 162 163 164 165
Topic Identifier
068 40.91 113 16.00 148 25.00 193 52.94 Ad−Hoc Robust Monolingual German track − Comparison to Median Mean Exact R−Precision by Topic (Topics 166 to 200)
1
069 28.57 114 10.71 149 50.00 194 100.00 deFSdeR2S
0.5
080 55.00 115 16.67 160 0.00 195 27.59
Difference
081 58.00 116 54.84 161 0.00 196 83.33 0
082 70.37 117 22.22 162 41.67 197 60.66 −0.5
083 11.11 118 0.00 163 48.94 198 16.67 −1
166 167 168 169 170 171 172 173 174 175 176 177 178 179 190 191 192 193 194 195 196 197 198 199 200
084 20.00 119 67.71 164 16.67 199 70.83 Topic Identifier
085 27.03 130 60.00 165 25.00 200 10.61
108
daedalus deFSdeR3S AH-ROBUST-MONO-DE-CLEF2006
Overall statistics for 100 queries : Priority 3
Total number of documents over all queries Query Construction AUTOMATIC
Retrieved 100,000 Source Language German
Relevant 3,641 Topic Fields title, description, narrative
Relevant retrieved 3,094 Pooled false
Geometric Mean Average Precision 0.1155 TDN run
Binary Preference (BPREF) 0.4186
Interploated Recall (%) Precision Averages (%)
Ad−Hoc Robust Monolingual German track − Interpolated Recall vs Average Precision
100%
0 68.65
deFSdeR3S
10 58.26 90%
20 52.51
80%
30 48.00
40 44.83 70%
50 40.74
Average Precision
60%
60 35.32
70 30.88 50%
80 25.23 40%
90 18.33
30%
100 10.34
Average precision (non-interpolated) for all 20%
relevant documents (averaged over queries)
38.03 10%
0%
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
Interpolated Recall
Mean Average Precision Ad−Hoc Robust Monolingual German track − Box plot of the Topics of the Experiment
Maximum 1.0000
Minimum 0.0000
First Quartile 0.1147
Second Quartile 0.3315
Third Quartile 0.6281
Interquartile range 0.5134
Mean 0.3803
Standard Deviation 0.3013
Lower Outlier Threshold 0.0000
Upper Outlier Threshold 1.0000 0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Mean Average Precision
Mean With No Outliers 0.3803
Std With No Outliers 0.3013
Ad−Hoc Robust Monolingual German track − Distribution of the Topics of the Experiment
20
Number of Topics of the Experiment
15
10
5
0
0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Mean Average Precision
Precision averages (%) for individual Ad−Hoc Robust Monolingual German track − Comparison to Median Mean Average Precision by Topic (Topics 041 to 085)
queries 1
deFSdeR3S
0.5
041 61.19 086 48.24 131 0.95 166 1.70
Difference
0
042 71.70 087 33.47 132 73.59 167 67.50
−0.5
043 3.68 088 60.77 133 63.38 168 2.38
044 0.00 089 50.92 134 88.39 169 16.32 −1
041 042 043 044 045 046 047 048 049 060 061 062 063 064 065 066 067 068 069 080 081 082 083 084 085
Topic Identifier
045 67.56 090 18.12 135 11.61 170 0.00 Ad−Hoc Robust Monolingual German track − Comparison to Median Mean Average Precision by Topic (Topics 086 to 130)
1
046 13.83 091 11.33 136 48.80 171 5.96 deFSdeR3S
047 12.81 092 19.99 137 0.00 172 0.00 0.5
Difference
048 10.93 093 89.49 138 71.97 173 97.33 0
049 17.75 094 76.09 139 1.06 174 63.22 −0.5
060 21.64 095 12.14 140 8.63 175 52.82 −1
086 087 088 089 090 091 092 093 094 095 096 097 098 099 110 111 112 113 114 115 116 117 118 119 130
061 3.85 096 20.67 141 75.00 176 97.53 Topic Identifier
Ad−Hoc Robust Monolingual German track − Comparison to Median Mean Average Precision by Topic (Topics 131 to 165)
062 48.80 097 65.79 142 33.98 177 0.00 1
deFSdeR3S
063 54.22 098 39.48 143 57.68 178 10.27 0.5
Difference
064 100.00 099 26.18 144 0.00 179 62.65
0
065 13.27 110 45.64 145 32.81 190 57.26
−0.5
066 77.36 111 0.77 146 0.00 191 0.00
−1
067 20.06 112 76.02 147 3.17 192 48.51 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 160 161 162 163 164 165
Topic Identifier
068 32.97 113 3.39 148 14.16 193 62.96 Ad−Hoc Robust Monolingual German track − Comparison to Median Mean Average Precision by Topic (Topics 166 to 200)
1
069 7.54 114 26.07 149 84.68 194 100.00 deFSdeR3S
0.5
080 50.22 115 17.09 160 25.00 195 23.51
Difference
081 64.50 116 56.88 161 33.33 196 95.83 0
082 55.21 117 19.18 162 74.97 197 58.56 −0.5
083 18.25 118 26.83 163 58.64 198 11.95 −1
166 167 168 169 170 171 172 173 174 175 176 177 178 179 190 191 192 193 194 195 196 197 198 199 200
084 0.22 119 62.02 164 9.73 199 78.31 Topic Identifier
085 39.69 130 78.43 165 40.91 200 23.54
109
daedalus deFSdeR3S AH-ROBUST-MONO-DE-CLEF2006
Docs Cutoff Levels Precision at DCL (%)
Ad−Hoc Robust Monolingual German track − Retrieved documents vs Precision
100%
5 docs 50.40
deFSdeR3S
10 docs 45.70 90%
15 docs 41.07
80%
20 docs 38.60
30 docs 33.90 70%
100 docs 19.34
60%
200 docs 12.21
R−Precision
500 docs 5.76 50%
1000 docs 3.09 40%
R-Precision (precision after R document retrieved,
where R = Relevant retrieved) 30%
37.76
20%
10%
0%
5 10 15 20 30 100 200 500 1000
Retrieved Documents (logarithmic scale)
Exact R-Precision Ad−Hoc Robust Monolingual German track − Box plot of the Topics of the Experiment
Maximum 1.0000
Minimum 0.0000
First Quartile 0.1422
Second Quartile 0.3842
Third Quartile 0.5846
Interquartile range 0.4425
Mean 0.3776
Standard Deviation 0.2817
Lower Outlier Threshold 0.0000
Upper Outlier Threshold 1.0000 0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Exact R−Precision
Mean With No Outliers 0.3776
Std With No Outliers 0.2817
Ad−Hoc Robust Monolingual German track − Distribution of the Topics of the Experiment
20
Number of Topics of the Experiment
15
10
5
0
0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Exact R−Precision
Precision averages (%) for individual Ad−Hoc Robust Monolingual German track − Comparison to Median Mean Exact R−Precision by Topic (Topics 041 to 085)
queries 1
deFSdeR3S
0.5
041 50.00 086 46.85 131 6.06 166 0.00
Difference
0
042 69.34 087 35.29 132 60.71 167 50.00
−0.5
043 0.00 088 55.17 133 64.55 168 5.88
044 0.00 089 53.10 134 71.43 169 20.00 −1
041 042 043 044 045 046 047 048 049 060 061 062 063 064 065 066 067 068 069 080 081 082 083 084 085
Topic Identifier
045 64.15 090 18.75 135 21.84 170 0.00 Ad−Hoc Robust Monolingual German track − Comparison to Median Mean Exact R−Precision by Topic (Topics 086 to 130)
1
046 20.00 091 15.62 136 42.86 171 0.00 deFSdeR3S
047 18.75 092 23.53 137 0.00 172 0.00 0.5
Difference
048 18.18 093 75.00 138 74.19 173 90.00 0
049 11.11 094 81.25 139 3.57 174 63.89 −0.5
060 28.07 095 14.55 140 18.52 175 54.55 −1
086 087 088 089 090 091 092 093 094 095 096 097 098 099 110 111 112 113 114 115 116 117 118 119 130
061 2.33 096 22.22 141 75.00 176 90.54 Topic Identifier
Ad−Hoc Robust Monolingual German track − Comparison to Median Mean Exact R−Precision by Topic (Topics 131 to 165)
062 66.67 097 53.33 142 40.00 177 0.00 1
deFSdeR3S
063 55.26 098 52.00 143 58.73 178 23.81 0.5
Difference
064 100.00 099 38.20 144 0.00 179 55.10
0
065 21.54 110 54.17 145 40.00 190 57.89
−0.5
066 66.67 111 0.00 146 0.00 191 0.00
−1
067 26.00 112 71.43 147 3.45 192 50.00 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 160 161 162 163 164 165
Topic Identifier
068 38.64 113 8.00 148 25.00 193 55.88 Ad−Hoc Robust Monolingual German track − Comparison to Median Mean Exact R−Precision by Topic (Topics 166 to 200)
1
069 0.00 114 25.00 149 83.33 194 100.00 deFSdeR3S
0.5
080 56.67 115 25.00 160 0.00 195 24.14
Difference
081 58.00 116 54.84 161 0.00 196 83.33 0
082 59.26 117 31.11 162 70.83 197 58.20 −0.5
083 11.11 118 44.44 163 57.45 198 16.67 −1
166 167 168 169 170 171 172 173 174 175 176 177 178 179 190 191 192 193 194 195 196 197 198 199 200
084 0.00 119 66.67 164 13.89 199 75.00 Topic Identifier
085 39.19 130 66.67 165 25.00 200 31.82
110
daedalus dex021deRFW3FS3 AH-ROBUST-MONO-DE-CLEF2006
Overall statistics for 100 queries : Priority 2
Total number of documents over all queries Query Construction AUTOMATIC
Retrieved 100,000 Source Language German
Relevant 3,641 Topic Fields title, description, narrative
Relevant retrieved 3,094 Pooled false
Geometric Mean Average Precision 0.1149 Combined run
Binary Preference (BPREF) 0.4258
Interploated Recall (%) Precision Averages (%)
Ad−Hoc Robust Monolingual German track − Interpolated Recall vs Average Precision
100%
0 66.76
dex021deRFW3FS3
10 59.02 90%
20 53.10
80%
30 48.46
40 45.35 70%
50 41.40
Average Precision
60%
60 35.66
70 31.21 50%
80 25.56 40%
90 18.66
30%
100 10.67
Average precision (non-interpolated) for all 20%
relevant documents (averaged over queries)
38.31 10%
0%
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
Interpolated Recall
Mean Average Precision Ad−Hoc Robust Monolingual German track − Box plot of the Topics of the Experiment
Maximum 1.0000
Minimum 0.0000
First Quartile 0.1089
Second Quartile 0.3208
Third Quartile 0.6281
Interquartile range 0.5191
Mean 0.3831
Standard Deviation 0.3063
Lower Outlier Threshold 0.0000
Upper Outlier Threshold 1.0000 0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Mean Average Precision
Mean With No Outliers 0.3831
Std With No Outliers 0.3063
Ad−Hoc Robust Monolingual German track − Distribution of the Topics of the Experiment
20
Number of Topics of the Experiment
15
10
5
0
0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Mean Average Precision
Precision averages (%) for individual Ad−Hoc Robust Monolingual German track − Comparison to Median Mean Average Precision by Topic (Topics 041 to 085)
queries 1
dex021deRFW3FS3
0.5
041 61.19 086 47.72 131 0.93 166 1.70
Difference
0
042 71.70 087 32.49 132 73.59 167 67.50
−0.5
043 3.68 088 60.77 133 63.38 168 2.38
044 0.00 089 51.56 134 88.39 169 16.32 −1
041 042 043 044 045 046 047 048 049 060 061 062 063 064 065 066 067 068 069 080 081 082 083 084 085
Topic Identifier
045 67.56 090 15.69 135 11.15 170 0.00 Ad−Hoc Robust Monolingual German track − Comparison to Median Mean Average Precision by Topic (Topics 086 to 130)
1
046 13.83 091 11.33 136 51.18 171 5.96 dex021deRFW3FS3
047 12.74 092 19.99 137 0.00 172 0.00 0.5
Difference
048 10.93 093 89.49 138 68.44 173 97.33 0
049 14.05 094 78.69 139 1.06 174 63.22 −0.5
060 19.31 095 12.14 140 7.99 175 52.82 −1
086 087 088 089 090 091 092 093 094 095 096 097 098 099 110 111 112 113 114 115 116 117 118 119 130
061 3.85 096 20.67 141 75.00 176 97.53 Topic Identifier
Ad−Hoc Robust Monolingual German track − Comparison to Median Mean Average Precision by Topic (Topics 131 to 165)
062 56.21 097 65.79 142 33.98 177 0.00 1
dex021deRFW3FS3
063 54.22 098 38.81 143 55.55 178 10.27 0.5
Difference
064 100.00 099 26.55 144 0.00 179 62.65
0
065 10.86 110 45.64 145 28.69 190 57.26
−0.5
066 77.36 111 0.77 146 0.00 191 0.00
−1
067 15.78 112 76.02 147 2.97 192 48.51 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 160 161 162 163 164 165
Topic Identifier
068 31.67 113 3.36 148 16.66 193 62.96 Ad−Hoc Robust Monolingual German track − Comparison to Median Mean Average Precision by Topic (Topics 166 to 200)
1
069 7.54 114 23.68 149 84.68 194 83.33 dex021deRFW3FS3
0.5
080 50.16 115 16.39 160 25.00 195 24.67
Difference
081 61.94 116 57.96 161 100.00 196 95.83 0
082 55.21 117 18.44 162 74.97 197 57.66 −0.5
083 18.25 118 25.34 163 58.64 198 10.56 −1
166 167 168 169 170 171 172 173 174 175 176 177 178 179 190 191 192 193 194 195 196 197 198 199 200
084 0.22 119 62.19 164 9.37 199 78.31 Topic Identifier
085 39.69 130 78.43 165 40.91 200 24.05
111
daedalus dex021deRFW3FS3 AH-ROBUST-MONO-DE-CLEF2006
Docs Cutoff Levels Precision at DCL (%)
Ad−Hoc Robust Monolingual German track − Retrieved documents vs Precision
100%
5 docs 49.60
dex021deRFW3FS3
10 docs 45.50 90%
15 docs 40.93
80%
20 docs 38.50
30 docs 33.87 70%
100 docs 19.34
60%
200 docs 12.21
R−Precision
500 docs 5.76 50%
1000 docs 3.09 40%
R-Precision (precision after R document retrieved,
where R = Relevant retrieved) 30%
38.15
20%
10%
0%
5 10 15 20 30 100 200 500 1000
Retrieved Documents (logarithmic scale)
Exact R-Precision Ad−Hoc Robust Monolingual German track − Box plot of the Topics of the Experiment
Maximum 1.0000
Minimum 0.0000
First Quartile 0.1509
Second Quartile 0.3842
Third Quartile 0.5846
Interquartile range 0.4338
Mean 0.3815
Standard Deviation 0.2794
Lower Outlier Threshold 0.0000
Upper Outlier Threshold 1.0000 0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Exact R−Precision
Mean With No Outliers 0.3815
Std With No Outliers 0.2794
Ad−Hoc Robust Monolingual German track − Distribution of the Topics of the Experiment
20
Number of Topics of the Experiment
15
10
5
0
0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Exact R−Precision
Precision averages (%) for individual Ad−Hoc Robust Monolingual German track − Comparison to Median Mean Exact R−Precision by Topic (Topics 041 to 085)
queries 1
dex021deRFW3FS3
0.5
041 50.00 086 46.85 131 6.06 166 0.00
Difference
0
042 69.34 087 35.29 132 60.71 167 50.00
−0.5
043 0.00 088 55.17 133 64.55 168 5.88
044 0.00 089 53.10 134 71.43 169 20.00 −1
041 042 043 044 045 046 047 048 049 060 061 062 063 064 065 066 067 068 069 080 081 082 083 084 085
Topic Identifier
045 64.15 090 18.75 135 21.84 170 0.00 Ad−Hoc Robust Monolingual German track − Comparison to Median Mean Exact R−Precision by Topic (Topics 086 to 130)
1
046 20.00 091 15.62 136 42.86 171 0.00 dex021deRFW3FS3
047 18.75 092 23.53 137 0.00 172 0.00 0.5
Difference
048 18.18 093 75.00 138 74.19 173 90.00 0
049 11.11 094 81.25 139 3.57 174 63.89 −0.5
060 28.07 095 14.55 140 18.52 175 54.55 −1
086 087 088 089 090 091 092 093 094 095 096 097 098 099 110 111 112 113 114 115 116 117 118 119 130
061 2.33 096 22.22 141 75.00 176 90.54 Topic Identifier
Ad−Hoc Robust Monolingual German track − Comparison to Median Mean Exact R−Precision by Topic (Topics 131 to 165)
062 66.67 097 53.33 142 40.00 177 0.00 1
dex021deRFW3FS3
063 55.26 098 52.00 143 58.73 178 23.81 0.5
Difference
064 100.00 099 38.20 144 0.00 179 55.10
0
065 21.54 110 54.17 145 40.00 190 57.89
−0.5
066 66.67 111 0.00 146 0.00 191 0.00
−1
067 26.00 112 71.43 147 3.45 192 50.00 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 160 161 162 163 164 165
Topic Identifier
068 38.64 113 8.00 148 25.00 193 55.88 Ad−Hoc Robust Monolingual German track − Comparison to Median Mean Exact R−Precision by Topic (Topics 166 to 200)
1
069 0.00 114 25.00 149 83.33 194 50.00 dex021deRFW3FS3
0.5
080 56.67 115 25.00 160 0.00 195 24.14
Difference
081 58.00 116 54.84 161 100.00 196 83.33 0
082 59.26 117 31.11 162 70.83 197 58.20 −0.5
083 11.11 118 33.33 163 57.45 198 16.67 −1
166 167 168 169 170 171 172 173 174 175 176 177 178 179 190 191 192 193 194 195 196 197 198 199 200
084 0.00 119 66.67 164 13.89 199 75.00 Topic Identifier
085 39.19 130 66.67 165 25.00 200 31.82
112
daedalus dex011deRFW3FS3 AH-ROBUST-MONO-DE-CLEF2006
Overall statistics for 100 queries : Priority 1
Total number of documents over all queries Query Construction AUTOMATIC
Retrieved 100,000 Source Language German
Relevant 3,641 Topic Fields title, description, narrative
Relevant retrieved 3,094 Pooled false
Geometric Mean Average Precision 0.1159 Combined run
Binary Preference (BPREF) 0.4288
Interploated Recall (%) Precision Averages (%)
Ad−Hoc Robust Monolingual German track − Interpolated Recall vs Average Precision
100%
0 67.09
dex011deRFW3FS3
10 59.06 90%
20 53.15
80%
30 48.65
40 45.49 70%
50 41.40
Average Precision
60%
60 35.99
70 31.54 50%
80 25.89 40%
90 18.99
30%
100 11.01
Average precision (non-interpolated) for all 20%
relevant documents (averaged over queries)
38.63 10%
0%
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
Interpolated Recall
Mean Average Precision Ad−Hoc Robust Monolingual German track − Box plot of the Topics of the Experiment
Maximum 1.0000
Minimum 0.0000
First Quartile 0.1161
Second Quartile 0.3265
Third Quartile 0.6281
Interquartile range 0.5120
Mean 0.3863
Standard Deviation 0.3090
Lower Outlier Threshold 0.0000
Upper Outlier Threshold 1.0000 0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Mean Average Precision
Mean With No Outliers 0.3863
Std With No Outliers 0.3090
Ad−Hoc Robust Monolingual German track − Distribution of the Topics of the Experiment
20
Number of Topics of the Experiment
15
10
5
0
0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Mean Average Precision
Precision averages (%) for individual Ad−Hoc Robust Monolingual German track − Comparison to Median Mean Average Precision by Topic (Topics 041 to 085)
queries 1
dex011deRFW3FS3
0.5
041 61.19 086 48.24 131 0.93 166 1.70
Difference
0
042 71.70 087 32.49 132 73.59 167 67.50
−0.5
043 3.68 088 60.77 133 63.38 168 2.38
044 0.00 089 51.27 134 88.39 169 16.32 −1
041 042 043 044 045 046 047 048 049 060 061 062 063 064 065 066 067 068 069 080 081 082 083 084 085
Topic Identifier
045 67.56 090 17.08 135 11.61 170 0.00 Ad−Hoc Robust Monolingual German track − Comparison to Median Mean Average Precision by Topic (Topics 086 to 130)
1
046 13.83 091 12.37 136 48.80 171 5.96 dex011deRFW3FS3
047 12.77 092 19.99 137 0.00 172 0.00 0.5
Difference
048 10.93 093 89.49 138 71.97 173 97.33 0
049 14.97 094 77.65 139 1.06 174 63.22 −0.5
060 19.31 095 12.14 140 8.22 175 52.82 −1
086 087 088 089 090 091 092 093 094 095 096 097 098 099 110 111 112 113 114 115 116 117 118 119 130
061 3.85 096 20.67 141 75.00 176 97.53 Topic Identifier
Ad−Hoc Robust Monolingual German track − Comparison to Median Mean Average Precision by Topic (Topics 131 to 165)
062 56.21 097 65.79 142 33.98 177 0.00 1
dex011deRFW3FS3
063 54.22 098 39.48 143 57.68 178 10.27 0.5
Difference
064 100.00 099 26.55 144 0.00 179 62.65
0
065 11.60 110 45.64 145 32.81 190 57.26
−0.5
066 77.36 111 0.77 146 0.00 191 0.00
−1
067 17.20 112 76.02 147 3.05 192 48.51 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 160 161 162 163 164 165
Topic Identifier
068 30.77 113 3.36 148 14.16 193 62.96 Ad−Hoc Robust Monolingual German track − Comparison to Median Mean Average Precision by Topic (Topics 166 to 200)
1
069 7.54 114 24.28 149 84.68 194 100.00 dex011deRFW3FS3
0.5
080 50.77 115 16.39 160 25.00 195 23.51
Difference
081 61.94 116 59.03 161 100.00 196 95.83 0
082 55.21 117 19.18 162 74.97 197 58.56 −0.5
083 18.25 118 26.83 163 58.64 198 11.95 −1
166 167 168 169 170 171 172 173 174 175 176 177 178 179 190 191 192 193 194 195 196 197 198 199 200
084 0.22 119 62.02 164 9.37 199 78.31 Topic Identifier
085 39.69 130 78.43 165 40.91 200 23.54
113
daedalus dex011deRFW3FS3 AH-ROBUST-MONO-DE-CLEF2006
Docs Cutoff Levels Precision at DCL (%)
Ad−Hoc Robust Monolingual German track − Retrieved documents vs Precision
100%
5 docs 50.40
dex011deRFW3FS3
10 docs 45.60 90%
15 docs 41.00
80%
20 docs 38.55
30 docs 33.87 70%
100 docs 19.34
60%
200 docs 12.21
R−Precision
500 docs 5.76 50%
1000 docs 3.09 40%
R-Precision (precision after R document retrieved,
where R = Relevant retrieved) 30%
38.76
20%
10%
0%
5 10 15 20 30 100 200 500 1000
Retrieved Documents (logarithmic scale)
Exact R-Precision Ad−Hoc Robust Monolingual German track − Box plot of the Topics of the Experiment
Maximum 1.0000
Minimum 0.0000
First Quartile 0.1509
Second Quartile 0.3891
Third Quartile 0.5899
Interquartile range 0.4391
Mean 0.3876
Standard Deviation 0.2859
Lower Outlier Threshold 0.0000
Upper Outlier Threshold 1.0000 0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Exact R−Precision
Mean With No Outliers 0.3876
Std With No Outliers 0.2859
Ad−Hoc Robust Monolingual German track − Distribution of the Topics of the Experiment
20
Number of Topics of the Experiment
15
10
5
0
0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Exact R−Precision
Precision averages (%) for individual Ad−Hoc Robust Monolingual German track − Comparison to Median Mean Exact R−Precision by Topic (Topics 041 to 085)
queries 1
dex011deRFW3FS3
0.5
041 50.00 086 46.85 131 6.06 166 0.00
Difference
0
042 69.34 087 35.29 132 60.71 167 50.00
−0.5
043 0.00 088 55.17 133 64.55 168 5.88
044 0.00 089 53.10 134 71.43 169 20.00 −1
041 042 043 044 045 046 047 048 049 060 061 062 063 064 065 066 067 068 069 080 081 082 083 084 085
Topic Identifier
045 64.15 090 18.75 135 21.84 170 0.00 Ad−Hoc Robust Monolingual German track − Comparison to Median Mean Exact R−Precision by Topic (Topics 086 to 130)
1
046 20.00 091 15.62 136 42.86 171 0.00 dex011deRFW3FS3
047 18.75 092 23.53 137 0.00 172 0.00 0.5
Difference
048 18.18 093 75.00 138 74.19 173 90.00 0
049 11.11 094 81.25 139 3.57 174 63.89 −0.5
060 28.07 095 14.55 140 18.52 175 54.55 −1
086 087 088 089 090 091 092 093 094 095 096 097 098 099 110 111 112 113 114 115 116 117 118 119 130
061 2.33 096 22.22 141 75.00 176 90.54 Topic Identifier
Ad−Hoc Robust Monolingual German track − Comparison to Median Mean Exact R−Precision by Topic (Topics 131 to 165)
062 66.67 097 53.33 142 40.00 177 0.00 1
dex011deRFW3FS3
063 55.26 098 52.00 143 58.73 178 23.81 0.5
Difference
064 100.00 099 38.20 144 0.00 179 55.10
0
065 21.54 110 54.17 145 40.00 190 57.89
−0.5
066 66.67 111 0.00 146 0.00 191 0.00
−1
067 26.00 112 71.43 147 3.45 192 50.00 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 160 161 162 163 164 165
Topic Identifier
068 38.64 113 8.00 148 25.00 193 55.88 Ad−Hoc Robust Monolingual German track − Comparison to Median Mean Exact R−Precision by Topic (Topics 166 to 200)
1
069 0.00 114 25.00 149 83.33 194 100.00 dex011deRFW3FS3
0.5
080 56.67 115 25.00 160 0.00 195 24.14
Difference
081 58.00 116 54.84 161 100.00 196 83.33 0
082 59.26 117 31.11 162 70.83 197 58.20 −0.5
083 11.11 118 44.44 163 57.45 198 16.67 −1
166 167 168 169 170 171 172 173 174 175 176 177 178 179 190 191 192 193 194 195 196 197 198 199 200
084 0.00 119 66.67 164 13.89 199 75.00 Topic Identifier
085 39.19 130 66.67 165 25.00 200 31.82
114
hummingbird humDE06Rtde AH-ROBUST-MONO-DE-CLEF2006
Overall statistics for 100 queries : Priority 2
Total number of documents over all queries Query Construction AUTOMATIC
Retrieved 100,000 Source Language German
Relevant 3,641 Topic Fields title, description
Relevant retrieved 3,370 Pooled false
Geometric Mean Average Precision 0.2253 same as humDE06Rtde0
Binary Preference (BPREF) 0.5291
Interploated Recall (%) Precision Averages (%)
Ad−Hoc Robust Monolingual German track − Interpolated Recall vs Average Precision
100%
0 74.50
humDE06Rtde
10 67.55 90%
20 61.86
80%
30 57.66
40 55.49 70%
50 52.54
Average Precision
60%
60 48.30
70 44.00 50%
80 38.31 40%
90 29.14
30%
100 17.92
Average precision (non-interpolated) for all 20%
relevant documents (averaged over queries)
48.30 10%
0%
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
Interpolated Recall
Mean Average Precision Ad−Hoc Robust Monolingual German track − Box plot of the Topics of the Experiment
Maximum 1.0000
Minimum 0.0000
First Quartile 0.2009
Second Quartile 0.4896
Third Quartile 0.7837
Interquartile range 0.5828
Mean 0.4830
Standard Deviation 0.3096
Lower Outlier Threshold 0.0000
Upper Outlier Threshold 1.0000 0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Mean Average Precision
Mean With No Outliers 0.4830
Std With No Outliers 0.3096
Ad−Hoc Robust Monolingual German track − Distribution of the Topics of the Experiment
10
Number of Topics of the Experiment
8
6
4
2
0
0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Mean Average Precision
Precision averages (%) for individual Ad−Hoc Robust Monolingual German track − Comparison to Median Mean Average Precision by Topic (Topics 041 to 085)
queries 1
humDE06Rtde
0.5
041 84.97 086 35.30 131 2.83 166 2.40
Difference
0
042 74.23 087 34.85 132 84.65 167 80.42
−0.5
043 41.67 088 64.10 133 65.63 168 15.12
044 0.00 089 65.54 134 85.30 169 6.53 −1
041 042 043 044 045 046 047 048 049 060 061 062 063 064 065 066 067 068 069 080 081 082 083 084 085
Topic Identifier
045 51.34 090 10.61 135 60.49 170 0.00 Ad−Hoc Robust Monolingual German track − Comparison to Median Mean Average Precision by Topic (Topics 086 to 130)
1
046 5.03 091 32.10 136 59.59 171 58.88 humDE06Rtde
047 15.89 092 23.49 137 5.56 172 100.00 0.5
Difference
048 20.95 093 86.72 138 65.90 173 96.52 0
049 24.57 094 77.99 139 16.10 174 89.36 −0.5
060 34.71 095 6.91 140 49.62 175 87.42 −1
086 087 088 089 090 091 092 093 094 095 096 097 098 099 110 111 112 113 114 115 116 117 118 119 130
061 46.12 096 23.76 141 97.05 176 98.77 Topic Identifier
Ad−Hoc Robust Monolingual German track − Comparison to Median Mean Average Precision by Topic (Topics 131 to 165)
062 46.64 097 50.57 142 88.99 177 28.81 1
humDE06Rtde
063 79.15 098 43.67 143 69.41 178 43.77 0.5
Difference
064 100.00 099 54.75 144 0.00 179 80.69
0
065 12.04 110 56.39 145 78.74 190 11.70
−0.5
066 83.83 111 16.71 146 0.00 191 0.00
−1
067 68.23 112 83.19 147 33.84 192 84.97 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 160 161 162 163 164 165
Topic Identifier
068 21.78 113 14.00 148 11.15 193 63.63 Ad−Hoc Robust Monolingual German track − Comparison to Median Mean Average Precision by Topic (Topics 166 to 200)
1
069 11.51 114 19.22 149 87.08 194 83.33 humDE06Rtde
0.5
080 42.66 115 26.82 160 25.00 195 53.45
Difference
081 52.80 116 48.29 161 1.64 196 94.44 0
082 84.60 117 45.53 162 89.21 197 73.21 −0.5
083 26.58 118 24.48 163 69.20 198 14.29 −1
166 167 168 169 170 171 172 173 174 175 176 177 178 179 190 191 192 193 194 195 196 197 198 199 200
084 21.67 119 72.50 164 39.65 199 82.27 Topic Identifier
085 18.06 130 72.76 165 65.00 200 69.16
115
hummingbird humDE06Rtde AH-ROBUST-MONO-DE-CLEF2006
Docs Cutoff Levels Precision at DCL (%)
Ad−Hoc Robust Monolingual German track − Retrieved documents vs Precision
100%
5 docs 55.80
humDE06Rtde
10 docs 51.20 90%
15 docs 48.27
80%
20 docs 44.90
30 docs 39.87 70%
100 docs 23.23
60%
200 docs 14.36
R−Precision
500 docs 6.50 50%
1000 docs 3.37 40%
R-Precision (precision after R document retrieved,
where R = Relevant retrieved) 30%
46.34
20%
10%
0%
5 10 15 20 30 100 200 500 1000
Retrieved Documents (logarithmic scale)
Exact R-Precision Ad−Hoc Robust Monolingual German track − Box plot of the Topics of the Experiment
Maximum 1.0000
Minimum 0.0000
First Quartile 0.2192
Second Quartile 0.5000
Third Quartile 0.6910
Interquartile range 0.4717
Mean 0.4634
Standard Deviation 0.2822
Lower Outlier Threshold 0.0000
Upper Outlier Threshold 1.0000 0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Exact R−Precision
Mean With No Outliers 0.4634
Std With No Outliers 0.2822
Ad−Hoc Robust Monolingual German track − Distribution of the Topics of the Experiment
15
Number of Topics of the Experiment
10
5
0
0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Exact R−Precision
Precision averages (%) for individual Ad−Hoc Robust Monolingual German track − Comparison to Median Mean Exact R−Precision by Topic (Topics 041 to 085)
queries 1
humDE06Rtde
0.5
041 75.00 086 47.75 131 3.03 166 0.00
Difference
0
042 69.34 087 35.29 132 78.57 167 75.00
−0.5
043 0.00 088 56.90 133 66.36 168 23.53
044 0.00 089 67.59 134 71.43 169 10.00 −1
041 042 043 044 045 046 047 048 049 060 061 062 063 064 065 066 067 068 069 080 081 082 083 084 085
Topic Identifier
045 53.77 090 12.50 135 58.62 170 0.00 Ad−Hoc Robust Monolingual German track − Comparison to Median Mean Exact R−Precision by Topic (Topics 086 to 130)
1
046 0.00 091 37.50 136 57.14 171 50.00 humDE06Rtde
047 12.50 092 29.41 137 0.00 172 100.00 0.5
Difference
048 29.09 093 80.56 138 64.52 173 80.00 0
049 16.67 094 75.00 139 10.71 174 83.33 −0.5
060 54.39 095 17.27 140 56.79 175 81.82 −1
086 087 088 089 090 091 092 093 094 095 096 097 098 099 110 111 112 113 114 115 116 117 118 119 130
061 58.14 096 29.63 141 87.50 176 93.24 Topic Identifier
Ad−Hoc Robust Monolingual German track − Comparison to Median Mean Exact R−Precision by Topic (Topics 131 to 165)
062 44.44 097 60.00 142 84.62 177 28.57 1
humDE06Rtde
063 68.42 098 48.00 143 66.67 178 42.86 0.5
Difference
064 100.00 099 53.93 144 0.00 179 75.51
0
065 20.00 110 62.50 145 60.00 190 22.81
−0.5
066 68.63 111 12.50 146 0.00 191 0.00
−1
067 66.00 112 71.43 147 37.93 192 73.08 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 160 161 162 163 164 165
Topic Identifier
068 34.09 113 24.00 148 8.33 193 58.82 Ad−Hoc Robust Monolingual German track − Comparison to Median Mean Exact R−Precision by Topic (Topics 166 to 200)
1
069 14.29 114 28.57 149 83.33 194 50.00 humDE06Rtde
0.5
080 47.50 115 37.50 160 0.00 195 58.62
Difference
081 48.00 116 45.16 161 0.00 196 83.33 0
082 77.78 117 53.33 162 79.17 197 68.85 −0.5
083 33.33 118 22.22 163 65.96 198 16.67 −1
166 167 168 169 170 171 172 173 174 175 176 177 178 179 190 191 192 193 194 195 196 197 198 199 200
084 20.00 119 73.96 164 44.44 199 75.00 Topic Identifier
085 21.62 130 73.33 165 50.00 200 59.09
116
hummingbird humDE06Rtd AH-ROBUST-MONO-DE-CLEF2006
Overall statistics for 100 queries : Priority 1
Total number of documents over all queries Query Construction AUTOMATIC
Retrieved 100,000 Source Language German
Relevant 3,641 Topic Fields title, description
Relevant retrieved 3,322 Pooled false
Geometric Mean Average Precision 0.1959 same as humDE06Rtd0
Binary Preference (BPREF) 0.4608
Interploated Recall (%) Precision Averages (%)
Ad−Hoc Robust Monolingual German track − Interpolated Recall vs Average Precision
100%
0 71.99
humDE06Rtd
10 62.55 90%
20 56.18
80%
30 51.22
40 48.65 70%
50 45.48
Average Precision
60%
60 41.53
70 36.56 50%
80 31.30 40%
90 23.06
30%
100 14.21
Average precision (non-interpolated) for all 20%
relevant documents (averaged over queries)
42.45 10%
0%
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
Interpolated Recall
Mean Average Precision Ad−Hoc Robust Monolingual German track − Box plot of the Topics of the Experiment
Maximum 1.0000
Minimum 0.0000
First Quartile 0.1654
Second Quartile 0.3889
Third Quartile 0.6906
Interquartile range 0.5252
Mean 0.4245
Standard Deviation 0.2966
Lower Outlier Threshold 0.0000
Upper Outlier Threshold 1.0000 0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Mean Average Precision
Mean With No Outliers 0.4245
Std With No Outliers 0.2966
Ad−Hoc Robust Monolingual German track − Distribution of the Topics of the Experiment
10
Number of Topics of the Experiment
8
6
4
2
0
0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Mean Average Precision
Precision averages (%) for individual Ad−Hoc Robust Monolingual German track − Comparison to Median Mean Average Precision by Topic (Topics 041 to 085)
queries 1
humDE06Rtd
0.5
041 80.82 086 41.33 131 2.60 166 3.09
Difference
0
042 72.76 087 27.47 132 79.94 167 68.45
−0.5
043 41.67 088 61.21 133 57.30 168 7.04
044 0.00 089 64.20 134 84.52 169 6.20 −1
041 042 043 044 045 046 047 048 049 060 061 062 063 064 065 066 067 068 069 080 081 082 083 084 085
Topic Identifier
045 57.93 090 10.04 135 32.39 170 0.00 Ad−Hoc Robust Monolingual German track − Comparison to Median Mean Average Precision by Topic (Topics 086 to 130)
1
046 5.71 091 21.15 136 52.20 171 37.55 humDE06Rtd
047 11.81 092 16.83 137 14.29 172 100.00 0.5
Difference
048 21.42 093 83.81 138 51.98 173 98.09 0
049 19.36 094 75.55 139 14.64 174 85.10 −0.5
060 30.40 095 1.71 140 38.48 175 81.80 −1
086 087 088 089 090 091 092 093 094 095 096 097 098 099 110 111 112 113 114 115 116 117 118 119 130
061 39.18 096 19.41 141 84.20 176 98.39 Topic Identifier
Ad−Hoc Robust Monolingual German track − Comparison to Median Mean Average Precision by Topic (Topics 131 to 165)
062 51.87 097 31.34 142 89.84 177 24.24 1
humDE06Rtd
063 71.31 098 30.35 143 61.79 178 41.51 0.5
Difference
064 100.00 099 21.95 144 0.00 179 68.86
0
065 9.62 110 51.71 145 47.05 190 16.25
−0.5
066 84.15 111 6.07 146 0.00 191 0.00
−1
067 40.70 112 79.59 147 26.11 192 74.17 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 160 161 162 163 164 165
Topic Identifier
068 20.55 113 9.13 148 12.20 193 61.23 Ad−Hoc Robust Monolingual German track − Comparison to Median Mean Average Precision by Topic (Topics 166 to 200)
1
069 13.46 114 12.17 149 80.79 194 83.33 humDE06Rtd
0.5
080 45.68 115 19.85 160 20.00 195 44.86
Difference
081 38.59 116 46.83 161 7.14 196 87.33 0
082 69.48 117 18.37 162 82.11 197 65.54 −0.5
083 21.66 118 19.78 163 62.66 198 14.11 −1
166 167 168 169 170 171 172 173 174 175 176 177 178 179 190 191 192 193 194 195 196 197 198 199 200
084 18.82 119 69.25 164 33.29 199 77.59 Topic Identifier
085 15.92 130 61.04 165 46.36 200 35.17
117
hummingbird humDE06Rtd AH-ROBUST-MONO-DE-CLEF2006
Docs Cutoff Levels Precision at DCL (%)
Ad−Hoc Robust Monolingual German track − Retrieved documents vs Precision
100%
5 docs 50.20
humDE06Rtd
10 docs 45.90 90%
15 docs 42.80
80%
20 docs 40.20
30 docs 35.70 70%
100 docs 21.26
60%
200 docs 13.40
R−Precision
500 docs 6.26 50%
1000 docs 3.32 40%
R-Precision (precision after R document retrieved,
where R = Relevant retrieved) 30%
41.68
20%
10%
0%
5 10 15 20 30 100 200 500 1000
Retrieved Documents (logarithmic scale)
Exact R-Precision Ad−Hoc Robust Monolingual German track − Box plot of the Topics of the Experiment
Maximum 1.0000
Minimum 0.0000
First Quartile 0.1775
Second Quartile 0.4303
Third Quartile 0.6216
Interquartile range 0.4441
Mean 0.4168
Standard Deviation 0.2675
Lower Outlier Threshold 0.0000
Upper Outlier Threshold 1.0000 0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Exact R−Precision
Mean With No Outliers 0.4168
Std With No Outliers 0.2675
Ad−Hoc Robust Monolingual German track − Distribution of the Topics of the Experiment
15
Number of Topics of the Experiment
10
5
0
0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Exact R−Precision
Precision averages (%) for individual Ad−Hoc Robust Monolingual German track − Comparison to Median Mean Exact R−Precision by Topic (Topics 041 to 085)
queries 1
humDE06Rtd
0.5
041 75.00 086 48.65 131 3.03 166 0.00
Difference
0
042 67.92 087 29.41 132 75.00 167 50.00
−0.5
043 50.00 088 55.17 133 61.82 168 11.76
044 0.00 089 66.21 134 71.43 169 10.00 −1
041 042 043 044 045 046 047 048 049 060 061 062 063 064 065 066 067 068 069 080 081 082 083 084 085
Topic Identifier
045 57.55 090 12.50 135 35.63 170 0.00 Ad−Hoc Robust Monolingual German track − Comparison to Median Mean Exact R−Precision by Topic (Topics 086 to 130)
1
046 0.00 091 15.62 136 57.14 171 37.50 humDE06Rtd
047 12.50 092 17.65 137 0.00 172 100.00 0.5
Difference
048 23.64 093 77.78 138 48.39 173 90.00 0
049 16.67 094 75.00 139 10.71 174 80.56 −0.5
060 43.86 095 4.55 140 43.21 175 63.64 −1
086 087 088 089 090 091 092 093 094 095 096 097 098 099 110 111 112 113 114 115 116 117 118 119 130
061 48.84 096 29.63 141 62.50 176 91.89 Topic Identifier
Ad−Hoc Robust Monolingual German track − Comparison to Median Mean Exact R−Precision by Topic (Topics 131 to 165)
062 44.44 097 33.33 142 81.54 177 28.57 1
humDE06Rtd
063 60.53 098 44.00 143 58.73 178 42.86 0.5
Difference
064 100.00 099 28.09 144 0.00 179 65.31
0
065 15.38 110 54.17 145 50.00 190 22.81
−0.5
066 70.59 111 12.50 146 0.00 191 0.00
−1
067 42.00 112 71.43 147 27.59 192 61.54 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 160 161 162 163 164 165
Topic Identifier
068 31.82 113 16.00 148 16.67 193 61.76 Ad−Hoc Robust Monolingual German track − Comparison to Median Mean Exact R−Precision by Topic (Topics 166 to 200)
1
069 28.57 114 17.86 149 75.00 194 50.00 humDE06Rtd
0.5
080 53.33 115 29.17 160 0.00 195 48.28
Difference
081 42.00 116 45.16 161 0.00 196 83.33 0
082 70.37 117 22.22 162 79.17 197 64.75 −0.5
083 33.33 118 22.22 163 59.57 198 16.67 −1
166 167 168 169 170 171 172 173 174 175 176 177 178 179 190 191 192 193 194 195 196 197 198 199 200
084 20.00 119 70.83 164 38.89 199 72.92 Topic Identifier
085 24.32 130 60.00 165 25.00 200 40.91
118
colesir CoLesIRenTst AH-ROBUST-MONO-EN-CLEF2006
Overall statistics for 100 queries : Priority 1
Total number of documents over all queries Query Construction AUTOMATIC
Retrieved 100,000 Source Language English
Relevant 1,533 Topic Fields title, description
Relevant retrieved 1,379 Pooled false
Geometric Mean Average Precision 0.0841 robust monolingual EN - TEST; automatic; TD
Binary Preference (BPREF) 0.3958
Interploated Recall (%) Precision Averages (%)
Ad−Hoc Robust Monolingual English track − Interpolated Recall vs Average Precision
100%
0 63.96
CoLesIRenTst
10 58.49 90%
20 52.10
80%
30 47.75
40 44.23 70%
50 39.58
Average Precision
60%
60 32.79
70 29.44 50%
80 25.30 40%
90 20.28
30%
100 17.30
Average precision (non-interpolated) for all 20%
relevant documents (averaged over queries)
37.64 10%
0%
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
Interpolated Recall
Mean Average Precision Ad−Hoc Robust Monolingual English track − Box plot of the Topics of the Experiment
Maximum 1.0000
Minimum 0.0000
First Quartile 0.0964
Second Quartile 0.2951
Third Quartile 0.5482
Interquartile range 0.4517
Mean 0.3764
Standard Deviation 0.3138
Lower Outlier Threshold 0.0000
Upper Outlier Threshold 1.0000 0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Mean Average Precision
Mean With No Outliers 0.3764
Std With No Outliers 0.3138
Ad−Hoc Robust Monolingual English track − Distribution of the Topics of the Experiment
20
Number of Topics of the Experiment
15
10
5
0
0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Mean Average Precision
Precision averages (%) for individual Ad−Hoc Robust Monolingual English track − Comparison to Median Mean Average Precision by Topic (Topics 041 to 085)
queries 1
CoLesIRenTst
0.5
041 26.85 086 28.28 131 40.26 166 0.00
Difference
0
042 21.00 087 34.40 132 0.00 167 23.94
−0.5
043 25.36 088 50.00 133 14.82 168 38.53
044 52.94 089 56.69 134 75.00 169 0.49 −1
041 042 043 044 045 046 047 048 049 060 061 062 063 064 065 066 067 068 069 080 081 082 083 084 085
Topic Identifier
045 57.39 090 27.39 135 16.79 170 100.00 Ad−Hoc Robust Monolingual English track − Comparison to Median Mean Average Precision by Topic (Topics 086 to 130)
1
046 24.67 091 54.15 136 25.00 171 9.04 CoLesIRenTst
047 54.48 092 53.80 137 25.00 172 9.04 0.5
Difference
048 20.78 093 0.00 138 41.49 173 100.00 0
049 28.72 094 77.39 139 3.97 174 100.00 −0.5
060 0.00 095 25.99 140 34.18 175 87.71 −1
086 087 088 089 090 091 092 093 094 095 096 097 098 099 110 111 112 113 114 115 116 117 118 119 130
061 39.52 096 0.00 141 100.00 176 90.72 Topic Identifier
Ad−Hoc Robust Monolingual English track − Comparison to Median Mean Average Precision by Topic (Topics 131 to 165)
062 100.00 097 100.00 142 88.74 177 1.58 1
CoLesIRenTst
063 50.00 098 100.00 143 82.68 178 14.07 0.5
Difference
064 55.15 099 35.62 144 58.68 179 28.34
0
065 16.34 110 0.00 145 12.15 190 6.24
−0.5
066 75.48 111 10.02 146 32.69 191 0.00
−1
067 2.50 112 58.33 147 41.42 192 47.30 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 160 161 162 163 164 165
Topic Identifier
068 40.96 113 0.44 148 4.29 193 53.35 Ad−Hoc Robust Monolingual English track − Comparison to Median Mean Average Precision by Topic (Topics 166 to 200)
1
069 19.42 114 49.28 149 0.00 194 25.00 CoLesIRenTst
0.5
080 9.26 115 17.16 160 6.48 195 0.00
Difference
081 76.08 116 30.30 161 0.00 196 50.00 0
082 56.64 117 0.00 162 25.00 197 24.94 −0.5
083 4.51 118 0.00 163 84.69 198 100.00 −1
166 167 168 169 170 171 172 173 174 175 176 177 178 179 190 191 192 193 194 195 196 197 198 199 200
084 44.71 119 77.42 164 3.60 199 48.67 Topic Identifier
085 18.57 130 34.90 165 100.00 200 45.56
119
colesir CoLesIRenTst AH-ROBUST-MONO-EN-CLEF2006
Docs Cutoff Levels Precision at DCL (%)
Ad−Hoc Robust Monolingual English track − Retrieved documents vs Precision
100%
5 docs 37.00
CoLesIRenTst
10 docs 28.60 90%
15 docs 24.93
80%
20 docs 21.85
30 docs 18.20 70%
100 docs 9.26
60%
200 docs 5.61
R−Precision
500 docs 2.61 50%
1000 docs 1.38 40%
R-Precision (precision after R document retrieved,
where R = Relevant retrieved) 30%
36.21
20%
10%
0%
5 10 15 20 30 100 200 500 1000
Retrieved Documents (logarithmic scale)
Exact R-Precision Ad−Hoc Robust Monolingual English track − Box plot of the Topics of the Experiment
Maximum 1.0000
Minimum 0.0000
First Quartile 0.0000
Second Quartile 0.3258
Third Quartile 0.5000
Interquartile range 0.5000
Mean 0.3621
Standard Deviation 0.3150
Lower Outlier Threshold 0.0000
Upper Outlier Threshold 1.0000 0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Exact R−Precision
Mean With No Outliers 0.3621
Std With No Outliers 0.3150
Ad−Hoc Robust Monolingual English track − Distribution of the Topics of the Experiment
30
Number of Topics of the Experiment
25
20
15
10
5
0
0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Exact R−Precision
Precision averages (%) for individual Ad−Hoc Robust Monolingual English track − Comparison to Median Mean Exact R−Precision by Topic (Topics 041 to 085)
queries 1
CoLesIRenTst
0.5
041 25.00 086 40.24 131 45.00 166 0.00
Difference
0
042 29.41 087 16.67 132 0.00 167 40.00
−0.5
043 27.27 088 0.00 133 40.00 168 38.89
044 50.00 089 50.00 134 75.00 169 0.00 −1
041 042 043 044 045 046 047 048 049 060 061 062 063 064 065 066 067 068 069 080 081 082 083 084 085
Topic Identifier
045 60.61 090 25.00 135 40.00 170 100.00 Ad−Hoc Robust Monolingual English track − Comparison to Median Mean Exact R−Precision by Topic (Topics 086 to 130)
1
046 33.33 091 42.86 136 0.00 171 15.79 CoLesIRenTst
047 48.78 092 59.26 137 50.00 172 0.00 0.5
Difference
048 24.24 093 0.00 138 50.00 173 100.00 0
049 35.71 094 66.67 139 0.00 174 100.00 −0.5
060 0.00 095 27.08 140 44.29 175 85.71 −1
086 087 088 089 090 091 092 093 094 095 096 097 098 099 110 111 112 113 114 115 116 117 118 119 130
061 40.00 096 0.00 141 100.00 176 87.18 Topic Identifier
Ad−Hoc Robust Monolingual English track − Comparison to Median Mean Exact R−Precision by Topic (Topics 131 to 165)
062 100.00 097 100.00 142 75.00 177 0.00 1
CoLesIRenTst
063 0.00 098 100.00 143 74.36 178 25.00 0.5
Difference
064 70.00 099 28.57 144 66.67 179 31.82
0
065 20.00 110 0.00 145 13.16 190 11.11
−0.5
066 72.73 111 20.00 146 50.00 191 0.00
−1
067 0.00 112 50.00 147 43.14 192 33.33 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 160 161 162 163 164 165
Topic Identifier
068 28.57 113 0.00 148 16.67 193 42.86 Ad−Hoc Robust Monolingual English track − Comparison to Median Mean Exact R−Precision by Topic (Topics 166 to 200)
1
069 25.00 114 42.86 149 0.00 194 0.00 CoLesIRenTst
0.5
080 16.07 115 20.00 160 0.00 195 0.00
Difference
081 62.50 116 27.27 161 0.00 196 0.00 0
082 50.00 117 0.00 162 0.00 197 24.00 −0.5
083 0.00 118 0.00 163 78.69 198 100.00 −1
166 167 168 169 170 171 172 173 174 175 176 177 178 179 190 191 192 193 194 195 196 197 198 199 200
084 46.67 119 71.43 164 7.41 199 50.00 Topic Identifier
085 12.50 130 44.44 165 100.00 200 55.56
120
daedalus enFSenR2S AH-ROBUST-MONO-EN-CLEF2006
Overall statistics for 100 queries : Priority 4
Total number of documents over all queries Query Construction AUTOMATIC
Retrieved 100,000 Source Language English
Relevant 1,533 Topic Fields title, description
Relevant retrieved 1,465 Pooled false
Geometric Mean Average Precision 0.0893 Mandatory TD run
Binary Preference (BPREF) 0.4176
Interploated Recall (%) Precision Averages (%)
Ad−Hoc Robust Monolingual English track − Interpolated Recall vs Average Precision
100%
0 65.70
enFSenR2S
10 61.34 90%
20 57.50
80%
30 51.96
40 47.62 70%
50 42.90
Average Precision
60%
60 34.04
70 29.74 50%
80 24.80 40%
90 21.21
30%
100 16.14
Average precision (non-interpolated) for all 20%
relevant documents (averaged over queries)
39.69 10%
0%
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
Interpolated Recall
Mean Average Precision Ad−Hoc Robust Monolingual English track − Box plot of the Topics of the Experiment
Maximum 1.0000
Minimum 0.0000
First Quartile 0.1136
Second Quartile 0.3855
Third Quartile 0.5847
Interquartile range 0.4711
Mean 0.3969
Standard Deviation 0.3050
Lower Outlier Threshold 0.0000
Upper Outlier Threshold 1.0000 0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Mean Average Precision
Mean With No Outliers 0.3969
Std With No Outliers 0.3050
Ad−Hoc Robust Monolingual English track − Distribution of the Topics of the Experiment
20
Number of Topics of the Experiment
15
10
5
0
0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Mean Average Precision
Precision averages (%) for individual Ad−Hoc Robust Monolingual English track − Comparison to Median Mean Average Precision by Topic (Topics 041 to 085)
queries 1
enFSenR2S
0.5
041 16.58 086 44.57 131 40.13 166 0.00
Difference
0
042 27.85 087 54.47 132 0.00 167 27.45
−0.5
043 50.55 088 11.11 133 24.29 168 43.98
044 52.50 089 68.75 134 75.18 169 2.57 −1
041 042 043 044 045 046 047 048 049 060 061 062 063 064 065 066 067 068 069 080 081 082 083 084 085
Topic Identifier
045 67.21 090 32.62 135 30.00 170 25.00 Ad−Hoc Robust Monolingual English track − Comparison to Median Mean Average Precision by Topic (Topics 086 to 130)
1
046 38.19 091 59.22 136 50.00 171 12.60 enFSenR2S
047 60.24 092 68.57 137 50.00 172 10.50 0.5
Difference
048 11.14 093 0.00 138 38.92 173 100.00 0
049 17.01 094 71.45 139 4.19 174 100.00 −0.5
060 0.00 095 28.14 140 36.21 175 75.55 −1
086 087 088 089 090 091 092 093 094 095 096 097 098 099 110 111 112 113 114 115 116 117 118 119 130
061 49.86 096 0.00 141 100.00 176 93.53 Topic Identifier
Ad−Hoc Robust Monolingual English track − Comparison to Median Mean Average Precision by Topic (Topics 131 to 165)
062 100.00 097 100.00 142 77.14 177 39.63 1
enFSenR2S
063 0.30 098 100.00 143 84.01 178 5.31 0.5
Difference
064 23.14 099 28.28 144 51.67 179 2.70
0
065 36.59 110 0.00 145 51.44 190 72.65
−0.5
066 77.67 111 29.89 146 51.20 191 0.00
−1
067 25.17 112 50.56 147 56.18 192 57.66 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 160 161 162 163 164 165
Topic Identifier
068 11.57 113 0.12 148 6.10 193 52.52 Ad−Hoc Robust Monolingual English track − Comparison to Median Mean Average Precision by Topic (Topics 166 to 200)
1
069 40.76 114 47.50 149 0.00 194 1.27 enFSenR2S
0.5
080 43.98 115 15.14 160 10.05 195 0.00
Difference
081 89.94 116 33.21 161 0.00 196 50.00 0
082 57.71 117 0.00 162 14.29 197 23.50 −0.5
083 34.33 118 0.00 163 87.59 198 11.11 −1
166 167 168 169 170 171 172 173 174 175 176 177 178 179 190 191 192 193 194 195 196 197 198 199 200
084 69.76 119 77.83 164 16.94 199 65.53 Topic Identifier
085 46.33 130 39.02 165 100.00 200 31.87
121
daedalus enFSenR2S AH-ROBUST-MONO-EN-CLEF2006
Docs Cutoff Levels Precision at DCL (%)
Ad−Hoc Robust Monolingual English track − Retrieved documents vs Precision
100%
5 docs 40.00
enFSenR2S
10 docs 32.10 90%
15 docs 28.07
80%
20 docs 24.90
30 docs 20.67 70%
100 docs 10.45
60%
200 docs 6.20
R−Precision
500 docs 2.80 50%
1000 docs 1.47 40%
R-Precision (precision after R document retrieved,
where R = Relevant retrieved) 30%
36.69
20%
10%
0%
5 10 15 20 30 100 200 500 1000
Retrieved Documents (logarithmic scale)
Exact R-Precision Ad−Hoc Robust Monolingual English track − Box plot of the Topics of the Experiment
Maximum 1.0000
Minimum 0.0000
First Quartile 0.0000
Second Quartile 0.3964
Third Quartile 0.5541
Interquartile range 0.5541
Mean 0.3669
Standard Deviation 0.3029
Lower Outlier Threshold 0.0000
Upper Outlier Threshold 1.0000 0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Exact R−Precision
Mean With No Outliers 0.3669
Std With No Outliers 0.3029
Ad−Hoc Robust Monolingual English track − Distribution of the Topics of the Experiment
30
Number of Topics of the Experiment
25
20
15
10
5
0
0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Exact R−Precision
Precision averages (%) for individual Ad−Hoc Robust Monolingual English track − Comparison to Median Mean Exact R−Precision by Topic (Topics 041 to 085)
queries 1
enFSenR2S
0.5
041 12.50 086 50.00 131 32.50 166 0.00
Difference
0
042 31.37 087 33.33 132 0.00 167 40.00
−0.5
043 45.45 088 0.00 133 30.00 168 33.33
044 50.00 089 50.00 134 75.00 169 0.00 −1
041 042 043 044 045 046 047 048 049 060 061 062 063 064 065 066 067 068 069 080 081 082 083 084 085
Topic Identifier
045 63.64 090 25.00 135 40.00 170 0.00 Ad−Hoc Robust Monolingual English track − Comparison to Median Mean Exact R−Precision by Topic (Topics 086 to 130)
1
046 55.56 091 42.86 136 0.00 171 10.53 enFSenR2S
047 56.10 092 62.96 137 50.00 172 0.00 0.5
Difference
048 15.15 093 0.00 138 50.00 173 100.00 0
049 21.43 094 66.67 139 0.00 174 100.00 −0.5
060 0.00 095 35.42 140 45.71 175 57.14 −1
086 087 088 089 090 091 092 093 094 095 096 097 098 099 110 111 112 113 114 115 116 117 118 119 130
061 40.00 096 0.00 141 100.00 176 84.62 Topic Identifier
Ad−Hoc Robust Monolingual English track − Comparison to Median Mean Exact R−Precision by Topic (Topics 131 to 165)
062 100.00 097 100.00 142 87.50 177 46.15 1
enFSenR2S
063 0.00 098 100.00 143 74.36 178 0.00 0.5
Difference
064 20.00 099 28.57 144 33.33 179 0.00
0
065 33.33 110 0.00 145 55.26 190 66.67
−0.5
066 72.73 111 40.00 146 50.00 191 0.00
−1
067 50.00 112 40.00 147 49.02 192 50.00 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 160 161 162 163 164 165
Topic Identifier
068 28.57 113 0.00 148 16.67 193 57.14 Ad−Hoc Robust Monolingual English track − Comparison to Median Mean Exact R−Precision by Topic (Topics 166 to 200)
1
069 37.50 114 42.86 149 0.00 194 0.00 enFSenR2S
0.5
080 39.29 115 20.00 160 0.00 195 0.00
Difference
081 75.00 116 27.27 161 0.00 196 0.00 0
082 60.00 117 0.00 162 0.00 197 14.00 −0.5
083 40.00 118 0.00 163 78.69 198 0.00 −1
166 167 168 169 170 171 172 173 174 175 176 177 178 179 190 191 192 193 194 195 196 197 198 199 200
084 66.67 119 66.07 164 14.81 199 50.00 Topic Identifier
085 43.75 130 42.86 165 100.00 200 44.44
122
daedalus enFSenR3S AH-ROBUST-MONO-EN-CLEF2006
Overall statistics for 100 queries : Priority 3
Total number of documents over all queries Query Construction AUTOMATIC
Retrieved 100,000 Source Language English
Relevant 1,533 Topic Fields title, description, narrative
Relevant retrieved 1,484 Pooled false
Geometric Mean Average Precision 0.1016 TDN run
Binary Preference (BPREF) 0.4520
Interploated Recall (%) Precision Averages (%)
Ad−Hoc Robust Monolingual English track − Interpolated Recall vs Average Precision
100%
0 67.23
enFSenR3S
10 64.58 90%
20 60.61
80%
30 55.34
40 50.51 70%
50 45.30
Average Precision
60%
60 38.75
70 34.60 50%
80 28.48 40%
90 22.90
30%
100 17.25
Average precision (non-interpolated) for all 20%
relevant documents (averaged over queries)
42.89 10%
0%
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
Interpolated Recall
Mean Average Precision Ad−Hoc Robust Monolingual English track − Box plot of the Topics of the Experiment
Maximum 1.0000
Minimum 0.0000
First Quartile 0.1414
Second Quartile 0.4498
Third Quartile 0.6378
Interquartile range 0.4963
Mean 0.4289
Standard Deviation 0.3093
Lower Outlier Threshold 0.0000
Upper Outlier Threshold 1.0000 0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Mean Average Precision
Mean With No Outliers 0.4289
Std With No Outliers 0.3093
Ad−Hoc Robust Monolingual English track − Distribution of the Topics of the Experiment
20
Number of Topics of the Experiment
15
10
5
0
0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Mean Average Precision
Precision averages (%) for individual Ad−Hoc Robust Monolingual English track − Comparison to Median Mean Average Precision by Topic (Topics 041 to 085)
queries 1
enFSenR3S
0.5
041 20.46 086 55.42 131 44.69 166 0.00
Difference
0
042 29.82 087 62.15 132 0.00 167 30.67
−0.5
043 54.10 088 14.29 133 24.29 168 45.98
044 52.78 089 100.00 134 75.00 169 3.23 −1
041 042 043 044 045 046 047 048 049 060 061 062 063 064 065 066 067 068 069 080 081 082 083 084 085
Topic Identifier
045 71.51 090 31.84 135 41.26 170 100.00 Ad−Hoc Robust Monolingual English track − Comparison to Median Mean Average Precision by Topic (Topics 086 to 130)
1
046 39.13 091 57.93 136 50.00 171 13.45 enFSenR3S
047 66.48 092 73.28 137 50.00 172 15.12 0.5
Difference
048 29.11 093 0.00 138 58.33 173 100.00 0
049 16.80 094 76.00 139 4.40 174 100.00 −0.5
060 0.00 095 39.88 140 41.43 175 76.42 −1
086 087 088 089 090 091 092 093 094 095 096 097 098 099 110 111 112 113 114 115 116 117 118 119 130
061 60.09 096 0.00 141 100.00 176 94.71 Topic Identifier
Ad−Hoc Robust Monolingual English track − Comparison to Median Mean Average Precision by Topic (Topics 131 to 165)
062 100.00 097 100.00 142 83.39 177 34.97 1
enFSenR3S
063 2.70 098 33.33 143 73.65 178 14.00 0.5
Difference
064 52.45 099 50.25 144 50.74 179 1.93
0
065 57.23 110 0.00 145 68.73 190 74.44
−0.5
066 77.30 111 37.94 146 50.68 191 0.00
−1
067 5.00 112 55.04 147 54.53 192 62.28 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 160 161 162 163 164 165
Topic Identifier
068 20.19 113 0.84 148 7.27 193 52.52 Ad−Hoc Robust Monolingual English track − Comparison to Median Mean Average Precision by Topic (Topics 166 to 200)
1
069 43.33 114 63.57 149 0.00 194 25.00 enFSenR3S
0.5
080 45.27 115 20.54 160 8.89 195 0.00
Difference
081 80.58 116 40.46 161 0.00 196 50.00 0
082 42.04 117 0.00 162 50.00 197 11.93 −0.5
083 15.55 118 0.00 163 85.23 198 2.04 −1
166 167 168 169 170 171 172 173 174 175 176 177 178 179 190 191 192 193 194 195 196 197 198 199 200
084 71.78 119 78.02 164 10.93 199 63.99 Topic Identifier
085 48.15 130 58.28 165 100.00 200 31.87
123
daedalus enFSenR3S AH-ROBUST-MONO-EN-CLEF2006
Docs Cutoff Levels Precision at DCL (%)
Ad−Hoc Robust Monolingual English track − Retrieved documents vs Precision
100%
5 docs 43.20
enFSenR3S
10 docs 34.80 90%
15 docs 29.60
80%
20 docs 26.25
30 docs 21.97 70%
100 docs 10.68
60%
200 docs 6.42
R−Precision
500 docs 2.87 50%
1000 docs 1.48 40%
R-Precision (precision after R document retrieved,
where R = Relevant retrieved) 30%
39.70
20%
10%
0%
5 10 15 20 30 100 200 500 1000
Retrieved Documents (logarithmic scale)
Exact R-Precision Ad−Hoc Robust Monolingual English track − Box plot of the Topics of the Experiment
Maximum 1.0000
Minimum 0.0000
First Quartile 0.0000
Second Quartile 0.4278
Third Quartile 0.6667
Interquartile range 0.6667
Mean 0.3970
Standard Deviation 0.3186
Lower Outlier Threshold 0.0000
Upper Outlier Threshold 1.0000 0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Exact R−Precision
Mean With No Outliers 0.3970
Std With No Outliers 0.3186
Ad−Hoc Robust Monolingual English track − Distribution of the Topics of the Experiment
30
Number of Topics of the Experiment
25
20
15
10
5
0
0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Exact R−Precision
Precision averages (%) for individual Ad−Hoc Robust Monolingual English track − Comparison to Median Mean Exact R−Precision by Topic (Topics 041 to 085)
queries 1
enFSenR3S
0.5
041 25.00 086 56.10 131 40.00 166 0.00
Difference
0
042 37.25 087 50.00 132 0.00 167 40.00
−0.5
043 54.55 088 0.00 133 30.00 168 50.00
044 50.00 089 100.00 134 75.00 169 0.00 −1
041 042 043 044 045 046 047 048 049 060 061 062 063 064 065 066 067 068 069 080 081 082 083 084 085
Topic Identifier
045 66.67 090 25.00 135 40.00 170 100.00 Ad−Hoc Robust Monolingual English track − Comparison to Median Mean Exact R−Precision by Topic (Topics 086 to 130)
1
046 55.56 091 42.86 136 0.00 171 5.26 enFSenR3S
047 60.98 092 66.67 137 50.00 172 14.29 0.5
Difference
048 33.33 093 0.00 138 50.00 173 100.00 0
049 14.29 094 66.67 139 0.00 174 100.00 −0.5
060 0.00 095 42.71 140 50.00 175 71.43 −1
086 087 088 089 090 091 092 093 094 095 096 097 098 099 110 111 112 113 114 115 116 117 118 119 130
061 40.00 096 0.00 141 100.00 176 87.18 Topic Identifier
Ad−Hoc Robust Monolingual English track − Comparison to Median Mean Exact R−Precision by Topic (Topics 131 to 165)
062 100.00 097 100.00 142 87.50 177 23.08 1
enFSenR3S
063 0.00 098 0.00 143 69.23 178 25.00 0.5
Difference
064 70.00 099 42.86 144 33.33 179 4.55
0
065 60.00 110 0.00 145 68.42 190 66.67
−0.5
066 72.73 111 40.00 146 50.00 191 0.00
−1
067 0.00 112 50.00 147 52.94 192 66.67 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 160 161 162 163 164 165
Topic Identifier
068 42.86 113 0.00 148 16.67 193 57.14 Ad−Hoc Robust Monolingual English track − Comparison to Median Mean Exact R−Precision by Topic (Topics 166 to 200)
1
069 37.50 114 71.43 149 0.00 194 0.00 enFSenR3S
0.5
080 46.43 115 24.00 160 0.00 195 0.00
Difference
081 75.00 116 45.45 161 0.00 196 0.00 0
082 40.00 117 0.00 162 0.00 197 10.00 −0.5
083 0.00 118 0.00 163 77.05 198 0.00 −1
166 167 168 169 170 171 172 173 174 175 176 177 178 179 190 191 192 193 194 195 196 197 198 199 200
084 66.67 119 69.64 164 11.11 199 60.00 Topic Identifier
085 46.88 130 53.97 165 100.00 200 44.44
124
daedalus enFSenR5S AH-ROBUST-MONO-EN-CLEF2006
Overall statistics for 100 queries : Priority 2
Total number of documents over all queries Query Construction AUTOMATIC
Retrieved 100,000 Source Language English
Relevant 1,533 Topic Fields title, description, narrative
Relevant retrieved 1,473 Pooled false
Geometric Mean Average Precision 0.0968 TDN run
Binary Preference (BPREF) 0.4474
Interploated Recall (%) Precision Averages (%)
Ad−Hoc Robust Monolingual English track − Interpolated Recall vs Average Precision
100%
0 67.98
enFSenR5S
10 65.55 90%
20 60.93
80%
30 55.93
40 50.46 70%
50 46.95
Average Precision
60%
60 38.24
70 32.52 50%
80 27.72 40%
90 23.06
30%
100 17.47
Average precision (non-interpolated) for all 20%
relevant documents (averaged over queries)
43.02 10%
0%
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
Interpolated Recall
Mean Average Precision Ad−Hoc Robust Monolingual English track − Box plot of the Topics of the Experiment
Maximum 1.0000
Minimum 0.0000
First Quartile 0.1361
Second Quartile 0.4538
Third Quartile 0.6510
Interquartile range 0.5149
Mean 0.4302
Standard Deviation 0.3128
Lower Outlier Threshold 0.0000
Upper Outlier Threshold 1.0000 0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Mean Average Precision
Mean With No Outliers 0.4302
Std With No Outliers 0.3128
Ad−Hoc Robust Monolingual English track − Distribution of the Topics of the Experiment
20
Number of Topics of the Experiment
15
10
5
0
0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Mean Average Precision
Precision averages (%) for individual Ad−Hoc Robust Monolingual English track − Comparison to Median Mean Average Precision by Topic (Topics 041 to 085)
queries 1
enFSenR5S
0.5
041 17.36 086 51.33 131 45.34 166 0.00
Difference
0
042 26.95 087 58.69 132 0.00 167 31.61
−0.5
043 60.02 088 16.67 133 35.24 168 49.70
044 52.00 089 64.51 134 75.00 169 2.62 −1
041 042 043 044 045 046 047 048 049 060 061 062 063 064 065 066 067 068 069 080 081 082 083 084 085
Topic Identifier
045 74.45 090 50.49 135 10.92 170 100.00 Ad−Hoc Robust Monolingual English track − Comparison to Median Mean Average Precision by Topic (Topics 086 to 130)
1
046 36.35 091 32.61 136 50.00 171 9.75 enFSenR5S
047 65.70 092 72.98 137 50.00 172 14.45 0.5
Difference
048 24.77 093 0.00 138 56.91 173 100.00 0
049 16.87 094 78.64 139 4.33 174 100.00 −0.5
060 0.00 095 31.71 140 30.67 175 73.45 −1
086 087 088 089 090 091 092 093 094 095 096 097 098 099 110 111 112 113 114 115 116 117 118 119 130
061 57.69 096 0.00 141 100.00 176 94.15 Topic Identifier
Ad−Hoc Robust Monolingual English track − Comparison to Median Mean Average Precision by Topic (Topics 131 to 165)
062 100.00 097 100.00 142 77.14 177 37.24 1
enFSenR5S
063 0.33 098 100.00 143 82.42 178 27.22 0.5
Difference
064 20.77 099 39.29 144 50.00 179 1.88
0
065 46.47 110 0.00 145 55.15 190 73.74
−0.5
066 77.19 111 34.83 146 50.26 191 0.00
−1
067 12.66 112 54.12 147 79.90 192 53.64 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 160 161 162 163 164 165
Topic Identifier
068 41.15 113 0.30 148 6.22 193 46.37 Ad−Hoc Robust Monolingual English track − Comparison to Median Mean Average Precision by Topic (Topics 166 to 200)
1
069 42.48 114 61.39 149 0.00 194 1.61 enFSenR5S
0.5
080 44.87 115 18.62 160 5.56 195 0.00
Difference
081 87.01 116 38.69 161 0.00 196 50.00 0
082 62.66 117 0.00 162 50.00 197 42.26 −0.5
083 25.87 118 0.00 163 86.72 198 2.00 −1
166 167 168 169 170 171 172 173 174 175 176 177 178 179 190 191 192 193 194 195 196 197 198 199 200
084 72.19 119 70.98 164 12.78 199 68.19 Topic Identifier
085 45.41 130 56.87 165 100.00 200 63.17
125
daedalus enFSenR5S AH-ROBUST-MONO-EN-CLEF2006
Docs Cutoff Levels Precision at DCL (%)
Ad−Hoc Robust Monolingual English track − Retrieved documents vs Precision
100%
5 docs 43.80
enFSenR5S
10 docs 34.00 90%
15 docs 29.67
80%
20 docs 26.25
30 docs 22.00 70%
100 docs 10.56
60%
200 docs 6.37
R−Precision
500 docs 2.87 50%
1000 docs 1.47 40%
R-Precision (precision after R document retrieved,
where R = Relevant retrieved) 30%
40.01
20%
10%
0%
5 10 15 20 30 100 200 500 1000
Retrieved Documents (logarithmic scale)
Exact R-Precision Ad−Hoc Robust Monolingual English track − Box plot of the Topics of the Experiment
Maximum 1.0000
Minimum 0.0000
First Quartile 0.0526
Second Quartile 0.4000
Third Quartile 0.6036
Interquartile range 0.5509
Mean 0.4001
Standard Deviation 0.3119
Lower Outlier Threshold 0.0000
Upper Outlier Threshold 1.0000 0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Exact R−Precision
Mean With No Outliers 0.4001
Std With No Outliers 0.3119
Ad−Hoc Robust Monolingual English track − Distribution of the Topics of the Experiment
25
Number of Topics of the Experiment
20
15
10
5
0
0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Exact R−Precision
Precision averages (%) for individual Ad−Hoc Robust Monolingual English track − Comparison to Median Mean Exact R−Precision by Topic (Topics 041 to 085)
queries 1
enFSenR5S
0.5
041 25.00 086 56.10 131 40.00 166 0.00
Difference
0
042 35.29 087 50.00 132 0.00 167 40.00
−0.5
043 54.55 088 0.00 133 30.00 168 38.89
044 50.00 089 50.00 134 75.00 169 0.00 −1
041 042 043 044 045 046 047 048 049 060 061 062 063 064 065 066 067 068 069 080 081 082 083 084 085
Topic Identifier
045 69.70 090 50.00 135 20.00 170 100.00 Ad−Hoc Robust Monolingual English track − Comparison to Median Mean Exact R−Precision by Topic (Topics 086 to 130)
1
046 44.44 091 28.57 136 0.00 171 10.53 enFSenR5S
047 60.98 092 66.67 137 50.00 172 28.57 0.5
Difference
048 30.30 093 0.00 138 50.00 173 100.00 0
049 14.29 094 66.67 139 0.00 174 100.00 −0.5
060 0.00 095 27.08 140 37.14 175 57.14 −1
086 087 088 089 090 091 092 093 094 095 096 097 098 099 110 111 112 113 114 115 116 117 118 119 130
061 60.00 096 0.00 141 100.00 176 84.62 Topic Identifier
Ad−Hoc Robust Monolingual English track − Comparison to Median Mean Exact R−Precision by Topic (Topics 131 to 165)
062 100.00 097 100.00 142 87.50 177 38.46 1
enFSenR5S
063 0.00 098 100.00 143 76.92 178 25.00 0.5
Difference
064 30.00 099 28.57 144 33.33 179 0.00
0
065 53.33 110 0.00 145 52.63 190 66.67
−0.5
066 72.73 111 40.00 146 50.00 191 0.00
−1
067 0.00 112 50.00 147 72.55 192 50.00 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 160 161 162 163 164 165
Topic Identifier
068 57.14 113 0.00 148 16.67 193 28.57 Ad−Hoc Robust Monolingual English track − Comparison to Median Mean Exact R−Precision by Topic (Topics 166 to 200)
1
069 37.50 114 71.43 149 0.00 194 0.00 enFSenR5S
0.5
080 42.86 115 24.00 160 0.00 195 0.00
Difference
081 75.00 116 45.45 161 0.00 196 0.00 0
082 60.00 117 0.00 162 0.00 197 44.00 −0.5
083 20.00 118 0.00 163 80.33 198 0.00 −1
166 167 168 169 170 171 172 173 174 175 176 177 178 179 190 191 192 193 194 195 196 197 198 199 200
084 66.67 119 60.71 164 11.11 199 60.00 Topic Identifier
085 46.88 130 57.14 165 100.00 200 66.67
126
daedalus enyenRFS3456 AH-ROBUST-MONO-EN-CLEF2006
Overall statistics for 100 queries : Priority 1
Total number of documents over all queries Query Construction AUTOMATIC
Retrieved 100,000 Source Language English
Relevant 1,533 Topic Fields title, description, narrative
Relevant retrieved 1,475 Pooled false
Geometric Mean Average Precision 0.0999 Average run
Binary Preference (BPREF) 0.4500
Interploated Recall (%) Precision Averages (%)
Ad−Hoc Robust Monolingual English track − Interpolated Recall vs Average Precision
100%
0 68.42
enyenRFS3456
10 65.51 90%
20 60.10
80%
30 56.13
40 51.16 70%
50 47.08
Average Precision
60%
60 39.19
70 33.60 50%
80 28.51 40%
90 23.57
30%
100 17.77
Average precision (non-interpolated) for all 20%
relevant documents (averaged over queries)
43.38 10%
0%
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
Interpolated Recall
Mean Average Precision Ad−Hoc Robust Monolingual English track − Box plot of the Topics of the Experiment
Maximum 1.0000
Minimum 0.0000
First Quartile 0.1498
Second Quartile 0.4630
Third Quartile 0.6558
Interquartile range 0.5061
Mean 0.4338
Standard Deviation 0.3135
Lower Outlier Threshold 0.0000
Upper Outlier Threshold 1.0000 0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Mean Average Precision
Mean With No Outliers 0.4338
Std With No Outliers 0.3135
Ad−Hoc Robust Monolingual English track − Distribution of the Topics of the Experiment
20
Number of Topics of the Experiment
15
10
5
0
0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Mean Average Precision
Precision averages (%) for individual Ad−Hoc Robust Monolingual English track − Comparison to Median Mean Average Precision by Topic (Topics 041 to 085)
queries 1
enyenRFS3456
0.5
041 17.94 086 52.93 131 45.08 166 0.00
Difference
0
042 27.93 087 58.52 132 0.00 167 30.75
−0.5
043 57.90 088 16.67 133 35.94 168 49.07
044 52.27 089 76.79 134 75.00 169 2.90 −1
041 042 043 044 045 046 047 048 049 060 061 062 063 064 065 066 067 068 069 080 081 082 083 084 085
Topic Identifier
045 73.85 090 38.40 135 22.35 170 100.00 Ad−Hoc Robust Monolingual English track − Comparison to Median Mean Average Precision by Topic (Topics 086 to 130)
1
046 36.10 091 48.74 136 50.00 171 12.68 enyenRFS3456
047 65.64 092 72.51 137 50.00 172 15.26 0.5
Difference
048 24.86 093 0.00 138 57.84 173 100.00 0
049 16.81 094 78.16 139 4.37 174 100.00 −0.5
060 0.00 095 34.30 140 35.09 175 74.92 −1
086 087 088 089 090 091 092 093 094 095 096 097 098 099 110 111 112 113 114 115 116 117 118 119 130
061 58.78 096 0.00 141 100.00 176 94.53 Topic Identifier
Ad−Hoc Robust Monolingual English track − Comparison to Median Mean Average Precision by Topic (Topics 131 to 165)
062 100.00 097 100.00 142 83.39 177 37.07 1
enyenRFS3456
063 0.91 098 100.00 143 81.70 178 14.70 0.5
Difference
064 29.86 099 40.73 144 51.67 179 2.24
0
065 49.33 110 0.00 145 59.09 190 75.31
−0.5
066 76.82 111 36.70 146 50.42 191 0.00
−1
067 12.62 112 54.22 147 78.72 192 56.14 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 160 161 162 163 164 165
Topic Identifier
068 33.16 113 0.38 148 6.76 193 46.93 Ad−Hoc Robust Monolingual English track − Comparison to Median Mean Average Precision by Topic (Topics 166 to 200)
1
069 42.73 114 62.00 149 0.00 194 1.64 enyenRFS3456
0.5
080 45.66 115 19.45 160 7.07 195 0.00
Difference
081 85.53 116 38.86 161 0.00 196 50.00 0
082 62.45 117 0.00 162 50.00 197 26.32 −0.5
083 23.19 118 0.00 163 87.66 198 2.56 −1
166 167 168 169 170 171 172 173 174 175 176 177 178 179 190 191 192 193 194 195 196 197 198 199 200
084 74.16 119 73.93 164 13.15 199 65.53 Topic Identifier
085 47.59 130 56.93 165 100.00 200 57.76
127
daedalus enyenRFS3456 AH-ROBUST-MONO-EN-CLEF2006
Docs Cutoff Levels Precision at DCL (%)
Ad−Hoc Robust Monolingual English track − Retrieved documents vs Precision
100%
5 docs 43.60
enyenRFS3456
10 docs 34.70 90%
15 docs 29.93
80%
20 docs 26.30
30 docs 22.03 70%
100 docs 10.69
60%
200 docs 6.45
R−Precision
500 docs 2.89 50%
1000 docs 1.48 40%
R-Precision (precision after R document retrieved,
where R = Relevant retrieved) 30%
40.19
20%
10%
0%
5 10 15 20 30 100 200 500 1000
Retrieved Documents (logarithmic scale)
Exact R-Precision Ad−Hoc Robust Monolingual English track − Box plot of the Topics of the Experiment
Maximum 1.0000
Minimum 0.0000
First Quartile 0.0000
Second Quartile 0.4196
Third Quartile 0.6075
Interquartile range 0.6075
Mean 0.4019
Standard Deviation 0.3147
Lower Outlier Threshold 0.0000
Upper Outlier Threshold 1.0000 0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Exact R−Precision
Mean With No Outliers 0.4019
Std With No Outliers 0.3147
Ad−Hoc Robust Monolingual English track − Distribution of the Topics of the Experiment
30
Number of Topics of the Experiment
25
20
15
10
5
0
0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Exact R−Precision
Precision averages (%) for individual Ad−Hoc Robust Monolingual English track − Comparison to Median Mean Exact R−Precision by Topic (Topics 041 to 085)
queries 1
enyenRFS3456
0.5
041 25.00 086 56.10 131 42.50 166 0.00
Difference
0
042 35.29 087 50.00 132 0.00 167 40.00
−0.5
043 54.55 088 0.00 133 30.00 168 38.89
044 50.00 089 50.00 134 75.00 169 0.00 −1
041 042 043 044 045 046 047 048 049 060 061 062 063 064 065 066 067 068 069 080 081 082 083 084 085
Topic Identifier
045 66.67 090 25.00 135 40.00 170 100.00 Ad−Hoc Robust Monolingual English track − Comparison to Median Mean Exact R−Precision by Topic (Topics 086 to 130)
1
046 55.56 091 42.86 136 0.00 171 5.26 enyenRFS3456
047 60.98 092 66.67 137 50.00 172 28.57 0.5
Difference
048 33.33 093 0.00 138 50.00 173 100.00 0
049 14.29 094 66.67 139 0.00 174 100.00 −0.5
060 0.00 095 31.25 140 41.43 175 71.43 −1
086 087 088 089 090 091 092 093 094 095 096 097 098 099 110 111 112 113 114 115 116 117 118 119 130
061 60.00 096 0.00 141 100.00 176 84.62 Topic Identifier
Ad−Hoc Robust Monolingual English track − Comparison to Median Mean Exact R−Precision by Topic (Topics 131 to 165)
062 100.00 097 100.00 142 87.50 177 38.46 1
enyenRFS3456
063 0.00 098 100.00 143 74.36 178 25.00 0.5
Difference
064 30.00 099 28.57 144 33.33 179 0.00
0
065 53.33 110 0.00 145 60.53 190 77.78
−0.5
066 72.73 111 40.00 146 50.00 191 0.00
−1
067 0.00 112 50.00 147 72.55 192 50.00 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 160 161 162 163 164 165
Topic Identifier
068 42.86 113 0.00 148 16.67 193 42.86 Ad−Hoc Robust Monolingual English track − Comparison to Median Mean Exact R−Precision by Topic (Topics 166 to 200)
1
069 37.50 114 71.43 149 0.00 194 0.00 enyenRFS3456
0.5
080 42.86 115 24.00 160 0.00 195 0.00
Difference
081 75.00 116 45.45 161 0.00 196 0.00 0
082 60.00 117 0.00 162 0.00 197 30.00 −0.5
083 0.00 118 0.00 163 80.33 198 0.00 −1
166 167 168 169 170 171 172 173 174 175 176 177 178 179 190 191 192 193 194 195 196 197 198 199 200
084 66.67 119 64.29 164 14.81 199 60.00 Topic Identifier
085 43.75 130 58.73 165 100.00 200 55.56
128
dcu dcunarreng1005 AH-ROBUST-MONO-EN-CLEF2006
Overall statistics for 100 queries : Priority 1
Total number of documents over all queries Query Construction AUTOMATIC
Retrieved 100,000 Source Language English
Relevant 1,533 Topic Fields title, description, narrative
Relevant retrieved 1,508 Pooled false
Geometric Mean Average Precision 0.1200 English mono standard
Binary Preference (BPREF) 0.5144
Interploated Recall (%) Precision Averages (%)
Ad−Hoc Robust Monolingual English track − Interpolated Recall vs Average Precision
100%
0 72.79
dcunarreng1005
10 69.34 90%
20 65.59
80%
30 59.81
40 56.32 70%
50 51.82
Average Precision
60%
60 47.17
70 41.87 50%
80 35.62 40%
90 28.58
30%
100 21.32
Average precision (non-interpolated) for all 20%
relevant documents (averaged over queries)
49.12 10%
0%
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
Interpolated Recall
Mean Average Precision Ad−Hoc Robust Monolingual English track − Box plot of the Topics of the Experiment
Maximum 1.0000
Minimum 0.0000
First Quartile 0.1885
Second Quartile 0.5104
Third Quartile 0.7500
Interquartile range 0.5615
Mean 0.4912
Standard Deviation 0.3342
Lower Outlier Threshold 0.0000
Upper Outlier Threshold 1.0000 0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Mean Average Precision
Mean With No Outliers 0.4912
Std With No Outliers 0.3342
Ad−Hoc Robust Monolingual English track − Distribution of the Topics of the Experiment
20
Number of Topics of the Experiment
15
10
5
0
0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Mean Average Precision
Precision averages (%) for individual Ad−Hoc Robust Monolingual English track − Comparison to Median Mean Average Precision by Topic (Topics 041 to 085)
queries 1
dcunarreng1005
0.5
041 18.74 086 52.09 131 61.99 166 0.00
Difference
0
042 38.70 087 70.44 132 0.00 167 31.05
−0.5
043 82.85 088 100.00 133 14.65 168 59.90
044 6.98 089 68.85 134 75.00 169 7.65 −1
041 042 043 044 045 046 047 048 049 060 061 062 063 064 065 066 067 068 069 080 081 082 083 084 085
Topic Identifier
045 64.50 090 6.30 135 39.36 170 100.00 Ad−Hoc Robust Monolingual English track − Comparison to Median Mean Average Precision by Topic (Topics 086 to 130)
1
046 38.14 091 33.99 136 50.00 171 8.64 dcunarreng1005
047 71.32 092 74.93 137 16.67 172 18.96 0.5
Difference
048 36.11 093 0.00 138 56.43 173 100.00 0
049 21.16 094 79.69 139 4.26 174 100.00 −0.5
060 0.00 095 42.55 140 61.48 175 86.29 −1
086 087 088 089 090 091 092 093 094 095 096 097 098 099 110 111 112 113 114 115 116 117 118 119 130
061 67.96 096 0.00 141 100.00 176 94.38 Topic Identifier
Ad−Hoc Robust Monolingual English track − Comparison to Median Mean Average Precision by Topic (Topics 131 to 165)
062 100.00 097 100.00 142 94.64 177 32.48 1
dcunarreng1005
063 4.55 098 33.33 143 64.51 178 27.16 0.5
Difference
064 83.54 099 62.08 144 75.00 179 87.72
0
065 60.51 110 0.00 145 79.01 190 34.83
−0.5
066 72.70 111 48.69 146 17.34 191 0.00
−1
067 2.48 112 60.19 147 78.73 192 59.88 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 160 161 162 163 164 165
Topic Identifier
068 25.51 113 32.44 148 20.56 193 43.53 Ad−Hoc Robust Monolingual English track − Comparison to Median Mean Average Precision by Topic (Topics 166 to 200)
1
069 61.35 114 64.08 149 0.00 194 100.00 dcunarreng1005
0.5
080 40.38 115 21.81 160 5.49 195 0.00
Difference
081 85.45 116 49.84 161 0.00 196 50.00 0
082 69.02 117 0.00 162 100.00 197 83.01 −0.5
083 28.26 118 0.00 163 88.10 198 100.00 −1
166 167 168 169 170 171 172 173 174 175 176 177 178 179 190 191 192 193 194 195 196 197 198 199 200
084 68.05 119 82.35 164 3.81 199 67.21 Topic Identifier
085 48.49 130 67.09 165 100.00 200 65.09
129
dcu dcunarreng1005 AH-ROBUST-MONO-EN-CLEF2006
Docs Cutoff Levels Precision at DCL (%)
Ad−Hoc Robust Monolingual English track − Retrieved documents vs Precision
100%
5 docs 47.00
dcunarreng1005
10 docs 38.40 90%
15 docs 32.80
80%
20 docs 29.05
30 docs 24.27 70%
100 docs 11.68
60%
200 docs 6.85
R−Precision
500 docs 2.95 50%
1000 docs 1.51 40%
R-Precision (precision after R document retrieved,
where R = Relevant retrieved) 30%
45.72
20%
10%
0%
5 10 15 20 30 100 200 500 1000
Retrieved Documents (logarithmic scale)
Exact R-Precision Ad−Hoc Robust Monolingual English track − Box plot of the Topics of the Experiment
Maximum 1.0000
Minimum 0.0000
First Quartile 0.0917
Second Quartile 0.4939
Third Quartile 0.7071
Interquartile range 0.6155
Mean 0.4572
Standard Deviation 0.3401
Lower Outlier Threshold 0.0000
Upper Outlier Threshold 1.0000 0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Exact R−Precision
Mean With No Outliers 0.4572
Std With No Outliers 0.3401
Ad−Hoc Robust Monolingual English track − Distribution of the Topics of the Experiment
25
Number of Topics of the Experiment
20
15
10
5
0
0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Exact R−Precision
Precision averages (%) for individual Ad−Hoc Robust Monolingual English track − Comparison to Median Mean Exact R−Precision by Topic (Topics 041 to 085)
queries 1
dcunarreng1005
0.5
041 25.00 086 48.78 131 65.00 166 0.00
Difference
0
042 41.18 087 50.00 132 0.00 167 40.00
−0.5
043 81.82 088 100.00 133 10.00 168 55.56
044 0.00 089 50.00 134 75.00 169 16.67 −1
041 042 043 044 045 046 047 048 049 060 061 062 063 064 065 066 067 068 069 080 081 082 083 084 085
Topic Identifier
045 63.64 090 8.33 135 40.00 170 100.00 Ad−Hoc Robust Monolingual English track − Comparison to Median Mean Exact R−Precision by Topic (Topics 086 to 130)
1
046 55.56 091 42.86 136 0.00 171 5.26 dcunarreng1005
047 65.85 092 66.67 137 0.00 172 14.29 0.5
Difference
048 36.36 093 0.00 138 50.00 173 100.00 0
049 28.57 094 75.00 139 0.00 174 100.00 −0.5
060 0.00 095 47.92 140 60.00 175 85.71 −1
086 087 088 089 090 091 092 093 094 095 096 097 098 099 110 111 112 113 114 115 116 117 118 119 130
061 60.00 096 0.00 141 100.00 176 82.05 Topic Identifier
Ad−Hoc Robust Monolingual English track − Comparison to Median Mean Exact R−Precision by Topic (Topics 131 to 165)
062 100.00 097 100.00 142 87.50 177 46.15 1
dcunarreng1005
063 0.00 098 0.00 143 53.85 178 25.00 0.5
Difference
064 90.00 099 57.14 144 66.67 179 77.27
0
065 60.00 110 0.00 145 73.68 190 33.33
−0.5
066 72.73 111 45.00 146 0.00 191 0.00
−1
067 0.00 112 50.00 147 68.63 192 66.67 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 160 161 162 163 164 165
Topic Identifier
068 28.57 113 25.00 148 16.67 193 28.57 Ad−Hoc Robust Monolingual English track − Comparison to Median Mean Exact R−Precision by Topic (Topics 166 to 200)
1
069 62.50 114 71.43 149 0.00 194 100.00 dcunarreng1005
0.5
080 42.86 115 24.00 160 0.00 195 0.00
Difference
081 87.50 116 45.45 161 0.00 196 0.00 0
082 70.00 117 0.00 162 100.00 197 72.00 −0.5
083 20.00 118 0.00 163 77.05 198 100.00 −1
166 167 168 169 170 171 172 173 174 175 176 177 178 179 190 191 192 193 194 195 196 197 198 199 200
084 66.67 119 69.64 164 0.00 199 70.00 Topic Identifier
085 46.88 130 60.32 165 100.00 200 66.67
130
dcu dcudesceng12075 AH-ROBUST-MONO-EN-CLEF2006
Overall statistics for 100 queries : Priority 2
Total number of documents over all queries Query Construction AUTOMATIC
Retrieved 100,000 Source Language English
Relevant 1,533 Topic Fields title, description
Relevant retrieved 1,493 Pooled false
Geometric Mean Average Precision 0.1011 english mono standard desc
Binary Preference (BPREF) 0.4620
Interploated Recall (%) Precision Averages (%)
Ad−Hoc Robust Monolingual English track − Interpolated Recall vs Average Precision
100%
0 68.43
dcudesceng12075
10 65.80 90%
20 58.90
80%
30 52.60
40 49.72 70%
50 46.54
Average Precision
60%
60 41.36
70 35.70 50%
80 28.70 40%
90 23.95
30%
100 19.35
Average precision (non-interpolated) for all 20%
relevant documents (averaged over queries)
43.48 10%
0%
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
Interpolated Recall
Mean Average Precision Ad−Hoc Robust Monolingual English track − Box plot of the Topics of the Experiment
Maximum 1.0000
Minimum 0.0000
First Quartile 0.1636
Second Quartile 0.3702
Third Quartile 0.6578
Interquartile range 0.4942
Mean 0.4348
Standard Deviation 0.3253
Lower Outlier Threshold 0.0000
Upper Outlier Threshold 1.0000 0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Mean Average Precision
Mean With No Outliers 0.4348
Std With No Outliers 0.3253
Ad−Hoc Robust Monolingual English track − Distribution of the Topics of the Experiment
20
Number of Topics of the Experiment
15
10
5
0
0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Mean Average Precision
Precision averages (%) for individual Ad−Hoc Robust Monolingual English track − Comparison to Median Mean Average Precision by Topic (Topics 041 to 085)
queries 1
dcudesceng12075
0.5
041 27.49 086 51.79 131 62.63 166 0.00
Difference
0
042 35.06 087 61.37 132 0.00 167 34.90
−0.5
043 64.93 088 25.00 133 55.52 168 71.85
044 5.77 089 27.87 134 75.22 169 1.16 −1
041 042 043 044 045 046 047 048 049 060 061 062 063 064 065 066 067 068 069 080 081 082 083 084 085
Topic Identifier
045 66.22 090 1.48 135 33.36 170 100.00 Ad−Hoc Robust Monolingual English track − Comparison to Median Mean Average Precision by Topic (Topics 086 to 130)
1
046 33.29 091 20.44 136 25.00 171 6.45 dcudesceng12075
047 66.02 092 60.35 137 6.25 172 14.69 0.5
Difference
048 20.08 093 0.00 138 31.59 173 100.00 0
049 31.54 094 78.81 139 3.97 174 100.00 −0.5
060 0.00 095 43.57 140 57.46 175 90.71 −1
086 087 088 089 090 091 092 093 094 095 096 097 098 099 110 111 112 113 114 115 116 117 118 119 130
061 63.07 096 0.00 141 100.00 176 93.15 Topic Identifier
Ad−Hoc Robust Monolingual English track − Comparison to Median Mean Average Precision by Topic (Topics 131 to 165)
062 100.00 097 100.00 142 93.63 177 53.64 1
dcudesceng12075
063 1.75 098 33.33 143 79.13 178 9.33 0.5
Difference
064 23.34 099 29.58 144 58.78 179 79.27
0
065 30.90 110 0.00 145 77.75 190 26.33
−0.5
066 65.54 111 45.66 146 50.78 191 0.00
−1
067 25.25 112 57.53 147 79.32 192 2.83 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 160 161 162 163 164 165
Topic Identifier
068 37.88 113 4.36 148 19.80 193 51.01 Ad−Hoc Robust Monolingual English track − Comparison to Median Mean Average Precision by Topic (Topics 166 to 200)
1
069 27.06 114 53.06 149 0.00 194 33.33 dcudesceng12075
0.5
080 32.56 115 17.35 160 6.35 195 0.00
Difference
081 93.08 116 42.25 161 0.00 196 50.00 0
082 62.41 117 0.00 162 100.00 197 76.09 −0.5
083 16.03 118 0.00 163 89.73 198 100.00 −1
166 167 168 169 170 171 172 173 174 175 176 177 178 179 190 191 192 193 194 195 196 197 198 199 200
084 64.31 119 80.48 164 16.69 199 54.19 Topic Identifier
085 56.64 130 55.10 165 100.00 200 36.15
131
dcu dcudesceng12075 AH-ROBUST-MONO-EN-CLEF2006
Docs Cutoff Levels Precision at DCL (%)
Ad−Hoc Robust Monolingual English track − Retrieved documents vs Precision
100%
5 docs 43.40
dcudesceng12075
10 docs 34.80 90%
15 docs 30.47
80%
20 docs 27.45
30 docs 22.97 70%
100 docs 11.47
60%
200 docs 6.66
R−Precision
500 docs 2.92 50%
1000 docs 1.49 40%
R-Precision (precision after R document retrieved,
where R = Relevant retrieved) 30%
40.51
20%
10%
0%
5 10 15 20 30 100 200 500 1000
Retrieved Documents (logarithmic scale)
Exact R-Precision Ad−Hoc Robust Monolingual English track − Box plot of the Topics of the Experiment
Maximum 1.0000
Minimum 0.0000
First Quartile 0.0000
Second Quartile 0.4143
Third Quartile 0.6515
Interquartile range 0.6515
Mean 0.4051
Standard Deviation 0.3313
Lower Outlier Threshold 0.0000
Upper Outlier Threshold 1.0000 0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Exact R−Precision
Mean With No Outliers 0.4051
Std With No Outliers 0.3313
Ad−Hoc Robust Monolingual English track − Distribution of the Topics of the Experiment
30
Number of Topics of the Experiment
25
20
15
10
5
0
0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Exact R−Precision
Precision averages (%) for individual Ad−Hoc Robust Monolingual English track − Comparison to Median Mean Exact R−Precision by Topic (Topics 041 to 085)
queries 1
dcudesceng12075
0.5
041 25.00 086 50.00 131 62.50 166 0.00
Difference
0
042 33.33 087 50.00 132 0.00 167 40.00
−0.5
043 63.64 088 0.00 133 50.00 168 66.67
044 0.00 089 25.00 134 75.00 169 0.00 −1
041 042 043 044 045 046 047 048 049 060 061 062 063 064 065 066 067 068 069 080 081 082 083 084 085
Topic Identifier
045 60.61 090 0.00 135 40.00 170 100.00 Ad−Hoc Robust Monolingual English track − Comparison to Median Mean Exact R−Precision by Topic (Topics 086 to 130)
1
046 33.33 091 14.29 136 0.00 171 5.26 dcudesceng12075
047 63.41 092 51.85 137 0.00 172 28.57 0.5
Difference
048 15.15 093 0.00 138 37.50 173 100.00 0
049 28.57 094 75.00 139 0.00 174 100.00 −0.5
060 0.00 095 47.92 140 58.57 175 85.71 −1
086 087 088 089 090 091 092 093 094 095 096 097 098 099 110 111 112 113 114 115 116 117 118 119 130
061 60.00 096 0.00 141 100.00 176 79.49 Topic Identifier
Ad−Hoc Robust Monolingual English track − Comparison to Median Mean Exact R−Precision by Topic (Topics 131 to 165)
062 100.00 097 100.00 142 87.50 177 53.85 1
dcudesceng12075
063 0.00 098 0.00 143 74.36 178 0.00 0.5
Difference
064 20.00 099 42.86 144 66.67 179 72.73
0
065 33.33 110 0.00 145 68.42 190 22.22
−0.5
066 72.73 111 45.00 146 50.00 191 0.00
−1
067 50.00 112 60.00 147 74.51 192 0.00 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 160 161 162 163 164 165
Topic Identifier
068 42.86 113 0.00 148 16.67 193 42.86 Ad−Hoc Robust Monolingual English track − Comparison to Median Mean Exact R−Precision by Topic (Topics 166 to 200)
1
069 25.00 114 57.14 149 0.00 194 0.00 dcudesceng12075
0.5
080 39.29 115 20.00 160 0.00 195 0.00
Difference
081 87.50 116 36.36 161 0.00 196 0.00 0
082 60.00 117 0.00 162 100.00 197 68.00 −0.5
083 20.00 118 0.00 163 81.97 198 100.00 −1
166 167 168 169 170 171 172 173 174 175 176 177 178 179 190 191 192 193 194 195 196 197 198 199 200
084 60.00 119 73.21 164 11.11 199 40.00 Topic Identifier
085 46.88 130 57.14 165 100.00 200 44.44
132
hildesheim uhienmo2 AH-ROBUST-MONO-EN-CLEF2006
Overall statistics for 100 queries : Priority 1
Total number of documents over all queries Query Construction AUTOMATIC
Retrieved 100,000 Source Language English
Relevant 1,533 Topic Fields title, description
Relevant retrieved 212 Pooled false
Geometric Mean Average Precision 0.0001 BRF 15 30, Lucene stemmer
Binary Preference (BPREF) 0.0718
Interploated Recall (%) Precision Averages (%)
Ad−Hoc Robust Monolingual English track − Interpolated Recall vs Average Precision
100%
0 11.23
uhienmo2
10 10.63 90%
20 8.94
80%
30 8.84
40 7.81 70%
50 7.45
Average Precision
60%
60 7.22
70 5.92 50%
80 5.52 40%
90 4.42
30%
100 3.20
Average precision (non-interpolated) for all 20%
relevant documents (averaged over queries)
7.12 10%
0%
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
Interpolated Recall
Mean Average Precision Ad−Hoc Robust Monolingual English track − Box plot of the Topics of the Experiment
Maximum 1.0000
Minimum 0.0000
First Quartile 0.0000
Second Quartile 0.0000
Third Quartile 0.0019
Interquartile range 0.0019
Mean 0.0712
Standard Deviation 0.2014
Lower Outlier Threshold 0.0000
Upper Outlier Threshold 0.0035 0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Mean Average Precision
Mean With No Outliers 0.0001
Std With No Outliers 0.0006
Ad−Hoc Robust Monolingual English track − Distribution of the Topics of the Experiment
100
Number of Topics of the Experiment
80
60
40
20
0
0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Mean Average Precision
Precision averages (%) for individual Ad−Hoc Robust Monolingual English track − Comparison to Median Mean Average Precision by Topic (Topics 041 to 085)
queries 1
uhienmo2
0.5
041 0.00 086 0.00 131 0.00 166 0.00
Difference
0
042 0.00 087 0.00 132 0.00 167 0.00
−0.5
043 0.00 088 0.00 133 0.00 168 47.90
044 0.00 089 0.00 134 0.00 169 0.00 −1
041 042 043 044 045 046 047 048 049 060 061 062 063 064 065 066 067 068 069 080 081 082 083 084 085
Topic Identifier
045 0.00 090 0.00 135 0.00 170 12.50 Ad−Hoc Robust Monolingual English track − Comparison to Median Mean Average Precision by Topic (Topics 086 to 130)
1
046 0.00 091 0.00 136 0.00 171 0.00 uhienmo2
047 0.00 092 0.00 137 0.00 172 2.11 0.5
Difference
048 0.00 093 0.00 138 0.00 173 0.00 0
049 0.00 094 0.00 139 0.00 174 100.00 −0.5
060 0.00 095 0.00 140 0.00 175 0.00 −1
086 087 088 089 090 091 092 093 094 095 096 097 098 099 110 111 112 113 114 115 116 117 118 119 130
061 0.00 096 0.00 141 0.32 176 0.32 Topic Identifier
Ad−Hoc Robust Monolingual English track − Comparison to Median Mean Average Precision by Topic (Topics 131 to 165)
062 0.00 097 0.00 142 62.50 177 58.47 1
uhienmo2
063 0.00 098 0.00 143 1.55 178 0.00 0.5
Difference
064 0.00 099 0.00 144 5.56 179 87.52
0
065 0.00 110 0.00 145 0.00 190 0.06
−0.5
066 0.00 111 0.00 146 0.35 191 0.00
−1
067 0.00 112 0.00 147 11.76 192 1.86 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 160 161 162 163 164 165
Topic Identifier
068 0.00 113 0.00 148 1.66 193 27.13 Ad−Hoc Robust Monolingual English track − Comparison to Median Mean Average Precision by Topic (Topics 166 to 200)
1
069 0.00 114 0.00 149 0.00 194 0.00 uhienmo2
0.5
080 0.00 115 0.00 160 6.94 195 0.00
Difference
081 0.00 116 0.00 161 0.00 196 50.00 0
082 0.00 117 0.00 162 33.33 197 87.86 −0.5
083 0.00 118 0.00 163 0.00 198 0.96 −1
166 167 168 169 170 171 172 173 174 175 176 177 178 179 190 191 192 193 194 195 196 197 198 199 200
084 0.00 119 0.00 164 21.52 199 67.99 Topic Identifier
085 0.00 130 0.00 165 20.00 200 1.98
133
hildesheim uhienmo2 AH-ROBUST-MONO-EN-CLEF2006
Docs Cutoff Levels Precision at DCL (%)
Ad−Hoc Robust Monolingual English track − Retrieved documents vs Precision
100%
5 docs 7.20
uhienmo2
10 docs 6.10 90%
15 docs 5.13
80%
20 docs 4.70
30 docs 3.80 70%
100 docs 1.58
60%
200 docs 0.91
R−Precision
500 docs 0.40 50%
1000 docs 0.21 40%
R-Precision (precision after R document retrieved,
where R = Relevant retrieved) 30%
5.78
20%
10%
0%
5 10 15 20 30 100 200 500 1000
Retrieved Documents (logarithmic scale)
Exact R-Precision Ad−Hoc Robust Monolingual English track − Box plot of the Topics of the Experiment
Maximum 1.0000
Minimum 0.0000
First Quartile 0.0000
Second Quartile 0.0000
Third Quartile 0.0000
Interquartile range 0.0000
Mean 0.0578
Standard Deviation 0.1923
Lower Outlier Threshold 0.0000
Upper Outlier Threshold 0.0000 0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Exact R−Precision
Mean With No Outliers 0.0000
Std With No Outliers 0.0000
Ad−Hoc Robust Monolingual English track − Distribution of the Topics of the Experiment
100
Number of Topics of the Experiment
80
60
40
20
0
0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Exact R−Precision
Precision averages (%) for individual Ad−Hoc Robust Monolingual English track − Comparison to Median Mean Exact R−Precision by Topic (Topics 041 to 085)
queries 1
uhienmo2
0.5
041 0.00 086 0.00 131 0.00 166 0.00
Difference
0
042 0.00 087 0.00 132 0.00 167 0.00
−0.5
043 0.00 088 0.00 133 0.00 168 44.44
044 0.00 089 0.00 134 0.00 169 0.00 −1
041 042 043 044 045 046 047 048 049 060 061 062 063 064 065 066 067 068 069 080 081 082 083 084 085
Topic Identifier
045 0.00 090 0.00 135 0.00 170 0.00 Ad−Hoc Robust Monolingual English track − Comparison to Median Mean Exact R−Precision by Topic (Topics 086 to 130)
1
046 0.00 091 0.00 136 0.00 171 0.00 uhienmo2
047 0.00 092 0.00 137 0.00 172 0.00 0.5
Difference
048 0.00 093 0.00 138 0.00 173 0.00 0
049 0.00 094 0.00 139 0.00 174 100.00 −0.5
060 0.00 095 0.00 140 0.00 175 0.00 −1
086 087 088 089 090 091 092 093 094 095 096 097 098 099 110 111 112 113 114 115 116 117 118 119 130
061 0.00 096 0.00 141 0.00 176 2.56 Topic Identifier
Ad−Hoc Robust Monolingual English track − Comparison to Median Mean Exact R−Precision by Topic (Topics 131 to 165)
062 0.00 097 0.00 142 62.50 177 53.85 1
uhienmo2
063 0.00 098 0.00 143 2.56 178 0.00 0.5
Difference
064 0.00 099 0.00 144 0.00 179 86.36
0
065 0.00 110 0.00 145 0.00 190 0.00
−0.5
066 0.00 111 0.00 146 0.00 191 0.00
−1
067 0.00 112 0.00 147 11.76 192 0.00 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 160 161 162 163 164 165
Topic Identifier
068 0.00 113 0.00 148 0.00 193 28.57 Ad−Hoc Robust Monolingual English track − Comparison to Median Mean Exact R−Precision by Topic (Topics 166 to 200)
1
069 0.00 114 0.00 149 0.00 194 0.00 uhienmo2
0.5
080 0.00 115 0.00 160 0.00 195 0.00
Difference
081 0.00 116 0.00 161 0.00 196 0.00 0
082 0.00 117 0.00 162 0.00 197 82.00 −0.5
083 0.00 118 0.00 163 0.00 198 0.00 −1
166 167 168 169 170 171 172 173 174 175 176 177 178 179 190 191 192 193 194 195 196 197 198 199 200
084 0.00 119 0.00 164 33.33 199 70.00 Topic Identifier
085 0.00 130 0.00 165 0.00 200 0.00
134
hildesheim uhienmo1 AH-ROBUST-MONO-EN-CLEF2006
Overall statistics for 100 queries : Priority 2
Total number of documents over all queries Query Construction AUTOMATIC
Retrieved 99,695 Source Language English
Relevant 1,533 Topic Fields title, description
Relevant retrieved 218 Pooled false
Geometric Mean Average Precision 0.0001 Lucene stemmer, BRF 5, 30
Binary Preference (BPREF) 0.0744
Interploated Recall (%) Precision Averages (%)
Ad−Hoc Robust Monolingual English track − Interpolated Recall vs Average Precision
100%
0 12.13
uhienmo1
10 11.43 90%
20 10.39
80%
30 10.24
40 8.85 70%
50 7.98
Average Precision
60%
60 7.68
70 6.42 50%
80 5.99 40%
90 5.31
30%
100 4.03
Average precision (non-interpolated) for all 20%
relevant documents (averaged over queries)
7.98 10%
0%
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
Interpolated Recall
Mean Average Precision Ad−Hoc Robust Monolingual English track − Box plot of the Topics of the Experiment
Maximum 1.0000
Minimum 0.0000
First Quartile 0.0000
Second Quartile 0.0000
Third Quartile 0.0034
Interquartile range 0.0034
Mean 0.0798
Standard Deviation 0.2186
Lower Outlier Threshold 0.0000
Upper Outlier Threshold 0.0082 0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Mean Average Precision
Mean With No Outliers 0.0003
Std With No Outliers 0.0013
Ad−Hoc Robust Monolingual English track − Distribution of the Topics of the Experiment
100
Number of Topics of the Experiment
80
60
40
20
0
0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Mean Average Precision
Precision averages (%) for individual Ad−Hoc Robust Monolingual English track − Comparison to Median Mean Average Precision by Topic (Topics 041 to 085)
queries 1
uhienmo1
0.5
041 0.00 086 0.00 131 0.00 166 0.00
Difference
0
042 0.00 087 0.00 132 0.00 167 0.00
−0.5
043 0.00 088 0.00 133 0.00 168 53.68
044 0.00 089 0.00 134 0.00 169 0.00 −1
041 042 043 044 045 046 047 048 049 060 061 062 063 064 065 066 067 068 069 080 081 082 083 084 085
Topic Identifier
045 0.00 090 0.00 135 0.00 170 16.67 Ad−Hoc Robust Monolingual English track − Comparison to Median Mean Average Precision by Topic (Topics 086 to 130)
1
046 0.00 091 0.00 136 0.00 171 0.00 uhienmo1
047 0.00 092 0.00 137 0.00 172 2.56 0.5
Difference
048 0.00 093 0.00 138 0.00 173 0.00 0
049 0.00 094 0.00 139 0.00 174 100.00 −0.5
060 0.00 095 0.00 140 0.00 175 0.00 −1
086 087 088 089 090 091 092 093 094 095 096 097 098 099 110 111 112 113 114 115 116 117 118 119 130
061 0.00 096 0.00 141 0.60 176 0.64 Topic Identifier
Ad−Hoc Robust Monolingual English track − Comparison to Median Mean Average Precision by Topic (Topics 131 to 165)
062 0.00 097 0.00 142 62.50 177 61.69 1
uhienmo1
063 0.00 098 0.00 143 2.86 178 0.00 0.5
Difference
064 0.00 099 0.00 144 11.11 179 77.21
0
065 0.00 110 0.00 145 0.00 190 0.08
−0.5
066 0.00 111 0.00 146 0.82 191 0.00
−1
067 0.00 112 0.00 147 9.89 192 16.84 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 160 161 162 163 164 165
Topic Identifier
068 0.00 113 0.00 148 2.26 193 36.32 Ad−Hoc Robust Monolingual English track − Comparison to Median Mean Average Precision by Topic (Topics 166 to 200)
1
069 0.00 114 0.00 149 0.00 194 0.00 uhienmo1
0.5
080 0.00 115 0.00 160 5.42 195 0.00
Difference
081 0.00 116 0.00 161 0.00 196 100.00 0
082 0.00 117 0.00 162 14.29 197 89.95 −0.5
083 0.00 118 0.00 163 0.00 198 5.56 −1
166 167 168 169 170 171 172 173 174 175 176 177 178 179 190 191 192 193 194 195 196 197 198 199 200
084 0.00 119 0.00 164 12.02 199 63.31 Topic Identifier
085 0.00 130 0.00 165 50.00 200 1.62
135
hildesheim uhienmo1 AH-ROBUST-MONO-EN-CLEF2006
Docs Cutoff Levels Precision at DCL (%)
Ad−Hoc Robust Monolingual English track − Retrieved documents vs Precision
100%
5 docs 8.00
uhienmo1
10 docs 6.00 90%
15 docs 5.27
80%
20 docs 4.65
30 docs 3.97 70%
100 docs 1.65
60%
200 docs 0.93
R−Precision
500 docs 0.41 50%
1000 docs 0.22 40%
R-Precision (precision after R document retrieved,
where R = Relevant retrieved) 30%
7.29
20%
10%
0%
5 10 15 20 30 100 200 500 1000
Retrieved Documents (logarithmic scale)
Exact R-Precision Ad−Hoc Robust Monolingual English track − Box plot of the Topics of the Experiment
Maximum 1.0000
Minimum 0.0000
First Quartile 0.0000
Second Quartile 0.0000
Third Quartile 0.0000
Interquartile range 0.0000
Mean 0.0729
Standard Deviation 0.2127
Lower Outlier Threshold 0.0000
Upper Outlier Threshold 0.0000 0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Exact R−Precision
Mean With No Outliers 0.0000
Std With No Outliers 0.0000
Ad−Hoc Robust Monolingual English track − Distribution of the Topics of the Experiment
100
Number of Topics of the Experiment
80
60
40
20
0
0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Exact R−Precision
Precision averages (%) for individual Ad−Hoc Robust Monolingual English track − Comparison to Median Mean Exact R−Precision by Topic (Topics 041 to 085)
queries 1
uhienmo1
0.5
041 0.00 086 0.00 131 0.00 166 0.00
Difference
0
042 0.00 087 0.00 132 0.00 167 0.00
−0.5
043 0.00 088 0.00 133 0.00 168 61.11
044 0.00 089 0.00 134 0.00 169 0.00 −1
041 042 043 044 045 046 047 048 049 060 061 062 063 064 065 066 067 068 069 080 081 082 083 084 085
Topic Identifier
045 0.00 090 0.00 135 0.00 170 0.00 Ad−Hoc Robust Monolingual English track − Comparison to Median Mean Exact R−Precision by Topic (Topics 086 to 130)
1
046 0.00 091 0.00 136 0.00 171 0.00 uhienmo1
047 0.00 092 0.00 137 0.00 172 0.00 0.5
Difference
048 0.00 093 0.00 138 0.00 173 0.00 0
049 0.00 094 0.00 139 0.00 174 100.00 −0.5
060 0.00 095 0.00 140 0.00 175 0.00 −1
086 087 088 089 090 091 092 093 094 095 096 097 098 099 110 111 112 113 114 115 116 117 118 119 130
061 0.00 096 0.00 141 0.00 176 2.56 Topic Identifier
Ad−Hoc Robust Monolingual English track − Comparison to Median Mean Exact R−Precision by Topic (Topics 131 to 165)
062 0.00 097 0.00 142 62.50 177 53.85 1
uhienmo1
063 0.00 098 0.00 143 7.69 178 0.00 0.5
Difference
064 0.00 099 0.00 144 33.33 179 72.73
0
065 0.00 110 0.00 145 0.00 190 0.00
−0.5
066 0.00 111 0.00 146 0.00 191 0.00
−1
067 0.00 112 0.00 147 11.76 192 33.33 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 160 161 162 163 164 165
Topic Identifier
068 0.00 113 0.00 148 0.00 193 42.86 Ad−Hoc Robust Monolingual English track − Comparison to Median Mean Exact R−Precision by Topic (Topics 166 to 200)
1
069 0.00 114 0.00 149 0.00 194 0.00 uhienmo1
0.5
080 0.00 115 0.00 160 0.00 195 0.00
Difference
081 0.00 116 0.00 161 0.00 196 100.00 0
082 0.00 117 0.00 162 0.00 197 86.00 −0.5
083 0.00 118 0.00 163 0.00 198 0.00 −1
166 167 168 169 170 171 172 173 174 175 176 177 178 179 190 191 192 193 194 195 196 197 198 199 200
084 0.00 119 0.00 164 11.11 199 50.00 Topic Identifier
085 0.00 130 0.00 165 0.00 200 0.00
136
hummingbird humEN06Rtd AH-ROBUST-MONO-EN-CLEF2006
Overall statistics for 100 queries : Priority 1
Total number of documents over all queries Query Construction AUTOMATIC
Retrieved 100,000 Source Language English
Relevant 1,533 Topic Fields title, description
Relevant retrieved 1,475 Pooled false
Geometric Mean Average Precision 0.1055 plain SearchServer stemming run
Binary Preference (BPREF) 0.4557
Interploated Recall (%) Precision Averages (%)
Ad−Hoc Robust Monolingual English track − Interpolated Recall vs Average Precision
100%
0 67.59
humEN06Rtd
10 65.02 90%
20 59.99
80%
30 53.97
40 49.40 70%
50 46.64
Average Precision
60%
60 39.90
70 34.11 50%
80 28.40 40%
90 22.96
30%
100 17.92
Average precision (non-interpolated) for all 20%
relevant documents (averaged over queries)
42.80 10%
0%
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
Interpolated Recall
Mean Average Precision Ad−Hoc Robust Monolingual English track − Box plot of the Topics of the Experiment
Maximum 1.0000
Minimum 0.0000
First Quartile 0.1669
Second Quartile 0.3777
Third Quartile 0.6431
Interquartile range 0.4762
Mean 0.4280
Standard Deviation 0.3169
Lower Outlier Threshold 0.0000
Upper Outlier Threshold 1.0000 0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Mean Average Precision
Mean With No Outliers 0.4280
Std With No Outliers 0.3169
Ad−Hoc Robust Monolingual English track − Distribution of the Topics of the Experiment
15
Number of Topics of the Experiment
10
5
0
0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Mean Average Precision
Precision averages (%) for individual Ad−Hoc Robust Monolingual English track − Comparison to Median Mean Average Precision by Topic (Topics 041 to 085)
queries 1
humEN06Rtd
0.5
041 16.02 086 41.74 131 28.08 166 0.00
Difference
0
042 34.54 087 56.52 132 0.00 167 31.35
−0.5
043 81.96 088 20.00 133 22.61 168 34.65
044 52.70 089 79.17 134 75.11 169 5.88 −1
041 042 043 044 045 046 047 048 049 060 061 062 063 064 065 066 067 068 069 080 081 082 083 084 085
Topic Identifier
045 62.99 090 27.37 135 17.20 170 100.00 Ad−Hoc Robust Monolingual English track − Comparison to Median Mean Average Precision by Topic (Topics 086 to 130)
1
046 35.04 091 16.72 136 33.33 171 11.61 humEN06Rtd
047 57.01 092 78.07 137 50.00 172 8.30 0.5
Difference
048 5.07 093 0.00 138 27.54 173 100.00 0
049 19.20 094 73.65 139 4.31 174 100.00 −0.5
060 0.00 095 26.77 140 34.13 175 72.34 −1
086 087 088 089 090 091 092 093 094 095 096 097 098 099 110 111 112 113 114 115 116 117 118 119 130
061 56.14 096 0.00 141 100.00 176 94.24 Topic Identifier
Ad−Hoc Robust Monolingual English track − Comparison to Median Mean Average Precision by Topic (Topics 131 to 165)
062 100.00 097 100.00 142 100.00 177 28.68 1
humEN06Rtd
063 2.22 098 50.00 143 93.66 178 5.86 0.5
Difference
064 30.66 099 53.40 144 86.67 179 59.41
0
065 36.62 110 0.00 145 57.86 190 43.42
−0.5
066 71.67 111 37.05 146 51.35 191 0.00
−1
067 25.13 112 62.57 147 81.52 192 41.57 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 160 161 162 163 164 165
Topic Identifier
068 40.79 113 2.21 148 11.65 193 53.71 Ad−Hoc Robust Monolingual English track − Comparison to Median Mean Average Precision by Topic (Topics 166 to 200)
1
069 25.20 114 44.72 149 0.00 194 16.67 humEN06Rtd
0.5
080 39.38 115 13.52 160 14.00 195 0.00
Difference
081 78.77 116 36.20 161 0.00 196 50.00 0
082 63.43 117 0.00 162 33.33 197 92.88 −0.5
083 28.45 118 0.00 163 83.18 198 100.00 −1
166 167 168 169 170 171 172 173 174 175 176 177 178 179 190 191 192 193 194 195 196 197 198 199 200
084 65.19 119 76.85 164 22.55 199 60.00 Topic Identifier
085 38.50 130 39.22 165 100.00 200 38.54
137
hummingbird humEN06Rtd AH-ROBUST-MONO-EN-CLEF2006
Docs Cutoff Levels Precision at DCL (%)
Ad−Hoc Robust Monolingual English track − Retrieved documents vs Precision
100%
5 docs 42.40
humEN06Rtd
10 docs 33.60 90%
15 docs 29.33
80%
20 docs 26.15
30 docs 22.10 70%
100 docs 10.73
60%
200 docs 6.38
R−Precision
500 docs 2.81 50%
1000 docs 1.47 40%
R-Precision (precision after R document retrieved,
where R = Relevant retrieved) 30%
39.78
20%
10%
0%
5 10 15 20 30 100 200 500 1000
Retrieved Documents (logarithmic scale)
Exact R-Precision Ad−Hoc Robust Monolingual English track − Box plot of the Topics of the Experiment
Maximum 1.0000
Minimum 0.0000
First Quartile 0.0152
Second Quartile 0.3909
Third Quartile 0.6182
Interquartile range 0.6030
Mean 0.3978
Standard Deviation 0.3206
Lower Outlier Threshold 0.0000
Upper Outlier Threshold 1.0000 0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Exact R−Precision
Mean With No Outliers 0.3978
Std With No Outliers 0.3206
Ad−Hoc Robust Monolingual English track − Distribution of the Topics of the Experiment
30
Number of Topics of the Experiment
25
20
15
10
5
0
0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Exact R−Precision
Precision averages (%) for individual Ad−Hoc Robust Monolingual English track − Comparison to Median Mean Exact R−Precision by Topic (Topics 041 to 085)
queries 1
humEN06Rtd
0.5
041 12.50 086 43.90 131 32.50 166 0.00
Difference
0
042 41.18 087 41.67 132 0.00 167 20.00
−0.5
043 72.73 088 0.00 133 30.00 168 38.89
044 50.00 089 75.00 134 75.00 169 0.00 −1
041 042 043 044 045 046 047 048 049 060 061 062 063 064 065 066 067 068 069 080 081 082 083 084 085
Topic Identifier
045 63.64 090 25.00 135 40.00 170 100.00 Ad−Hoc Robust Monolingual English track − Comparison to Median Mean Exact R−Precision by Topic (Topics 086 to 130)
1
046 44.44 091 28.57 136 0.00 171 10.53 humEN06Rtd
047 53.66 092 70.37 137 50.00 172 0.00 0.5
Difference
048 3.03 093 0.00 138 37.50 173 100.00 0
049 21.43 094 75.00 139 0.00 174 100.00 −0.5
060 0.00 095 33.33 140 42.86 175 71.43 −1
086 087 088 089 090 091 092 093 094 095 096 097 098 099 110 111 112 113 114 115 116 117 118 119 130
061 60.00 096 0.00 141 100.00 176 84.62 Topic Identifier
Ad−Hoc Robust Monolingual English track − Comparison to Median Mean Exact R−Precision by Topic (Topics 131 to 165)
062 100.00 097 100.00 142 100.00 177 38.46 1
humEN06Rtd
063 0.00 098 0.00 143 84.62 178 0.00 0.5
Difference
064 30.00 099 57.14 144 66.67 179 59.09
0
065 33.33 110 0.00 145 57.89 190 33.33
−0.5
066 72.73 111 40.00 146 50.00 191 0.00
−1
067 50.00 112 60.00 147 74.51 192 33.33 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 160 161 162 163 164 165
Topic Identifier
068 42.86 113 0.00 148 16.67 193 42.86 Ad−Hoc Robust Monolingual English track − Comparison to Median Mean Exact R−Precision by Topic (Topics 166 to 200)
1
069 25.00 114 42.86 149 0.00 194 0.00 humEN06Rtd
0.5
080 39.29 115 16.00 160 0.00 195 0.00
Difference
081 75.00 116 27.27 161 0.00 196 0.00 0
082 60.00 117 0.00 162 0.00 197 84.00 −0.5
083 20.00 118 0.00 163 73.77 198 100.00 −1
166 167 168 169 170 171 172 173 174 175 176 177 178 179 190 191 192 193 194 195 196 197 198 199 200
084 60.00 119 69.64 164 25.93 199 50.00 Topic Identifier
085 34.38 130 38.10 165 100.00 200 44.44
138
hummingbird humEN06Rtde AH-ROBUST-MONO-EN-CLEF2006
Overall statistics for 100 queries : Priority 2
Total number of documents over all queries Query Construction AUTOMATIC
Retrieved 100,000 Source Language English
Relevant 1,533 Topic Fields title, description
Relevant retrieved 1,483 Pooled false
Geometric Mean Average Precision 0.1169 same as humEN06Rtd except expansion from blind
Binary Preference (BPREF) 0.5164 feedback also used
Interploated Recall (%) Precision Averages (%)
Ad−Hoc Robust Monolingual English track − Interpolated Recall vs Average Precision
100%
0 69.18
humEN06Rtde
10 67.30 90%
20 64.57
80%
30 58.92
40 55.85 70%
50 52.88
Average Precision
60%
60 46.21
70 40.63 50%
80 34.90 40%
90 27.05
30%
100 20.03
Average precision (non-interpolated) for all 20%
relevant documents (averaged over queries)
47.63 10%
0%
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
Interpolated Recall
Mean Average Precision Ad−Hoc Robust Monolingual English track − Box plot of the Topics of the Experiment
Maximum 1.0000
Minimum 0.0000
First Quartile 0.1746
Second Quartile 0.5000
Third Quartile 0.7356
Interquartile range 0.5610
Mean 0.4763
Standard Deviation 0.3285
Lower Outlier Threshold 0.0000
Upper Outlier Threshold 1.0000 0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Mean Average Precision
Mean With No Outliers 0.4763
Std With No Outliers 0.3285
Ad−Hoc Robust Monolingual English track − Distribution of the Topics of the Experiment
20
Number of Topics of the Experiment
15
10
5
0
0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Mean Average Precision
Precision averages (%) for individual Ad−Hoc Robust Monolingual English track − Comparison to Median Mean Average Precision by Topic (Topics 041 to 085)
queries 1
humEN06Rtde
0.5
041 17.92 086 54.65 131 54.90 166 0.00
Difference
0
042 38.50 087 77.53 132 0.00 167 25.45
−0.5
043 64.22 088 20.00 133 24.87 168 37.10
044 52.50 089 100.00 134 75.41 169 5.99 −1
041 042 043 044 045 046 047 048 049 060 061 062 063 064 065 066 067 068 069 080 081 082 083 084 085
Topic Identifier
045 61.58 090 38.60 135 17.00 170 100.00 Ad−Hoc Robust Monolingual English track − Comparison to Median Mean Average Precision by Topic (Topics 086 to 130)
1
046 38.33 091 20.76 136 33.33 171 18.44 humEN06Rtde
047 68.81 092 74.38 137 50.00 172 11.08 0.5
Difference
048 5.35 093 0.00 138 26.33 173 100.00 0
049 27.95 094 78.15 139 4.89 174 100.00 −0.5
060 0.00 095 47.50 140 60.85 175 88.22 −1
086 087 088 089 090 091 092 093 094 095 096 097 098 099 110 111 112 113 114 115 116 117 118 119 130
061 72.00 096 0.00 141 100.00 176 93.30 Topic Identifier
Ad−Hoc Robust Monolingual English track − Comparison to Median Mean Average Precision by Topic (Topics 131 to 165)
062 100.00 097 100.00 142 97.50 177 39.17 1
humEN06Rtde
063 2.86 098 50.00 143 93.50 178 2.81 0.5
Difference
064 63.54 099 51.76 144 72.22 179 89.54
0
065 40.22 110 0.00 145 61.23 190 54.24
−0.5
066 72.74 111 47.67 146 50.95 191 0.00
−1
067 25.57 112 59.27 147 87.44 192 77.22 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 160 161 162 163 164 165
Topic Identifier
068 53.85 113 2.07 148 12.08 193 42.83 Ad−Hoc Robust Monolingual English track − Comparison to Median Mean Average Precision by Topic (Topics 166 to 200)
1
069 25.43 114 55.85 149 0.00 194 16.67 humEN06Rtde
0.5
080 47.09 115 15.74 160 16.67 195 0.00
Difference
081 82.07 116 39.51 161 0.00 196 100.00 0
082 68.01 117 0.00 162 50.00 197 77.54 −0.5
083 36.49 118 0.00 163 88.06 198 100.00 −1
166 167 168 169 170 171 172 173 174 175 176 177 178 179 190 191 192 193 194 195 196 197 198 199 200
084 67.88 119 83.84 164 14.57 199 64.75 Topic Identifier
085 59.48 130 52.93 165 100.00 200 63.90
139
hummingbird humEN06Rtde AH-ROBUST-MONO-EN-CLEF2006
Docs Cutoff Levels Precision at DCL (%)
Ad−Hoc Robust Monolingual English track − Retrieved documents vs Precision
100%
5 docs 45.60
humEN06Rtde
10 docs 38.30 90%
15 docs 32.33
80%
20 docs 28.90
30 docs 24.43 70%
100 docs 11.68
60%
200 docs 6.75
R−Precision
500 docs 2.92 50%
1000 docs 1.48 40%
R-Precision (precision after R document retrieved,
where R = Relevant retrieved) 30%
44.80
20%
10%
0%
5 10 15 20 30 100 200 500 1000
Retrieved Documents (logarithmic scale)
Exact R-Precision Ad−Hoc Robust Monolingual English track − Box plot of the Topics of the Experiment
Maximum 1.0000
Minimum 0.0000
First Quartile 0.0741
Second Quartile 0.5000
Third Quartile 0.6900
Interquartile range 0.6159
Mean 0.4480
Standard Deviation 0.3359
Lower Outlier Threshold 0.0000
Upper Outlier Threshold 1.0000 0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Exact R−Precision
Mean With No Outliers 0.4480
Std With No Outliers 0.3359
Ad−Hoc Robust Monolingual English track − Distribution of the Topics of the Experiment
25
Number of Topics of the Experiment
20
15
10
5
0
0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Exact R−Precision
Precision averages (%) for individual Ad−Hoc Robust Monolingual English track − Comparison to Median Mean Exact R−Precision by Topic (Topics 041 to 085)
queries 1
humEN06Rtde
0.5
041 25.00 086 51.22 131 55.00 166 0.00
Difference
0
042 45.10 087 66.67 132 0.00 167 20.00
−0.5
043 63.64 088 0.00 133 30.00 168 44.44
044 50.00 089 100.00 134 75.00 169 0.00 −1
041 042 043 044 045 046 047 048 049 060 061 062 063 064 065 066 067 068 069 080 081 082 083 084 085
Topic Identifier
045 54.55 090 33.33 135 20.00 170 100.00 Ad−Hoc Robust Monolingual English track − Comparison to Median Mean Exact R−Precision by Topic (Topics 086 to 130)
1
046 44.44 091 28.57 136 0.00 171 15.79 humEN06Rtde
047 60.98 092 66.67 137 50.00 172 0.00 0.5
Difference
048 0.00 093 0.00 138 37.50 173 100.00 0
049 28.57 094 66.67 139 0.00 174 100.00 −0.5
060 0.00 095 52.08 140 57.14 175 85.71 −1
086 087 088 089 090 091 092 093 094 095 096 097 098 099 110 111 112 113 114 115 116 117 118 119 130
061 80.00 096 0.00 141 100.00 176 79.49 Topic Identifier
Ad−Hoc Robust Monolingual English track − Comparison to Median Mean Exact R−Precision by Topic (Topics 131 to 165)
062 100.00 097 100.00 142 87.50 177 46.15 1
humEN06Rtde
063 0.00 098 0.00 143 87.18 178 0.00 0.5
Difference
064 70.00 099 57.14 144 66.67 179 77.27
0
065 33.33 110 0.00 145 60.53 190 55.56
−0.5
066 72.73 111 45.00 146 50.00 191 0.00
−1
067 50.00 112 60.00 147 78.43 192 83.33 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 160 161 162 163 164 165
Topic Identifier
068 71.43 113 0.00 148 16.67 193 28.57 Ad−Hoc Robust Monolingual English track − Comparison to Median Mean Exact R−Precision by Topic (Topics 166 to 200)
1
069 37.50 114 57.14 149 0.00 194 0.00 humEN06Rtde
0.5
080 48.21 115 20.00 160 0.00 195 0.00
Difference
081 87.50 116 36.36 161 0.00 196 100.00 0
082 60.00 117 0.00 162 0.00 197 68.00 −0.5
083 20.00 118 0.00 163 79.51 198 100.00 −1
166 167 168 169 170 171 172 173 174 175 176 177 178 179 190 191 192 193 194 195 196 197 198 199 200
084 60.00 119 75.00 164 14.81 199 60.00 Topic Identifier
085 50.00 130 53.97 165 100.00 200 66.67
140
reina reinaENttest AH-ROBUST-MONO-EN-CLEF2006
Overall statistics for 100 queries : Priority 2
Total number of documents over all queries Query Construction AUTOMATIC
Retrieved 100,000 Source Language English
Relevant 1,533 Topic Fields title
Relevant retrieved 1,432 Pooled false
Geometric Mean Average Precision 0.0782 local expansion with term windows, only title field
Binary Preference (BPREF) 0.4410
Interploated Recall (%) Precision Averages (%)
Ad−Hoc Robust Monolingual English track − Interpolated Recall vs Average Precision
100%
0 65.98
reinaENttest
10 62.04 90%
20 55.25
80%
30 50.93
40 46.68 70%
50 42.87
Average Precision
60%
60 36.30
70 31.63 50%
80 27.97 40%
90 23.02
30%
100 17.48
Average precision (non-interpolated) for all 20%
relevant documents (averaged over queries)
40.66 10%
0%
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
Interpolated Recall
Mean Average Precision Ad−Hoc Robust Monolingual English track − Box plot of the Topics of the Experiment
Maximum 1.0000
Minimum 0.0000
First Quartile 0.0971
Second Quartile 0.3525
Third Quartile 0.6850
Interquartile range 0.5879
Mean 0.4066
Standard Deviation 0.3412
Lower Outlier Threshold 0.0000
Upper Outlier Threshold 1.0000 0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Mean Average Precision
Mean With No Outliers 0.4066
Std With No Outliers 0.3412
Ad−Hoc Robust Monolingual English track − Distribution of the Topics of the Experiment
25
Number of Topics of the Experiment
20
15
10
5
0
0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Mean Average Precision
Precision averages (%) for individual Ad−Hoc Robust Monolingual English track − Comparison to Median Mean Average Precision by Topic (Topics 041 to 085)
queries 1
reinaENttest
0.5
041 19.36 086 38.27 131 46.61 166 0.00
Difference
0
042 32.22 087 59.51 132 0.00 167 31.66
−0.5
043 81.92 088 100.00 133 10.21 168 39.45
044 27.33 089 13.55 134 60.42 169 5.79 −1
041 042 043 044 045 046 047 048 049 060 061 062 063 064 065 066 067 068 069 080 081 082 083 084 085
Topic Identifier
045 69.83 090 62.36 135 0.36 170 100.00 Ad−Hoc Robust Monolingual English track − Comparison to Median Mean Average Precision by Topic (Topics 086 to 130)
1
046 35.24 091 0.17 136 33.33 171 11.14 reinaENttest
047 61.92 092 73.57 137 50.00 172 1.99 0.5
Difference
048 7.45 093 0.00 138 53.36 173 100.00 0
049 15.18 094 81.94 139 4.22 174 100.00 −0.5
060 0.00 095 23.00 140 12.02 175 57.64 −1
086 087 088 089 090 091 092 093 094 095 096 097 098 099 110 111 112 113 114 115 116 117 118 119 130
061 33.89 096 0.00 141 100.00 176 91.66 Topic Identifier
Ad−Hoc Robust Monolingual English track − Comparison to Median Mean Average Precision by Topic (Topics 131 to 165)
062 100.00 097 100.00 142 100.00 177 37.47 1
reinaENttest
063 2.56 098 0.00 143 88.67 178 2.10 0.5
Difference
064 35.25 099 11.12 144 64.65 179 69.73
0
065 37.51 110 0.00 145 63.84 190 58.34
−0.5
066 76.85 111 15.60 146 50.00 191 0.00
−1
067 11.15 112 25.06 147 46.98 192 13.63 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 160 161 162 163 164 165
Topic Identifier
068 80.16 113 3.51 148 19.62 193 22.70 Ad−Hoc Robust Monolingual English track − Comparison to Median Mean Average Precision by Topic (Topics 166 to 200)
1
069 41.90 114 40.07 149 0.00 194 7.14 reinaENttest
0.5
080 33.44 115 15.28 160 4.07 195 0.00
Difference
081 73.51 116 44.73 161 0.00 196 100.00 0
082 79.85 117 0.00 162 100.00 197 59.79 −0.5
083 22.82 118 0.00 163 76.15 198 100.00 −1
166 167 168 169 170 171 172 173 174 175 176 177 178 179 190 191 192 193 194 195 196 197 198 199 200
084 68.80 119 56.26 164 9.21 199 68.20 Topic Identifier
085 18.88 130 47.39 165 100.00 200 15.58
141
reina reinaENttest AH-ROBUST-MONO-EN-CLEF2006
Docs Cutoff Levels Precision at DCL (%)
Ad−Hoc Robust Monolingual English track − Retrieved documents vs Precision
100%
5 docs 38.60
reinaENttest
10 docs 32.50 90%
15 docs 27.73
80%
20 docs 24.70
30 docs 20.57 70%
100 docs 9.63
60%
200 docs 5.85
R−Precision
500 docs 2.75 50%
1000 docs 1.43 40%
R-Precision (precision after R document retrieved,
where R = Relevant retrieved) 30%
39.47
20%
10%
0%
5 10 15 20 30 100 200 500 1000
Retrieved Documents (logarithmic scale)
Exact R-Precision Ad−Hoc Robust Monolingual English track − Box plot of the Topics of the Experiment
Maximum 1.0000
Minimum 0.0000
First Quartile 0.0566
Second Quartile 0.3923
Third Quartile 0.6461
Interquartile range 0.5894
Mean 0.3947
Standard Deviation 0.3343
Lower Outlier Threshold 0.0000
Upper Outlier Threshold 1.0000 0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Exact R−Precision
Mean With No Outliers 0.3947
Std With No Outliers 0.3343
Ad−Hoc Robust Monolingual English track − Distribution of the Topics of the Experiment
25
Number of Topics of the Experiment
20
15
10
5
0
0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Exact R−Precision
Precision averages (%) for individual Ad−Hoc Robust Monolingual English track − Comparison to Median Mean Exact R−Precision by Topic (Topics 041 to 085)
queries 1
reinaENttest
0.5
041 12.50 086 43.90 131 47.50 166 0.00
Difference
0
042 33.33 087 50.00 132 0.00 167 40.00
−0.5
043 72.73 088 100.00 133 10.00 168 44.44
044 50.00 089 25.00 134 75.00 169 16.67 −1
041 042 043 044 045 046 047 048 049 060 061 062 063 064 065 066 067 068 069 080 081 082 083 084 085
Topic Identifier
045 66.67 090 58.33 135 0.00 170 100.00 Ad−Hoc Robust Monolingual English track − Comparison to Median Mean Exact R−Precision by Topic (Topics 086 to 130)
1
046 33.33 091 0.00 136 0.00 171 5.26 reinaENttest
047 56.10 092 66.67 137 50.00 172 0.00 0.5
Difference
048 6.06 093 0.00 138 50.00 173 100.00 0
049 14.29 094 66.67 139 0.00 174 100.00 −0.5
060 0.00 095 23.96 140 24.29 175 42.86 −1
086 087 088 089 090 091 092 093 094 095 096 097 098 099 110 111 112 113 114 115 116 117 118 119 130
061 20.00 096 0.00 141 100.00 176 87.18 Topic Identifier
Ad−Hoc Robust Monolingual English track − Comparison to Median Mean Exact R−Precision by Topic (Topics 131 to 165)
062 100.00 097 100.00 142 100.00 177 38.46 1
reinaENttest
063 0.00 098 0.00 143 79.49 178 0.00 0.5
Difference
064 30.00 099 28.57 144 66.67 179 63.64
0
065 40.00 110 0.00 145 57.89 190 44.44
−0.5
066 72.73 111 20.00 146 50.00 191 0.00
−1
067 0.00 112 30.00 147 49.02 192 16.67 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 160 161 162 163 164 165
Topic Identifier
068 71.43 113 0.00 148 16.67 193 14.29 Ad−Hoc Robust Monolingual English track − Comparison to Median Mean Exact R−Precision by Topic (Topics 166 to 200)
1
069 37.50 114 42.86 149 0.00 194 0.00 reinaENttest
0.5
080 41.07 115 16.00 160 0.00 195 0.00
Difference
081 62.50 116 45.45 161 0.00 196 100.00 0
082 80.00 117 0.00 162 100.00 197 54.00 −0.5
083 20.00 118 0.00 163 65.57 198 100.00 −1
166 167 168 169 170 171 172 173 174 175 176 177 178 179 190 191 192 193 194 195 196 197 198 199 200
084 60.00 119 55.36 164 11.11 199 70.00 Topic Identifier
085 18.75 130 50.79 165 100.00 200 33.33
142
reina reinaENtdtest AH-ROBUST-MONO-EN-CLEF2006
Overall statistics for 100 queries : Priority 1
Total number of documents over all queries Query Construction AUTOMATIC
Retrieved 100,000 Source Language English
Relevant 1,533 Topic Fields title, description
Relevant retrieved 1,466 Pooled false
Geometric Mean Average Precision 0.1053 local expansion with term windows, title+desc
Binary Preference (BPREF) 0.4531 fields
Interploated Recall (%) Precision Averages (%)
Ad−Hoc Robust Monolingual English track − Interpolated Recall vs Average Precision
100%
0 69.03
reinaENtdtest
10 65.88 90%
20 60.30
80%
30 53.05
40 49.25 70%
50 45.39
Average Precision
60%
60 39.85
70 35.65 50%
80 30.80 40%
90 25.39
30%
100 19.55
Average precision (non-interpolated) for all 20%
relevant documents (averaged over queries)
43.66 10%
0%
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
Interpolated Recall
Mean Average Precision Ad−Hoc Robust Monolingual English track − Box plot of the Topics of the Experiment
Maximum 1.0000
Minimum 0.0000
First Quartile 0.1572
Second Quartile 0.3909
Third Quartile 0.7118
Interquartile range 0.5546
Mean 0.4366
Standard Deviation 0.3276
Lower Outlier Threshold 0.0000
Upper Outlier Threshold 1.0000 0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Mean Average Precision
Mean With No Outliers 0.4366
Std With No Outliers 0.3276
Ad−Hoc Robust Monolingual English track − Distribution of the Topics of the Experiment
20
Number of Topics of the Experiment
15
10
5
0
0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Mean Average Precision
Precision averages (%) for individual Ad−Hoc Robust Monolingual English track − Comparison to Median Mean Average Precision by Topic (Topics 041 to 085)
queries 1
reinaENtdtest
0.5
041 25.35 086 33.54 131 42.88 166 0.00
Difference
0
042 37.02 087 50.83 132 0.00 167 23.81
−0.5
043 83.49 088 100.00 133 16.62 168 41.99
044 27.50 089 38.23 134 75.00 169 2.51 −1
041 042 043 044 045 046 047 048 049 060 061 062 063 064 065 066 067 068 069 080 081 082 083 084 085
Topic Identifier
045 58.97 090 30.86 135 35.09 170 100.00 Ad−Hoc Robust Monolingual English track − Comparison to Median Mean Average Precision by Topic (Topics 086 to 130)
1
046 38.18 091 10.25 136 50.00 171 12.64 reinaENtdtest
047 61.02 092 79.72 137 16.67 172 12.78 0.5
Difference
048 5.97 093 0.00 138 41.29 173 100.00 0
049 14.93 094 65.13 139 4.01 174 100.00 −0.5
060 0.00 095 32.32 140 40.40 175 71.27 −1
086 087 088 089 090 091 092 093 094 095 096 097 098 099 110 111 112 113 114 115 116 117 118 119 130
061 51.27 096 0.00 141 100.00 176 91.88 Topic Identifier
Ad−Hoc Robust Monolingual English track − Comparison to Median Mean Average Precision by Topic (Topics 131 to 165)
062 100.00 097 100.00 142 98.61 177 16.51 1
reinaENtdtest
063 1.18 098 100.00 143 81.23 178 8.75 0.5
Difference
064 64.59 099 8.99 144 70.00 179 64.07
0
065 39.45 110 0.00 145 66.22 190 72.53
−0.5
066 81.84 111 23.93 146 50.21 191 0.00
−1
067 16.80 112 54.15 147 78.04 192 32.60 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 160 161 162 163 164 165
Topic Identifier
068 40.47 113 4.98 148 20.62 193 39.55 Ad−Hoc Robust Monolingual English track − Comparison to Median Mean Average Precision by Topic (Topics 166 to 200)
1
069 33.77 114 57.74 149 0.00 194 25.00 reinaENtdtest
0.5
080 36.61 115 17.16 160 11.81 195 0.00
Difference
081 76.93 116 38.60 161 0.00 196 100.00 0
082 38.73 117 0.00 162 50.00 197 76.60 −0.5
083 28.83 118 0.00 163 86.45 198 100.00 −1
166 167 168 169 170 171 172 173 174 175 176 177 178 179 190 191 192 193 194 195 196 197 198 199 200
084 61.78 119 75.99 164 8.81 199 71.08 Topic Identifier
085 42.76 130 42.34 165 100.00 200 26.55
143
reina reinaENtdtest AH-ROBUST-MONO-EN-CLEF2006
Docs Cutoff Levels Precision at DCL (%)
Ad−Hoc Robust Monolingual English track − Retrieved documents vs Precision
100%
5 docs 41.40
reinaENtdtest
10 docs 34.00 90%
15 docs 29.73
80%
20 docs 26.60
30 docs 22.23 70%
100 docs 10.58
60%
200 docs 6.28
R−Precision
500 docs 2.82 50%
1000 docs 1.47 40%
R-Precision (precision after R document retrieved,
where R = Relevant retrieved) 30%
40.74
20%
10%
0%
5 10 15 20 30 100 200 500 1000
Retrieved Documents (logarithmic scale)
Exact R-Precision Ad−Hoc Robust Monolingual English track − Box plot of the Topics of the Experiment
Maximum 1.0000
Minimum 0.0000
First Quartile 0.0897
Second Quartile 0.4000
Third Quartile 0.6726
Interquartile range 0.5830
Mean 0.4074
Standard Deviation 0.3278
Lower Outlier Threshold 0.0000
Upper Outlier Threshold 1.0000 0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Exact R−Precision
Mean With No Outliers 0.4074
Std With No Outliers 0.3278
Ad−Hoc Robust Monolingual English track − Distribution of the Topics of the Experiment
25
Number of Topics of the Experiment
20
15
10
5
0
0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Exact R−Precision
Precision averages (%) for individual Ad−Hoc Robust Monolingual English track − Comparison to Median Mean Exact R−Precision by Topic (Topics 041 to 085)
queries 1
reinaENtdtest
0.5
041 37.50 086 34.15 131 42.50 166 0.00
Difference
0
042 43.14 087 41.67 132 0.00 167 20.00
−0.5
043 72.73 088 100.00 133 30.00 168 44.44
044 50.00 089 25.00 134 75.00 169 0.00 −1
041 042 043 044 045 046 047 048 049 060 061 062 063 064 065 066 067 068 069 080 081 082 083 084 085
Topic Identifier
045 60.61 090 25.00 135 40.00 170 100.00 Ad−Hoc Robust Monolingual English track − Comparison to Median Mean Exact R−Precision by Topic (Topics 086 to 130)
1
046 33.33 091 28.57 136 0.00 171 10.53 reinaENtdtest
047 58.54 092 74.07 137 0.00 172 0.00 0.5
Difference
048 0.00 093 0.00 138 50.00 173 100.00 0
049 14.29 094 58.33 139 0.00 174 100.00 −0.5
060 0.00 095 40.62 140 47.14 175 71.43 −1
086 087 088 089 090 091 092 093 094 095 096 097 098 099 110 111 112 113 114 115 116 117 118 119 130
061 40.00 096 0.00 141 100.00 176 82.05 Topic Identifier
Ad−Hoc Robust Monolingual English track − Comparison to Median Mean Exact R−Precision by Topic (Topics 131 to 165)
062 100.00 097 100.00 142 87.50 177 23.08 1
reinaENtdtest
063 0.00 098 100.00 143 71.79 178 25.00 0.5
Difference
064 70.00 099 14.29 144 33.33 179 54.55
0
065 40.00 110 0.00 145 63.16 190 77.78
−0.5
066 81.82 111 35.00 146 50.00 191 0.00
−1
067 0.00 112 50.00 147 66.67 192 33.33 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 160 161 162 163 164 165
Topic Identifier
068 28.57 113 0.00 148 16.67 193 28.57 Ad−Hoc Robust Monolingual English track − Comparison to Median Mean Exact R−Precision by Topic (Topics 166 to 200)
1
069 25.00 114 57.14 149 0.00 194 0.00 reinaENtdtest
0.5
080 37.50 115 20.00 160 0.00 195 0.00
Difference
081 50.00 116 45.45 161 0.00 196 100.00 0
082 40.00 117 0.00 162 0.00 197 68.00 −0.5
083 20.00 118 0.00 163 78.69 198 100.00 −1
166 167 168 169 170 171 172 173 174 175 176 177 178 179 190 191 192 193 194 195 196 197 198 199 200
084 60.00 119 67.86 164 7.41 199 70.00 Topic Identifier
085 40.62 130 50.79 165 100.00 200 33.33
144
colesir CoLesIResTst AH-ROBUST-MONO-ES-CLEF2006
Overall statistics for 100 queries : Priority 1
Total number of documents over all queries Query Construction AUTOMATIC
Retrieved 100,000 Source Language Spanish; Castilian
Relevant 5,008 Topic Fields title, description
Relevant retrieved 4,105 Pooled false
Geometric Mean Average Precision 0.1885 robust monolingual ES - TEST; automatic; TD
Binary Preference (BPREF) 0.5027
Interploated Recall (%) Precision Averages (%)
Ad−Hoc Robust Monolingual Spanish track − Interpolated Recall vs Average Precision
100%
0 71.62
CoLesIResTst
10 61.36 90%
20 55.85
80%
30 51.15
40 46.85 70%
50 43.08
Average Precision
60%
60 38.32
70 33.88 50%
80 28.16 40%
90 18.03
30%
100 10.51
Average precision (non-interpolated) for all 20%
relevant documents (averaged over queries)
40.17 10%
0%
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
Interpolated Recall
Mean Average Precision Ad−Hoc Robust Monolingual Spanish track − Box plot of the Topics of the Experiment
Maximum 1.0000
Minimum 0.0000
First Quartile 0.1275
Second Quartile 0.3407
Third Quartile 0.6869
Interquartile range 0.5595
Mean 0.4017
Standard Deviation 0.2999
Lower Outlier Threshold 0.0000
Upper Outlier Threshold 1.0000 0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Mean Average Precision
Mean With No Outliers 0.4017
Std With No Outliers 0.2999
Ad−Hoc Robust Monolingual Spanish track − Distribution of the Topics of the Experiment
15
Number of Topics of the Experiment
10
5
0
0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Mean Average Precision
Precision averages (%) for individual Ad−Hoc Robust Monolingual Spanish track − Comparison to Median Mean Average Precision by Topic (Topics 041 to 085)
queries 1
CoLesIResTst
0.5
041 32.86 086 26.30 131 35.40 166 4.20
Difference
0
042 63.48 087 50.03 132 60.44 167 49.18
−0.5
043 8.64 088 34.00 133 1.21 168 12.46
044 29.94 089 44.60 134 91.67 169 0.00 −1
041 042 043 044 045 046 047 048 049 060 061 062 063 064 065 066 067 068 069 080 081 082 083 084 085
Topic Identifier
045 45.12 090 7.50 135 50.70 170 53.76 Ad−Hoc Robust Monolingual Spanish track − Comparison to Median Mean Average Precision by Topic (Topics 086 to 130)
1
046 2.82 091 4.80 136 46.78 171 13.42 CoLesIResTst
047 11.00 092 21.50 137 16.64 172 13.61 0.5
Difference
048 7.17 093 13.03 138 70.50 173 32.02 0
049 9.84 094 79.09 139 11.63 174 78.62 −0.5
060 22.66 095 0.45 140 3.31 175 100.00 −1
086 087 088 089 090 091 092 093 094 095 096 097 098 099 110 111 112 113 114 115 116 117 118 119 130
061 0.00 096 13.38 141 81.06 176 95.94 Topic Identifier
Ad−Hoc Robust Monolingual Spanish track − Comparison to Median Mean Average Precision by Topic (Topics 131 to 165)
062 25.77 097 56.63 142 80.67 177 11.29 1
CoLesIResTst
063 81.02 098 67.27 143 46.60 178 47.14 0.5
Difference
064 100.00 099 19.48 144 50.89 179 39.88
0
065 16.69 110 61.68 145 34.14 190 2.56
−0.5
066 89.25 111 0.23 146 7.78 191 100.00
−1
067 10.54 112 39.09 147 0.18 192 51.95 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 160 161 162 163 164 165
Topic Identifier
068 9.42 113 24.94 148 14.43 193 14.22 Ad−Hoc Robust Monolingual Spanish track − Comparison to Median Mean Average Precision by Topic (Topics 166 to 200)
1
069 77.43 114 7.03 149 73.31 194 68.23 CoLesIResTst
0.5
080 32.38 115 21.18 160 60.00 195 0.00
Difference
081 69.15 116 24.05 161 80.00 196 75.98 0
082 86.69 117 23.58 162 85.13 197 70.74 −0.5
083 53.31 118 25.83 163 65.14 198 24.52 −1
166 167 168 169 170 171 172 173 174 175 176 177 178 179 190 191 192 193 194 195 196 197 198 199 200
084 26.93 119 75.10 164 1.73 199 53.14 Topic Identifier
085 72.24 130 69.66 165 69.96 200 70.29
145
colesir CoLesIResTst AH-ROBUST-MONO-ES-CLEF2006
Docs Cutoff Levels Precision at DCL (%)
Ad−Hoc Robust Monolingual Spanish track − Retrieved documents vs Precision
100%
5 docs 48.00
CoLesIResTst
10 docs 43.10 90%
15 docs 39.40
80%
20 docs 35.95
30 docs 32.33 70%
100 docs 20.69
60%
200 docs 13.82
R−Precision
500 docs 7.32 50%
1000 docs 4.11 40%
R-Precision (precision after R document retrieved,
where R = Relevant retrieved) 30%
40.03
20%
10%
0%
5 10 15 20 30 100 200 500 1000
Retrieved Documents (logarithmic scale)
Exact R-Precision Ad−Hoc Robust Monolingual Spanish track − Box plot of the Topics of the Experiment
Maximum 1.0000
Minimum 0.0000
First Quartile 0.1538
Second Quartile 0.3750
Third Quartile 0.6354
Interquartile range 0.4816
Mean 0.4003
Standard Deviation 0.2797
Lower Outlier Threshold 0.0000
Upper Outlier Threshold 1.0000 0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Exact R−Precision
Mean With No Outliers 0.4003
Std With No Outliers 0.2797
Ad−Hoc Robust Monolingual Spanish track − Distribution of the Topics of the Experiment
15
Number of Topics of the Experiment
10
5
0
0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Exact R−Precision
Precision averages (%) for individual Ad−Hoc Robust Monolingual Spanish track − Comparison to Median Mean Exact R−Precision by Topic (Topics 041 to 085)
queries 1
CoLesIResTst
0.5
041 37.50 086 25.00 131 36.36 166 0.00
Difference
0
042 61.30 087 47.73 132 50.00 167 40.00
−0.5
043 11.11 088 14.29 133 0.00 168 15.38
044 30.00 089 56.25 134 75.00 169 0.00 −1
041 042 043 044 045 046 047 048 049 060 061 062 063 064 065 066 067 068 069 080 081 082 083 084 085
Topic Identifier
045 49.21 090 13.33 135 53.85 170 40.00 Ad−Hoc Robust Monolingual Spanish track − Comparison to Median Mean Exact R−Precision by Topic (Topics 086 to 130)
1
046 0.00 091 7.53 136 75.00 171 25.00 CoLesIResTst
047 17.59 092 25.58 137 20.00 172 15.38 0.5
Difference
048 15.62 093 28.57 138 71.43 173 50.00 0
049 15.79 094 81.82 139 14.37 174 81.82 −0.5
060 32.00 095 3.43 140 10.92 175 100.00 −1
086 087 088 089 090 091 092 093 094 095 096 097 098 099 110 111 112 113 114 115 116 117 118 119 130
061 0.00 096 20.43 141 76.92 176 89.61 Topic Identifier
Ad−Hoc Robust Monolingual Spanish track − Comparison to Median Mean Exact R−Precision by Topic (Topics 131 to 165)
062 13.33 097 72.00 142 67.86 177 17.39 1
CoLesIResTst
063 76.27 098 60.00 143 51.39 178 48.86 0.5
Difference
064 100.00 099 25.00 144 50.00 179 43.36
0
065 27.03 110 57.14 145 31.91 190 0.00
−0.5
066 81.82 111 0.00 146 0.00 191 100.00
−1
067 14.29 112 50.00 147 0.00 192 52.27 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 160 161 162 163 164 165
Topic Identifier
068 11.11 113 30.38 148 9.09 193 20.59 Ad−Hoc Robust Monolingual Spanish track − Comparison to Median Mean Exact R−Precision by Topic (Topics 166 to 200)
1
069 66.67 114 6.67 149 69.23 194 62.50 CoLesIResTst
0.5
080 37.14 115 21.05 160 50.00 195 0.00
Difference
081 65.52 116 28.57 161 80.00 196 73.33 0
082 76.32 117 28.57 162 77.52 197 64.58 −0.5
083 44.44 118 20.00 163 56.67 198 12.50 −1
166 167 168 169 170 171 172 173 174 175 176 177 178 179 190 191 192 193 194 195 196 197 198 199 200
084 37.50 119 70.48 164 2.78 199 55.29 Topic Identifier
085 75.70 130 61.54 165 44.44 200 68.09
146
daedalus esx011esRFW3FS3 AH-ROBUST-MONO-ES-CLEF2006
Overall statistics for 100 queries : Priority 2
Total number of documents over all queries Query Construction AUTOMATIC
Retrieved 100,000 Source Language Spanish; Castilian
Relevant 5,008 Topic Fields title, description, narrative
Relevant retrieved 4,500 Pooled false
Geometric Mean Average Precision 0.2605 Combined run
Binary Preference (BPREF) 0.5613
Interploated Recall (%) Precision Averages (%)
Ad−Hoc Robust Monolingual Spanish track − Interpolated Recall vs Average Precision
100%
0 76.50
esx011esRFW3FS3
10 67.98 90%
20 62.00
80%
30 57.03
40 53.47 70%
50 49.02
Average Precision
60%
60 43.20
70 38.28 50%
80 30.25 40%
90 22.12
30%
100 12.68
Average precision (non-interpolated) for all 20%
relevant documents (averaged over queries)
45.09 10%
0%
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
Interpolated Recall
Mean Average Precision Ad−Hoc Robust Monolingual Spanish track − Box plot of the Topics of the Experiment
Maximum 1.0000
Minimum 0.0000
First Quartile 0.2090
Second Quartile 0.4207
Third Quartile 0.6951
Interquartile range 0.4861
Mean 0.4509
Standard Deviation 0.2855
Lower Outlier Threshold 0.0000
Upper Outlier Threshold 1.0000 0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Mean Average Precision
Mean With No Outliers 0.4509
Std With No Outliers 0.2855
Ad−Hoc Robust Monolingual Spanish track − Distribution of the Topics of the Experiment
8
Number of Topics of the Experiment
7
6
5
4
3
2
1
0
0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Mean Average Precision
Precision averages (%) for individual Ad−Hoc Robust Monolingual Spanish track − Comparison to Median Mean Average Precision by Topic (Topics 041 to 085)
queries 1
esx011esRFW3FS3
0.5
041 35.87 086 32.07 131 36.74 166 7.22
Difference
0
042 71.30 087 33.19 132 64.69 167 44.02
−0.5
043 41.68 088 28.05 133 20.40 168 9.93
044 24.40 089 43.79 134 20.15 169 0.00 −1
041 042 043 044 045 046 047 048 049 060 061 062 063 064 065 066 067 068 069 080 081 082 083 084 085
Topic Identifier
045 60.76 090 11.65 135 55.12 170 85.88 Ad−Hoc Robust Monolingual Spanish track − Comparison to Median Mean Average Precision by Topic (Topics 086 to 130)
1
046 3.91 091 32.69 136 55.07 171 51.84 esx011esRFW3FS3
047 19.88 092 28.48 137 16.07 172 30.20 0.5
Difference
048 16.16 093 28.19 138 75.46 173 66.13 0
049 11.63 094 87.23 139 11.91 174 90.30 −0.5
060 48.79 095 10.05 140 11.84 175 100.00 −1
086 087 088 089 090 091 092 093 094 095 096 097 098 099 110 111 112 113 114 115 116 117 118 119 130
061 0.00 096 19.57 141 84.95 176 94.08 Topic Identifier
Ad−Hoc Robust Monolingual Spanish track − Comparison to Median Mean Average Precision by Topic (Topics 131 to 165)
062 21.41 097 65.86 142 73.34 177 28.89 1
esx011esRFW3FS3
063 71.62 098 45.89 143 36.07 178 23.11 0.5
Difference
064 100.00 099 29.26 144 50.76 179 31.87
0
065 25.13 110 67.23 145 62.72 190 4.48
−0.5
066 93.34 111 0.46 146 56.25 191 20.00
−1
067 11.71 112 42.45 147 9.86 192 70.28 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 160 161 162 163 164 165
Topic Identifier
068 9.04 113 23.67 148 13.16 193 13.75 Ad−Hoc Robust Monolingual Spanish track − Comparison to Median Mean Average Precision by Topic (Topics 166 to 200)
1
069 88.33 114 47.95 149 93.43 194 66.15 esx011esRFW3FS3
0.5
080 47.38 115 29.82 160 45.00 195 0.00
Difference
081 79.81 116 27.32 161 100.00 196 73.26 0
082 84.38 117 23.66 162 88.32 197 64.65 −0.5
083 52.34 118 42.88 163 64.72 198 40.02 −1
166 167 168 169 170 171 172 173 174 175 176 177 178 179 190 191 192 193 194 195 196 197 198 199 200
084 35.62 119 73.33 164 32.85 199 72.08 Topic Identifier
085 83.89 130 68.73 165 73.43 200 86.93
147
daedalus esx011esRFW3FS3 AH-ROBUST-MONO-ES-CLEF2006
Docs Cutoff Levels Precision at DCL (%)
Ad−Hoc Robust Monolingual Spanish track − Retrieved documents vs Precision
100%
5 docs 55.20
esx011esRFW3FS3
10 docs 49.10 90%
15 docs 45.07
80%
20 docs 41.75
30 docs 36.67 70%
100 docs 22.50
60%
200 docs 15.36
R−Precision
500 docs 8.13 50%
1000 docs 4.50 40%
R-Precision (precision after R document retrieved,
where R = Relevant retrieved) 30%
44.57
20%
10%
0%
5 10 15 20 30 100 200 500 1000
Retrieved Documents (logarithmic scale)
Exact R-Precision Ad−Hoc Robust Monolingual Spanish track − Box plot of the Topics of the Experiment
Maximum 1.0000
Minimum 0.0000
First Quartile 0.2500
Second Quartile 0.4118
Third Quartile 0.6667
Interquartile range 0.4167
Mean 0.4457
Standard Deviation 0.2702
Lower Outlier Threshold 0.0000
Upper Outlier Threshold 1.0000 0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Exact R−Precision
Mean With No Outliers 0.4457
Std With No Outliers 0.2702
Ad−Hoc Robust Monolingual Spanish track − Distribution of the Topics of the Experiment
15
Number of Topics of the Experiment
10
5
0
0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Exact R−Precision
Precision averages (%) for individual Ad−Hoc Robust Monolingual Spanish track − Comparison to Median Mean Exact R−Precision by Topic (Topics 041 to 085)
queries 1
esx011esRFW3FS3
0.5
041 37.50 086 37.50 131 38.18 166 0.00
Difference
0
042 67.05 087 37.50 132 57.69 167 40.00
−0.5
043 38.89 088 42.86 133 26.32 168 10.26
044 25.00 089 50.00 134 0.00 169 0.00 −1
041 042 043 044 045 046 047 048 049 060 061 062 063 064 065 066 067 068 069 080 081 082 083 084 085
Topic Identifier
045 58.64 090 13.33 135 56.41 170 80.00 Ad−Hoc Robust Monolingual Spanish track − Comparison to Median Mean Exact R−Precision by Topic (Topics 086 to 130)
1
046 6.67 091 32.26 136 75.00 171 50.00 esx011esRFW3FS3
047 25.00 092 32.56 137 20.00 172 46.15 0.5
Difference
048 15.62 093 28.57 138 85.71 173 50.00 0
049 15.79 094 81.82 139 10.34 174 81.82 −0.5
060 50.67 095 22.12 140 18.49 175 100.00 −1
086 087 088 089 090 091 092 093 094 095 096 097 098 099 110 111 112 113 114 115 116 117 118 119 130
061 0.00 096 27.96 141 76.92 176 87.01 Topic Identifier
Ad−Hoc Robust Monolingual Spanish track − Comparison to Median Mean Exact R−Precision by Topic (Topics 131 to 165)
062 20.00 097 68.00 142 71.43 177 30.43 1
esx011esRFW3FS3
063 64.41 098 40.00 143 42.36 178 35.23 0.5
Difference
064 100.00 099 35.71 144 59.09 179 32.74
0
065 37.84 110 57.14 145 55.32 190 0.00
−0.5
066 86.87 111 0.00 146 50.00 191 0.00
−1
067 14.29 112 37.50 147 11.11 192 70.45 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 160 161 162 163 164 165
Topic Identifier
068 0.00 113 27.85 148 9.09 193 17.65 Ad−Hoc Robust Monolingual Spanish track − Comparison to Median Mean Exact R−Precision by Topic (Topics 166 to 200)
1
069 83.33 114 46.67 149 84.62 194 75.00 esx011esRFW3FS3
0.5
080 48.98 115 31.58 160 50.00 195 0.00
Difference
081 79.31 116 14.29 161 100.00 196 73.33 0
082 78.95 117 37.50 162 79.07 197 62.08 −0.5
083 44.44 118 40.00 163 66.67 198 50.00 −1
166 167 168 169 170 171 172 173 174 175 176 177 178 179 190 191 192 193 194 195 196 197 198 199 200
084 37.50 119 66.67 164 38.89 199 63.53 Topic Identifier
085 75.70 130 53.85 165 66.67 200 76.60
148
daedalus esFSesR2S AH-ROBUST-MONO-ES-CLEF2006
Overall statistics for 100 queries : Priority 4
Total number of documents over all queries Query Construction AUTOMATIC
Retrieved 100,000 Source Language Spanish; Castilian
Relevant 5,008 Topic Fields title, description
Relevant retrieved 4,314 Pooled false
Geometric Mean Average Precision 0.1964 Mandatory TD run
Binary Preference (BPREF) 0.5054
Interploated Recall (%) Precision Averages (%)
Ad−Hoc Robust Monolingual Spanish track − Interpolated Recall vs Average Precision
100%
0 73.32
esFSesR2S
10 62.91 90%
20 57.78
80%
30 51.77
40 48.53 70%
50 43.45
Average Precision
60%
60 37.55
70 32.87 50%
80 25.28 40%
90 18.07
30%
100 9.76
Average precision (non-interpolated) for all 20%
relevant documents (averaged over queries)
40.40 10%
0%
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
Interpolated Recall
Mean Average Precision Ad−Hoc Robust Monolingual Spanish track − Box plot of the Topics of the Experiment
Maximum 1.0000
Minimum 0.0000
First Quartile 0.1609
Second Quartile 0.3550
Third Quartile 0.6507
Interquartile range 0.4898
Mean 0.4040
Standard Deviation 0.2825
Lower Outlier Threshold 0.0000
Upper Outlier Threshold 1.0000 0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Mean Average Precision
Mean With No Outliers 0.4040
Std With No Outliers 0.2825
Ad−Hoc Robust Monolingual Spanish track − Distribution of the Topics of the Experiment
15
Number of Topics of the Experiment
10
5
0
0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Mean Average Precision
Precision averages (%) for individual Ad−Hoc Robust Monolingual Spanish track − Comparison to Median Mean Average Precision by Topic (Topics 041 to 085)
queries 1
esFSesR2S
0.5
041 23.67 086 24.68 131 36.84 166 3.17
Difference
0
042 63.85 087 39.12 132 65.12 167 44.73
−0.5
043 41.32 088 48.28 133 0.00 168 4.50
044 18.55 089 44.20 134 12.56 169 0.00 −1
041 042 043 044 045 046 047 048 049 060 061 062 063 064 065 066 067 068 069 080 081 082 083 084 085
Topic Identifier
045 63.51 090 11.61 135 54.82 170 71.43 Ad−Hoc Robust Monolingual Spanish track − Comparison to Median Mean Average Precision by Topic (Topics 086 to 130)
1
046 2.85 091 31.67 136 44.86 171 34.54 esFSesR2S
047 12.00 092 25.81 137 17.65 172 14.57 0.5
Difference
048 10.65 093 15.78 138 80.87 173 59.52 0
049 16.82 094 85.23 139 12.57 174 82.27 −0.5
060 60.51 095 4.98 140 10.96 175 100.00 −1
086 087 088 089 090 091 092 093 094 095 096 097 098 099 110 111 112 113 114 115 116 117 118 119 130
061 0.00 096 22.79 141 83.01 176 92.45 Topic Identifier
Ad−Hoc Robust Monolingual Spanish track − Comparison to Median Mean Average Precision by Topic (Topics 131 to 165)
062 24.05 097 37.64 142 73.97 177 28.67 1
esFSesR2S
063 75.82 098 34.93 143 29.50 178 36.07 0.5
Difference
064 100.00 099 25.05 144 40.78 179 6.97
0
065 12.48 110 66.42 145 65.03 190 3.84
−0.5
066 94.05 111 0.05 146 14.29 191 25.00
−1
067 21.01 112 36.77 147 15.17 192 52.16 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 160 161 162 163 164 165
Topic Identifier
068 14.21 113 25.54 148 14.81 193 18.21 Ad−Hoc Robust Monolingual Spanish track − Comparison to Median Mean Average Precision by Topic (Topics 166 to 200)
1
069 89.58 114 1.40 149 62.15 194 43.41 esFSesR2S
0.5
080 45.54 115 20.82 160 70.00 195 0.00
Difference
081 84.99 116 16.40 161 100.00 196 71.71 0
082 72.01 117 22.37 162 66.14 197 56.26 −0.5
083 41.68 118 32.26 163 65.99 198 29.41 −1
166 167 168 169 170 171 172 173 174 175 176 177 178 179 190 191 192 193 194 195 196 197 198 199 200
084 31.17 119 69.48 164 23.36 199 61.53 Topic Identifier
085 77.34 130 71.50 165 56.61 200 70.22
149
daedalus esFSesR2S AH-ROBUST-MONO-ES-CLEF2006
Docs Cutoff Levels Precision at DCL (%)
Ad−Hoc Robust Monolingual Spanish track − Retrieved documents vs Precision
100%
5 docs 50.20
esFSesR2S
10 docs 44.60 90%
15 docs 40.27
80%
20 docs 37.20
30 docs 33.93 70%
100 docs 21.12
60%
200 docs 14.34
R−Precision
500 docs 7.51 50%
1000 docs 4.31 40%
R-Precision (precision after R document retrieved,
where R = Relevant retrieved) 30%
40.15
20%
10%
0%
5 10 15 20 30 100 200 500 1000
Retrieved Documents (logarithmic scale)
Exact R-Precision Ad−Hoc Robust Monolingual Spanish track − Box plot of the Topics of the Experiment
Maximum 1.0000
Minimum 0.0000
First Quartile 0.2019
Second Quartile 0.3944
Third Quartile 0.5916
Interquartile range 0.3898
Mean 0.4015
Standard Deviation 0.2661
Lower Outlier Threshold 0.0000
Upper Outlier Threshold 1.0000 0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Exact R−Precision
Mean With No Outliers 0.4015
Std With No Outliers 0.2661
Ad−Hoc Robust Monolingual Spanish track − Distribution of the Topics of the Experiment
15
Number of Topics of the Experiment
10
5
0
0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Exact R−Precision
Precision averages (%) for individual Ad−Hoc Robust Monolingual Spanish track − Comparison to Median Mean Exact R−Precision by Topic (Topics 041 to 085)
queries 1
esFSesR2S
0.5
041 31.25 086 32.14 131 37.27 166 0.00
Difference
0
042 60.54 087 42.05 132 57.69 167 40.00
−0.5
043 38.89 088 42.86 133 0.00 168 2.56
044 20.00 089 56.25 134 0.00 169 0.00 −1
041 042 043 044 045 046 047 048 049 060 061 062 063 064 065 066 067 068 069 080 081 082 083 084 085
Topic Identifier
045 59.16 090 6.67 135 54.49 170 40.00 Ad−Hoc Robust Monolingual Spanish track − Comparison to Median Mean Exact R−Precision by Topic (Topics 086 to 130)
1
046 0.00 091 34.41 136 62.50 171 25.00 esFSesR2S
047 20.37 092 30.23 137 20.00 172 15.38 0.5
Difference
048 10.42 093 35.71 138 85.71 173 50.00 0
049 26.32 094 81.82 139 10.92 174 81.82 −0.5
060 60.00 095 13.71 140 18.49 175 100.00 −1
086 087 088 089 090 091 092 093 094 095 096 097 098 099 110 111 112 113 114 115 116 117 118 119 130
061 0.00 096 30.11 141 76.92 176 76.62 Topic Identifier
Ad−Hoc Robust Monolingual Spanish track − Comparison to Median Mean Exact R−Precision by Topic (Topics 131 to 165)
062 26.67 097 48.00 142 71.43 177 34.78 1
esFSesR2S
063 69.49 098 40.00 143 34.03 178 46.59 0.5
Difference
064 100.00 099 32.14 144 54.55 179 7.96
0
065 22.97 110 57.14 145 57.45 190 0.00
−0.5
066 86.87 111 0.00 146 0.00 191 0.00
−1
067 30.61 112 50.00 147 22.22 192 52.27 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 160 161 162 163 164 165
Topic Identifier
068 22.22 113 27.85 148 9.09 193 26.47 Ad−Hoc Robust Monolingual Spanish track − Comparison to Median Mean Exact R−Precision by Topic (Topics 166 to 200)
1
069 83.33 114 2.22 149 53.85 194 50.00 esFSesR2S
0.5
080 48.57 115 21.05 160 50.00 195 0.00
Difference
081 80.46 116 14.29 161 100.00 196 73.33 0
082 71.05 117 32.14 162 64.34 197 59.17 −0.5
083 44.44 118 40.00 163 63.33 198 12.50 −1
166 167 168 169 170 171 172 173 174 175 176 177 178 179 190 191 192 193 194 195 196 197 198 199 200
084 37.50 119 60.95 164 31.94 199 65.88 Topic Identifier
085 75.70 130 53.85 165 44.44 200 61.70
150
daedalus esx021esRFW3FS3 AH-ROBUST-MONO-ES-CLEF2006
Overall statistics for 100 queries : Priority 3
Total number of documents over all queries Query Construction AUTOMATIC
Retrieved 100,000 Source Language Spanish; Castilian
Relevant 5,008 Topic Fields title, description, narrative
Relevant retrieved 4,500 Pooled false
Geometric Mean Average Precision 0.2561 Combined run
Binary Preference (BPREF) 0.5584
Interploated Recall (%) Precision Averages (%)
Ad−Hoc Robust Monolingual Spanish track − Interpolated Recall vs Average Precision
100%
0 75.39
esx021esRFW3FS3
10 66.88 90%
20 60.74
80%
30 56.19
40 52.86 70%
50 48.48
Average Precision
60%
60 43.00
70 38.08 50%
80 30.07 40%
90 21.99
30%
100 12.55
Average precision (non-interpolated) for all 20%
relevant documents (averaged over queries)
44.46 10%
0%
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
Interpolated Recall
Mean Average Precision Ad−Hoc Robust Monolingual Spanish track − Box plot of the Topics of the Experiment
Maximum 1.0000
Minimum 0.0000
First Quartile 0.2078
Second Quartile 0.4064
Third Quartile 0.6669
Interquartile range 0.4591
Mean 0.4446
Standard Deviation 0.2841
Lower Outlier Threshold 0.0000
Upper Outlier Threshold 1.0000 0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Mean Average Precision
Mean With No Outliers 0.4446
Std With No Outliers 0.2841
Ad−Hoc Robust Monolingual Spanish track − Distribution of the Topics of the Experiment
8
Number of Topics of the Experiment
7
6
5
4
3
2
1
0
0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Mean Average Precision
Precision averages (%) for individual Ad−Hoc Robust Monolingual Spanish track − Comparison to Median Mean Average Precision by Topic (Topics 041 to 085)
queries 1
esx021esRFW3FS3
0.5
041 37.44 086 31.60 131 37.16 166 7.22
Difference
0
042 71.30 087 33.19 132 64.69 167 44.02
−0.5
043 41.12 088 30.43 133 17.19 168 9.93
044 24.40 089 44.83 134 20.15 169 0.00 −1
041 042 043 044 045 046 047 048 049 060 061 062 063 064 065 066 067 068 069 080 081 082 083 084 085
Topic Identifier
045 60.76 090 11.32 135 55.61 170 85.88 Ad−Hoc Robust Monolingual Spanish track − Comparison to Median Mean Average Precision by Topic (Topics 086 to 130)
1
046 3.91 091 31.92 136 55.07 171 43.51 esx021esRFW3FS3
047 19.72 092 28.39 137 14.53 172 25.47 0.5
Difference
048 16.16 093 27.00 138 62.03 173 66.13 0
049 11.63 094 87.23 139 11.91 174 90.30 −0.5
060 48.79 095 10.00 140 11.66 175 100.00 −1
086 087 088 089 090 091 092 093 094 095 096 097 098 099 110 111 112 113 114 115 116 117 118 119 130
061 0.00 096 19.45 141 84.95 176 94.08 Topic Identifier
Ad−Hoc Robust Monolingual Spanish track − Comparison to Median Mean Average Precision by Topic (Topics 131 to 165)
062 21.41 097 65.86 142 73.34 177 30.34 1
esx021esRFW3FS3
063 71.62 098 45.89 143 35.67 178 22.92 0.5
Difference
064 100.00 099 29.26 144 48.11 179 30.86
0
065 25.36 110 67.23 145 62.72 190 4.48
−0.5
066 93.34 111 0.46 146 56.25 191 16.67
−1
067 11.26 112 42.45 147 8.93 192 70.28 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 160 161 162 163 164 165
Topic Identifier
068 9.01 113 23.67 148 12.41 193 13.75 Ad−Hoc Robust Monolingual Spanish track − Comparison to Median Mean Average Precision by Topic (Topics 166 to 200)
1
069 88.33 114 47.95 149 93.43 194 66.15 esx021esRFW3FS3
0.5
080 47.38 115 29.82 160 45.00 195 0.00
Difference
081 78.72 116 27.32 161 100.00 196 73.26 0
082 77.88 117 23.66 162 88.32 197 64.82 −0.5
083 48.82 118 40.15 163 64.72 198 40.02 −1
166 167 168 169 170 171 172 173 174 175 176 177 178 179 190 191 192 193 194 195 196 197 198 199 200
084 30.78 119 73.33 164 33.82 199 72.08 Topic Identifier
085 83.89 130 64.25 165 69.72 200 86.93
151
daedalus esx021esRFW3FS3 AH-ROBUST-MONO-ES-CLEF2006
Docs Cutoff Levels Precision at DCL (%)
Ad−Hoc Robust Monolingual Spanish track − Retrieved documents vs Precision
100%
5 docs 53.80
esx021esRFW3FS3
10 docs 48.40 90%
15 docs 44.93
80%
20 docs 41.65
30 docs 36.63 70%
100 docs 22.50
60%
200 docs 15.36
R−Precision
500 docs 8.13 50%
1000 docs 4.50 40%
R-Precision (precision after R document retrieved,
where R = Relevant retrieved) 30%
44.20
20%
10%
0%
5 10 15 20 30 100 200 500 1000
Retrieved Documents (logarithmic scale)
Exact R-Precision Ad−Hoc Robust Monolingual Spanish track − Box plot of the Topics of the Experiment
Maximum 1.0000
Minimum 0.0000
First Quartile 0.2356
Second Quartile 0.4118
Third Quartile 0.6667
Interquartile range 0.4311
Mean 0.4420
Standard Deviation 0.2691
Lower Outlier Threshold 0.0000
Upper Outlier Threshold 1.0000 0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Exact R−Precision
Mean With No Outliers 0.4420
Std With No Outliers 0.2691
Ad−Hoc Robust Monolingual Spanish track − Distribution of the Topics of the Experiment
15
Number of Topics of the Experiment
10
5
0
0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Exact R−Precision
Precision averages (%) for individual Ad−Hoc Robust Monolingual Spanish track − Comparison to Median Mean Exact R−Precision by Topic (Topics 041 to 085)
queries 1
esx021esRFW3FS3
0.5
041 37.50 086 37.50 131 38.18 166 0.00
Difference
0
042 67.05 087 37.50 132 57.69 167 40.00
−0.5
043 38.89 088 42.86 133 26.32 168 10.26
044 25.00 089 50.00 134 0.00 169 0.00 −1
041 042 043 044 045 046 047 048 049 060 061 062 063 064 065 066 067 068 069 080 081 082 083 084 085
Topic Identifier
045 58.64 090 13.33 135 56.41 170 80.00 Ad−Hoc Robust Monolingual Spanish track − Comparison to Median Mean Exact R−Precision by Topic (Topics 086 to 130)
1
046 6.67 091 32.26 136 75.00 171 50.00 esx021esRFW3FS3
047 25.00 092 32.56 137 20.00 172 46.15 0.5
Difference
048 15.62 093 28.57 138 71.43 173 50.00 0
049 15.79 094 81.82 139 10.34 174 81.82 −0.5
060 50.67 095 22.12 140 18.49 175 100.00 −1
086 087 088 089 090 091 092 093 094 095 096 097 098 099 110 111 112 113 114 115 116 117 118 119 130
061 0.00 096 27.96 141 76.92 176 87.01 Topic Identifier
Ad−Hoc Robust Monolingual Spanish track − Comparison to Median Mean Exact R−Precision by Topic (Topics 131 to 165)
062 20.00 097 68.00 142 71.43 177 30.43 1
esx021esRFW3FS3
063 64.41 098 40.00 143 42.36 178 35.23 0.5
Difference
064 100.00 099 35.71 144 59.09 179 32.74
0
065 37.84 110 57.14 145 55.32 190 0.00
−0.5
066 86.87 111 0.00 146 50.00 191 0.00
−1
067 14.29 112 37.50 147 11.11 192 70.45 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 160 161 162 163 164 165
Topic Identifier
068 0.00 113 27.85 148 9.09 193 17.65 Ad−Hoc Robust Monolingual Spanish track − Comparison to Median Mean Exact R−Precision by Topic (Topics 166 to 200)
1
069 83.33 114 46.67 149 84.62 194 75.00 esx021esRFW3FS3
0.5
080 48.98 115 31.58 160 50.00 195 0.00
Difference
081 79.31 116 14.29 161 100.00 196 73.33 0
082 76.32 117 37.50 162 79.07 197 62.08 −0.5
083 44.44 118 20.00 163 66.67 198 50.00 −1
166 167 168 169 170 171 172 173 174 175 176 177 178 179 190 191 192 193 194 195 196 197 198 199 200
084 37.50 119 66.67 164 38.89 199 63.53 Topic Identifier
085 75.70 130 53.85 165 66.67 200 76.60
152
daedalus esFSesR3S AH-ROBUST-MONO-ES-CLEF2006
Overall statistics for 100 queries : Priority 1
Total number of documents over all queries Query Construction AUTOMATIC
Retrieved 100,000 Source Language Spanish; Castilian
Relevant 5,008 Topic Fields title, description, narrative
Relevant retrieved 4,500 Pooled false
Geometric Mean Average Precision 0.2650 TDN run
Binary Preference (BPREF) 0.5657
Interploated Recall (%) Precision Averages (%)
Ad−Hoc Robust Monolingual Spanish track − Interpolated Recall vs Average Precision
100%
0 81.86
esFSesR3S
10 69.47 90%
20 62.57
80%
30 57.75
40 54.17 70%
50 49.64
Average Precision
60%
60 43.33
70 38.40 50%
80 30.38 40%
90 22.24
30%
100 12.81
Average precision (non-interpolated) for all 20%
relevant documents (averaged over queries)
45.84 10%
0%
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
Interpolated Recall
Mean Average Precision Ad−Hoc Robust Monolingual Spanish track − Box plot of the Topics of the Experiment
Maximum 1.0000
Minimum 0.0000
First Quartile 0.2254
Second Quartile 0.4267
Third Quartile 0.7079
Interquartile range 0.4826
Mean 0.4584
Standard Deviation 0.2891
Lower Outlier Threshold 0.0000
Upper Outlier Threshold 1.0000 0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Mean Average Precision
Mean With No Outliers 0.4584
Std With No Outliers 0.2891
Ad−Hoc Robust Monolingual Spanish track − Distribution of the Topics of the Experiment
10
Number of Topics of the Experiment
8
6
4
2
0
0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Mean Average Precision
Precision averages (%) for individual Ad−Hoc Robust Monolingual Spanish track − Comparison to Median Mean Average Precision by Topic (Topics 041 to 085)
queries 1
esFSesR3S
0.5
041 33.79 086 31.18 131 36.22 166 7.22
Difference
0
042 71.30 087 33.19 132 64.69 167 44.02
−0.5
043 41.68 088 28.05 133 26.98 168 9.93
044 27.45 089 43.79 134 20.15 169 0.00 −1
041 042 043 044 045 046 047 048 049 060 061 062 063 064 065 066 067 068 069 080 081 082 083 084 085
Topic Identifier
045 60.76 090 11.65 135 54.68 170 85.88 Ad−Hoc Robust Monolingual Spanish track − Comparison to Median Mean Average Precision by Topic (Topics 086 to 130)
1
046 3.91 091 33.86 136 46.74 171 51.84 esFSesR3S
047 20.34 092 28.92 137 18.40 172 36.61 0.5
Difference
048 16.69 093 28.19 138 100.00 173 66.13 0
049 11.63 094 87.23 139 11.91 174 90.30 −0.5
060 48.79 095 10.20 140 12.26 175 100.00 −1
086 087 088 089 090 091 092 093 094 095 096 097 098 099 110 111 112 113 114 115 116 117 118 119 130
061 0.00 096 19.67 141 84.95 176 94.08 Topic Identifier
Ad−Hoc Robust Monolingual Spanish track − Comparison to Median Mean Average Precision by Topic (Topics 131 to 165)
062 21.41 097 65.86 142 73.34 177 28.89 1
esFSesR3S
063 71.62 098 45.89 143 36.53 178 23.87 0.5
Difference
064 100.00 099 31.04 144 50.76 179 31.54
0
065 25.13 110 67.23 145 62.72 190 4.48
−0.5
066 93.34 111 0.46 146 56.25 191 20.00
−1
067 12.35 112 42.45 147 9.86 192 70.28 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 160 161 162 163 164 165
Topic Identifier
068 9.04 113 23.67 148 13.16 193 13.75 Ad−Hoc Robust Monolingual Spanish track − Comparison to Median Mean Average Precision by Topic (Topics 166 to 200)
1
069 88.33 114 47.95 149 93.43 194 66.15 esFSesR3S
0.5
080 47.38 115 29.82 160 70.00 195 0.00
Difference
081 79.81 116 27.32 161 100.00 196 73.26 0
082 84.38 117 24.55 162 88.32 197 64.44 −0.5
083 52.34 118 42.88 163 72.32 198 46.27 −1
166 167 168 169 170 171 172 173 174 175 176 177 178 179 190 191 192 193 194 195 196 197 198 199 200
084 31.45 119 73.33 164 35.21 199 72.08 Topic Identifier
085 83.89 130 68.73 165 73.43 200 86.93
153
daedalus esFSesR3S AH-ROBUST-MONO-ES-CLEF2006
Docs Cutoff Levels Precision at DCL (%)
Ad−Hoc Robust Monolingual Spanish track − Retrieved documents vs Precision
100%
5 docs 56.40
esFSesR3S
10 docs 49.40 90%
15 docs 45.13
80%
20 docs 41.80
30 docs 36.67 70%
100 docs 22.50
60%
200 docs 15.36
R−Precision
500 docs 8.13 50%
1000 docs 4.50 40%
R-Precision (precision after R document retrieved,
where R = Relevant retrieved) 30%
44.77
20%
10%
0%
5 10 15 20 30 100 200 500 1000
Retrieved Documents (logarithmic scale)
Exact R-Precision Ad−Hoc Robust Monolingual Spanish track − Box plot of the Topics of the Experiment
Maximum 1.0000
Minimum 0.0000
First Quartile 0.2500
Second Quartile 0.4118
Third Quartile 0.6667
Interquartile range 0.4167
Mean 0.4477
Standard Deviation 0.2724
Lower Outlier Threshold 0.0000
Upper Outlier Threshold 1.0000 0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Exact R−Precision
Mean With No Outliers 0.4477
Std With No Outliers 0.2724
Ad−Hoc Robust Monolingual Spanish track − Distribution of the Topics of the Experiment
15
Number of Topics of the Experiment
10
5
0
0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Exact R−Precision
Precision averages (%) for individual Ad−Hoc Robust Monolingual Spanish track − Comparison to Median Mean Exact R−Precision by Topic (Topics 041 to 085)
queries 1
esFSesR3S
0.5
041 37.50 086 37.50 131 38.18 166 0.00
Difference
0
042 67.05 087 37.50 132 57.69 167 40.00
−0.5
043 38.89 088 42.86 133 31.58 168 10.26
044 25.00 089 50.00 134 0.00 169 0.00 −1
041 042 043 044 045 046 047 048 049 060 061 062 063 064 065 066 067 068 069 080 081 082 083 084 085
Topic Identifier
045 58.64 090 13.33 135 56.41 170 80.00 Ad−Hoc Robust Monolingual Spanish track − Comparison to Median Mean Exact R−Precision by Topic (Topics 086 to 130)
1
046 6.67 091 32.26 136 75.00 171 50.00 esFSesR3S
047 25.00 092 32.56 137 20.00 172 46.15 0.5
Difference
048 15.62 093 28.57 138 100.00 173 50.00 0
049 15.79 094 81.82 139 10.34 174 81.82 −0.5
060 50.67 095 22.12 140 18.49 175 100.00 −1
086 087 088 089 090 091 092 093 094 095 096 097 098 099 110 111 112 113 114 115 116 117 118 119 130
061 0.00 096 27.96 141 76.92 176 87.01 Topic Identifier
Ad−Hoc Robust Monolingual Spanish track − Comparison to Median Mean Exact R−Precision by Topic (Topics 131 to 165)
062 20.00 097 68.00 142 71.43 177 30.43 1
esFSesR3S
063 64.41 098 40.00 143 42.36 178 35.23 0.5
Difference
064 100.00 099 35.71 144 59.09 179 32.74
0
065 37.84 110 57.14 145 55.32 190 0.00
−0.5
066 86.87 111 0.00 146 50.00 191 0.00
−1
067 14.29 112 37.50 147 11.11 192 70.45 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 160 161 162 163 164 165
Topic Identifier
068 0.00 113 27.85 148 9.09 193 17.65 Ad−Hoc Robust Monolingual Spanish track − Comparison to Median Mean Exact R−Precision by Topic (Topics 166 to 200)
1
069 83.33 114 46.67 149 84.62 194 75.00 esFSesR3S
0.5
080 48.98 115 31.58 160 50.00 195 0.00
Difference
081 79.31 116 14.29 161 100.00 196 73.33 0
082 78.95 117 37.50 162 79.07 197 62.08 −0.5
083 44.44 118 40.00 163 66.67 198 50.00 −1
166 167 168 169 170 171 172 173 174 175 176 177 178 179 190 191 192 193 194 195 196 197 198 199 200
084 37.50 119 66.67 164 38.89 199 63.53 Topic Identifier
085 75.70 130 53.85 165 66.67 200 76.60
154
dcu dcu1500narrsp1005.qy1 AH-ROBUST-MONO-ES-CLEF2006
Overall statistics for 100 queries : Priority 1
Total number of documents over all queries Query Construction AUTOMATIC
Retrieved 100,000 Source Language Spanish; Castilian
Relevant 5,008 Topic Fields title, description, narrative
Relevant retrieved 4,438 Pooled false
Geometric Mean Average Precision 0.2549 spanish mono reranked
Binary Preference (BPREF) 0.5643
Interploated Recall (%) Precision Averages (%)
Ad−Hoc Robust Monolingual Spanish track − Interpolated Recall vs Average Precision
100%
0 78.25
dcu1500narrsp1005.qy1
10 68.93 90%
20 63.20
80%
30 57.84
40 53.30 70%
50 49.31
Average Precision
60%
60 43.39
70 37.57 50%
80 31.41 40%
90 23.19
30%
100 13.04
Average precision (non-interpolated) for all 20%
relevant documents (averaged over queries)
45.81 10%
0%
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
Interpolated Recall
Mean Average Precision Ad−Hoc Robust Monolingual Spanish track − Box plot of the Topics of the Experiment
Maximum 1.0000
Minimum 0.0000
First Quartile 0.2225
Second Quartile 0.4163
Third Quartile 0.7101
Interquartile range 0.4876
Mean 0.4581
Standard Deviation 0.2927
Lower Outlier Threshold 0.0000
Upper Outlier Threshold 1.0000 0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Mean Average Precision
Mean With No Outliers 0.4581
Std With No Outliers 0.2927
Ad−Hoc Robust Monolingual Spanish track − Distribution of the Topics of the Experiment
10
Number of Topics of the Experiment
8
6
4
2
0
0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Mean Average Precision
Precision averages (%) for individual Ad−Hoc Robust Monolingual Spanish track − Comparison to Median Mean Average Precision by Topic (Topics 041 to 085)
queries 1
dcu1500narrsp1005.qy1
0.5
041 32.98 086 26.84 131 31.16 166 1.44
Difference
0
042 62.25 087 26.37 132 62.78 167 43.08
−0.5
043 39.16 088 34.24 133 46.40 168 17.02
044 23.47 089 48.57 134 88.75 169 0.00 −1
041 042 043 044 045 046 047 048 049 060 061 062 063 064 065 066 067 068 069 080 081 082 083 084 085
Topic Identifier
045 54.26 090 5.98 135 50.64 170 94.29 Ad−Hoc Robust Monolingual Spanish track − Comparison to Median Mean Average Precision by Topic (Topics 086 to 130)
1
046 2.42 091 36.67 136 51.51 171 16.20 dcu1500narrsp1005.qy1
047 16.67 092 34.94 137 11.39 172 13.70 0.5
Difference
048 17.88 093 22.96 138 29.93 173 62.61 0
049 14.09 094 83.97 139 11.52 174 88.78 −0.5
060 25.36 095 6.99 140 15.06 175 100.00 −1
086 087 088 089 090 091 092 093 094 095 096 097 098 099 110 111 112 113 114 115 116 117 118 119 130
061 0.00 096 5.93 141 87.53 176 97.74 Topic Identifier
Ad−Hoc Robust Monolingual Spanish track − Comparison to Median Mean Average Precision by Topic (Topics 131 to 165)
062 26.52 097 77.31 142 90.41 177 32.48 1
dcu1500narrsp1005.qy1
063 87.85 098 68.33 143 63.38 178 26.76 0.5
Difference
064 100.00 099 28.27 144 39.92 179 64.73
0
065 23.30 110 74.01 145 61.65 190 2.44
−0.5
066 89.56 111 1.08 146 52.38 191 25.00
−1
067 10.66 112 47.72 147 15.34 192 76.56 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 160 161 162 163 164 165
Topic Identifier
068 6.97 113 21.54 148 13.84 193 10.29 Ad−Hoc Robust Monolingual Spanish track − Comparison to Median Mean Average Precision by Topic (Topics 166 to 200)
1
069 59.64 114 78.27 149 79.57 194 67.82 dcu1500narrsp1005.qy1
0.5
080 47.24 115 27.36 160 64.29 195 0.00
Difference
081 76.82 116 37.46 161 71.00 196 71.02 0
082 82.42 117 30.43 162 89.32 197 50.22 −0.5
083 60.94 118 40.18 163 61.74 198 56.61 −1
166 167 168 169 170 171 172 173 174 175 176 177 178 179 190 191 192 193 194 195 196 197 198 199 200
084 37.83 119 74.73 164 36.56 199 70.29 Topic Identifier
085 74.41 130 84.21 165 86.45 200 82.06
155
dcu dcu1500narrsp1005.qy1 AH-ROBUST-MONO-ES-CLEF2006
Docs Cutoff Levels Precision at DCL (%)
Ad−Hoc Robust Monolingual Spanish track − Retrieved documents vs Precision
100%
5 docs 55.80
dcu1500narrsp1005.qy1
10 docs 49.20 90%
15 docs 44.87
80%
20 docs 41.10
30 docs 36.33 70%
100 docs 22.73
60%
200 docs 15.24
R−Precision
500 docs 8.02 50%
1000 docs 4.44 40%
R-Precision (precision after R document retrieved,
where R = Relevant retrieved) 30%
43.96
20%
10%
0%
5 10 15 20 30 100 200 500 1000
Retrieved Documents (logarithmic scale)
Exact R-Precision Ad−Hoc Robust Monolingual Spanish track − Box plot of the Topics of the Experiment
Maximum 1.0000
Minimum 0.0000
First Quartile 0.2210
Second Quartile 0.4105
Third Quartile 0.6854
Interquartile range 0.4643
Mean 0.4396
Standard Deviation 0.2723
Lower Outlier Threshold 0.0000
Upper Outlier Threshold 1.0000 0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Exact R−Precision
Mean With No Outliers 0.4396
Std With No Outliers 0.2723
Ad−Hoc Robust Monolingual Spanish track − Distribution of the Topics of the Experiment
10
Number of Topics of the Experiment
8
6
4
2
0
0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Exact R−Precision
Precision averages (%) for individual Ad−Hoc Robust Monolingual Spanish track − Comparison to Median Mean Exact R−Precision by Topic (Topics 041 to 085)
queries 1
dcu1500narrsp1005.qy1
0.5
041 43.75 086 33.93 131 36.36 166 0.00
Difference
0
042 62.07 087 29.55 132 57.69 167 50.00
−0.5
043 44.44 088 28.57 133 42.11 168 15.38
044 20.00 089 56.25 134 75.00 169 0.00 −1
041 042 043 044 045 046 047 048 049 060 061 062 063 064 065 066 067 068 069 080 081 082 083 084 085
Topic Identifier
045 53.40 090 6.67 135 57.69 170 80.00 Ad−Hoc Robust Monolingual Spanish track − Comparison to Median Mean Exact R−Precision by Topic (Topics 086 to 130)
1
046 0.00 091 38.71 136 62.50 171 25.00 dcu1500narrsp1005.qy1
047 23.15 092 38.37 137 20.00 172 15.38 0.5
Difference
048 20.83 093 28.57 138 14.29 173 33.33 0
049 21.05 094 86.36 139 11.49 174 81.82 −0.5
060 30.67 095 17.13 140 23.53 175 100.00 −1
086 087 088 089 090 091 092 093 094 095 096 097 098 099 110 111 112 113 114 115 116 117 118 119 130
061 0.00 096 5.38 141 84.62 176 93.51 Topic Identifier
Ad−Hoc Robust Monolingual Spanish track − Comparison to Median Mean Exact R−Precision by Topic (Topics 131 to 165)
062 20.00 097 72.00 142 82.14 177 26.09 1
dcu1500narrsp1005.qy1
063 79.66 098 40.00 143 56.25 178 36.36 0.5
Difference
064 100.00 099 32.14 144 50.00 179 67.26
0
065 35.14 110 71.43 145 53.19 190 0.00
−0.5
066 83.84 111 0.00 146 50.00 191 0.00
−1
067 6.12 112 37.50 147 11.11 192 68.18 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 160 161 162 163 164 165
Topic Identifier
068 0.00 113 26.58 148 9.09 193 11.76 Ad−Hoc Robust Monolingual Spanish track − Comparison to Median Mean Exact R−Precision by Topic (Topics 166 to 200)
1
069 66.67 114 68.89 149 69.23 194 75.00 dcu1500narrsp1005.qy1
0.5
080 51.02 115 31.58 160 50.00 195 0.00
Difference
081 73.56 116 28.57 161 80.00 196 66.67 0
082 73.68 117 33.93 162 82.17 197 52.92 −0.5
083 44.44 118 40.00 163 53.33 198 62.50 −1
166 167 168 169 170 171 172 173 174 175 176 177 178 179 190 191 192 193 194 195 196 197 198 199 200
084 37.50 119 71.43 164 34.72 199 64.71 Topic Identifier
085 70.09 130 69.23 165 77.78 200 72.34
156
dcu dcudescsp12075 AH-ROBUST-MONO-ES-CLEF2006
Overall statistics for 100 queries : Priority 2
Total number of documents over all queries Query Construction AUTOMATIC
Retrieved 100,000 Source Language Spanish; Castilian
Relevant 5,008 Topic Fields title, description
Relevant retrieved 4,474 Pooled false
Geometric Mean Average Precision 0.2132 spanish mono standard
Binary Preference (BPREF) 0.5208
Interploated Recall (%) Precision Averages (%)
Ad−Hoc Robust Monolingual Spanish track − Interpolated Recall vs Average Precision
100%
0 70.06
dcudescsp12075
10 61.06 90%
20 56.90
80%
30 51.94
40 49.15 70%
50 45.16
Average Precision
60%
60 41.62
70 37.17 50%
80 31.08 40%
90 22.05
30%
100 12.54
Average precision (non-interpolated) for all 20%
relevant documents (averaged over queries)
42.14 10%
0%
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
Interpolated Recall
Mean Average Precision Ad−Hoc Robust Monolingual Spanish track − Box plot of the Topics of the Experiment
Maximum 1.0000
Minimum 0.0000
First Quartile 0.1684
Second Quartile 0.3390
Third Quartile 0.6772
Interquartile range 0.5088
Mean 0.4214
Standard Deviation 0.3021
Lower Outlier Threshold 0.0000
Upper Outlier Threshold 1.0000 0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Mean Average Precision
Mean With No Outliers 0.4214
Std With No Outliers 0.3021
Ad−Hoc Robust Monolingual Spanish track − Distribution of the Topics of the Experiment
10
Number of Topics of the Experiment
8
6
4
2
0
0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Mean Average Precision
Precision averages (%) for individual Ad−Hoc Robust Monolingual Spanish track − Comparison to Median Mean Average Precision by Topic (Topics 041 to 085)
queries 1
dcudescsp12075
0.5
041 21.19 086 32.07 131 32.46 166 0.41
Difference
0
042 57.51 087 26.97 132 60.59 167 50.62
−0.5
043 34.97 088 37.65 133 5.64 168 5.46
044 23.33 089 45.37 134 89.29 169 0.00 −1
041 042 043 044 045 046 047 048 049 060 061 062 063 064 065 066 067 068 069 080 081 082 083 084 085
Topic Identifier
045 67.32 090 12.57 135 32.84 170 94.29 Ad−Hoc Robust Monolingual Spanish track − Comparison to Median Mean Average Precision by Topic (Topics 086 to 130)
1
046 1.89 091 44.04 136 57.14 171 8.75 dcudescsp12075
047 16.43 092 19.20 137 21.81 172 13.22 0.5
Difference
048 17.55 093 29.20 138 20.68 173 59.66 0
049 18.18 094 77.58 139 9.77 174 90.30 −0.5
060 66.40 095 20.02 140 11.43 175 100.00 −1
086 087 088 089 090 091 092 093 094 095 096 097 098 099 110 111 112 113 114 115 116 117 118 119 130
061 0.00 096 9.02 141 89.23 176 96.16 Topic Identifier
Ad−Hoc Robust Monolingual Spanish track − Comparison to Median Mean Average Precision by Topic (Topics 131 to 165)
062 20.67 097 73.16 142 90.58 177 19.08 1
dcudescsp12075
063 79.39 098 63.46 143 44.93 178 37.62 0.5
Difference
064 100.00 099 38.92 144 14.74 179 64.83
0
065 17.96 110 64.65 145 63.83 190 4.67
−0.5
066 89.47 111 0.07 146 5.70 191 16.67
−1
067 26.01 112 31.61 147 20.18 192 88.57 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 160 161 162 163 164 165
Topic Identifier
068 11.80 113 28.57 148 13.72 193 8.08 Ad−Hoc Robust Monolingual Spanish track − Comparison to Median Mean Average Precision by Topic (Topics 166 to 200)
1
069 64.65 114 1.45 149 68.97 194 72.05 dcudescsp12075
0.5
080 44.09 115 16.03 160 27.78 195 0.00
Difference
081 76.99 116 24.28 161 71.00 196 70.88 0
082 98.23 117 23.75 162 89.07 197 65.23 −0.5
083 53.83 118 25.77 163 58.36 198 35.60 −1
166 167 168 169 170 171 172 173 174 175 176 177 178 179 190 191 192 193 194 195 196 197 198 199 200
084 17.01 119 70.83 164 7.05 199 83.60 Topic Identifier
085 73.48 130 68.12 165 59.57 200 79.47
157
dcu dcudescsp12075 AH-ROBUST-MONO-ES-CLEF2006
Docs Cutoff Levels Precision at DCL (%)
Ad−Hoc Robust Monolingual Spanish track − Retrieved documents vs Precision
100%
5 docs 47.80
dcudescsp12075
10 docs 43.60 90%
15 docs 39.53
80%
20 docs 37.90
30 docs 34.70 70%
100 docs 22.53
60%
200 docs 15.19
R−Precision
500 docs 8.04 50%
1000 docs 4.47 40%
R-Precision (precision after R document retrieved,
where R = Relevant retrieved) 30%
40.58
20%
10%
0%
5 10 15 20 30 100 200 500 1000
Retrieved Documents (logarithmic scale)
Exact R-Precision Ad−Hoc Robust Monolingual Spanish track − Box plot of the Topics of the Experiment
Maximum 1.0000
Minimum 0.0000
First Quartile 0.1576
Second Quartile 0.3985
Third Quartile 0.6632
Interquartile range 0.5056
Mean 0.4058
Standard Deviation 0.2920
Lower Outlier Threshold 0.0000
Upper Outlier Threshold 1.0000 0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Exact R−Precision
Mean With No Outliers 0.4058
Std With No Outliers 0.2920
Ad−Hoc Robust Monolingual Spanish track − Distribution of the Topics of the Experiment
20
Number of Topics of the Experiment
15
10
5
0
0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Exact R−Precision
Precision averages (%) for individual Ad−Hoc Robust Monolingual Spanish track − Comparison to Median Mean Exact R−Precision by Topic (Topics 041 to 085)
queries 1
dcudescsp12075
0.5
041 37.50 086 41.07 131 36.36 166 0.00
Difference
0
042 56.70 087 32.95 132 50.00 167 50.00
−0.5
043 38.89 088 28.57 133 5.26 168 0.00
044 22.50 089 56.25 134 75.00 169 0.00 −1
041 042 043 044 045 046 047 048 049 060 061 062 063 064 065 066 067 068 069 080 081 082 083 084 085
Topic Identifier
045 65.97 090 13.33 135 44.23 170 80.00 Ad−Hoc Robust Monolingual Spanish track − Comparison to Median Mean Exact R−Precision by Topic (Topics 086 to 130)
1
046 0.00 091 41.94 136 50.00 171 0.00 dcudescsp12075
047 22.22 092 18.60 137 20.00 172 7.69 0.5
Difference
048 22.92 093 28.57 138 0.00 173 50.00 0
049 31.58 094 81.82 139 4.60 174 81.82 −0.5
060 61.33 095 30.84 140 21.01 175 100.00 −1
086 087 088 089 090 091 092 093 094 095 096 097 098 099 110 111 112 113 114 115 116 117 118 119 130
061 0.00 096 2.15 141 84.62 176 89.61 Topic Identifier
Ad−Hoc Robust Monolingual Spanish track − Comparison to Median Mean Exact R−Precision by Topic (Topics 131 to 165)
062 13.33 097 64.00 142 82.14 177 21.74 1
dcudescsp12075
063 77.97 098 60.00 143 43.75 178 45.45 0.5
Difference
064 100.00 099 53.57 144 18.18 179 67.26
0
065 33.78 110 57.14 145 51.06 190 0.00
−0.5
066 82.83 111 0.00 146 0.00 191 0.00
−1
067 40.82 112 37.50 147 22.22 192 79.55 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 160 161 162 163 164 165
Topic Identifier
068 0.00 113 38.61 148 9.09 193 11.76 Ad−Hoc Robust Monolingual Spanish track − Comparison to Median Mean Exact R−Precision by Topic (Topics 166 to 200)
1
069 66.67 114 0.00 149 69.23 194 75.00 dcudescsp12075
0.5
080 53.06 115 21.05 160 0.00 195 0.00
Difference
081 73.56 116 28.57 161 80.00 196 66.67 0
082 94.74 117 28.57 162 80.62 197 58.75 −0.5
083 44.44 118 20.00 163 53.33 198 25.00 −1
166 167 168 169 170 171 172 173 174 175 176 177 178 179 190 191 192 193 194 195 196 197 198 199 200
084 12.50 119 67.62 164 9.72 199 80.00 Topic Identifier
085 72.90 130 61.54 165 44.44 200 72.34
158
hummingbird humES06Rtde AH-ROBUST-MONO-ES-CLEF2006
Overall statistics for 100 queries : Priority 2
Total number of documents over all queries Query Construction AUTOMATIC
Retrieved 100,000 Source Language Spanish; Castilian
Relevant 5,008 Topic Fields title, description
Relevant retrieved 4,471 Pooled false
Geometric Mean Average Precision 0.2361 same as humES06Rtd except expansion from blind
Binary Preference (BPREF) 0.5503 feedback also used
Interploated Recall (%) Precision Averages (%)
Ad−Hoc Robust Monolingual Spanish track − Interpolated Recall vs Average Precision
100%
0 71.80
humES06Rtde
10 63.77 90%
20 59.93
80%
30 55.86
40 53.02 70%
50 50.26
Average Precision
60%
60 44.63
70 41.01 50%
80 34.44 40%
90 24.71
30%
100 13.08
Average precision (non-interpolated) for all 20%
relevant documents (averaged over queries)
45.66 10%
0%
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
Interpolated Recall
Mean Average Precision Ad−Hoc Robust Monolingual Spanish track − Box plot of the Topics of the Experiment
Maximum 1.0000
Minimum 0.0000
First Quartile 0.1515
Second Quartile 0.4092
Third Quartile 0.7701
Interquartile range 0.6186
Mean 0.4566
Standard Deviation 0.3146
Lower Outlier Threshold 0.0000
Upper Outlier Threshold 1.0000 0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Mean Average Precision
Mean With No Outliers 0.4566
Std With No Outliers 0.3146
Ad−Hoc Robust Monolingual Spanish track − Distribution of the Topics of the Experiment
10
Number of Topics of the Experiment
8
6
4
2
0
0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Mean Average Precision
Precision averages (%) for individual Ad−Hoc Robust Monolingual Spanish track − Comparison to Median Mean Average Precision by Topic (Topics 041 to 085)
queries 1
humES06Rtde
0.5
041 66.18 086 30.12 131 34.51 166 1.55
Difference
0
042 66.25 087 76.50 132 64.31 167 51.29
−0.5
043 15.05 088 42.71 133 0.55 168 13.51
044 33.88 089 44.92 134 7.29 169 0.00 −1
041 042 043 044 045 046 047 048 049 060 061 062 063 064 065 066 067 068 069 080 081 082 083 084 085
Topic Identifier
045 63.35 090 3.99 135 70.35 170 86.00 Ad−Hoc Robust Monolingual Spanish track − Comparison to Median Mean Average Precision by Topic (Topics 086 to 130)
1
046 2.87 091 38.49 136 49.80 171 37.44 humES06Rtde
047 13.97 092 29.46 137 22.03 172 14.06 0.5
Difference
048 8.88 093 18.44 138 12.15 173 74.62 0
049 25.33 094 86.86 139 8.45 174 78.81 −0.5
060 74.75 095 26.49 140 6.01 175 100.00 −1
086 087 088 089 090 091 092 093 094 095 096 097 098 099 110 111 112 113 114 115 116 117 118 119 130
061 0.00 096 9.97 141 92.79 176 97.23 Topic Identifier
Ad−Hoc Robust Monolingual Spanish track − Comparison to Median Mean Average Precision by Topic (Topics 131 to 165)
062 25.30 097 73.42 142 92.56 177 31.61 1
humES06Rtde
063 85.87 098 82.33 143 50.86 178 48.33 0.5
Difference
064 100.00 099 26.70 144 57.02 179 80.06
0
065 18.62 110 69.68 145 72.51 190 3.19
−0.5
066 93.43 111 1.07 146 10.10 191 14.29
−1
067 35.70 112 26.51 147 14.29 192 81.65 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 160 161 162 163 164 165
Topic Identifier
068 8.77 113 33.17 148 15.26 193 18.83 Ad−Hoc Robust Monolingual Spanish track − Comparison to Median Mean Average Precision by Topic (Topics 166 to 200)
1
069 87.50 114 5.51 149 75.32 194 79.21 humES06Rtde
0.5
080 39.14 115 15.27 160 45.00 195 0.00
Difference
081 86.19 116 14.82 161 87.67 196 70.78 0
082 82.67 117 27.74 162 82.61 197 71.35 −0.5
083 55.98 118 44.96 163 60.67 198 23.94 −1
166 167 168 169 170 171 172 173 174 175 176 177 178 179 190 191 192 193 194 195 196 197 198 199 200
084 19.70 119 80.63 164 29.39 199 85.80 Topic Identifier
085 80.29 130 78.62 165 77.53 200 85.80
159
hummingbird humES06Rtde AH-ROBUST-MONO-ES-CLEF2006
Docs Cutoff Levels Precision at DCL (%)
Ad−Hoc Robust Monolingual Spanish track − Retrieved documents vs Precision
100%
5 docs 52.20
humES06Rtde
10 docs 48.50 90%
15 docs 44.53
80%
20 docs 41.85
30 docs 37.67 70%
100 docs 24.40
60%
200 docs 15.92
R−Precision
500 docs 8.11 50%
1000 docs 4.47 40%
R-Precision (precision after R document retrieved,
where R = Relevant retrieved) 30%
43.99
20%
10%
0%
5 10 15 20 30 100 200 500 1000
Retrieved Documents (logarithmic scale)
Exact R-Precision Ad−Hoc Robust Monolingual Spanish track − Box plot of the Topics of the Experiment
Maximum 1.0000
Minimum 0.0000
First Quartile 0.2026
Second Quartile 0.4286
Third Quartile 0.7199
Interquartile range 0.5173
Mean 0.4399
Standard Deviation 0.2991
Lower Outlier Threshold 0.0000
Upper Outlier Threshold 1.0000 0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Exact R−Precision
Mean With No Outliers 0.4399
Std With No Outliers 0.2991
Ad−Hoc Robust Monolingual Spanish track − Distribution of the Topics of the Experiment
20
Number of Topics of the Experiment
15
10
5
0
0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Exact R−Precision
Precision averages (%) for individual Ad−Hoc Robust Monolingual Spanish track − Comparison to Median Mean Exact R−Precision by Topic (Topics 041 to 085)
queries 1
humES06Rtde
0.5
041 56.25 086 30.36 131 37.27 166 0.00
Difference
0
042 60.15 087 71.59 132 53.85 167 50.00
−0.5
043 22.22 088 42.86 133 5.26 168 20.51
044 37.50 089 50.00 134 0.00 169 0.00 −1
041 042 043 044 045 046 047 048 049 060 061 062 063 064 065 066 067 068 069 080 081 082 083 084 085
Topic Identifier
045 61.26 090 6.67 135 67.95 170 80.00 Ad−Hoc Robust Monolingual Spanish track − Comparison to Median Mean Exact R−Precision by Topic (Topics 086 to 130)
1
046 0.00 091 37.63 136 50.00 171 25.00 humES06Rtde
047 18.52 092 37.21 137 30.00 172 15.38 0.5
Difference
048 11.46 093 35.71 138 0.00 173 50.00 0
049 36.84 094 86.36 139 1.15 174 81.82 −0.5
060 68.00 095 39.88 140 11.76 175 100.00 −1
086 087 088 089 090 091 092 093 094 095 096 097 098 099 110 111 112 113 114 115 116 117 118 119 130
061 0.00 096 3.23 141 92.31 176 90.91 Topic Identifier
Ad−Hoc Robust Monolingual Spanish track − Comparison to Median Mean Exact R−Precision by Topic (Topics 131 to 165)
062 20.00 097 76.00 142 85.71 177 26.09 1
humES06Rtde
063 77.97 098 80.00 143 52.08 178 52.27 0.5
Difference
064 100.00 099 28.57 144 72.73 179 75.22
0
065 32.43 110 57.14 145 68.09 190 0.00
−0.5
066 86.87 111 0.00 146 0.00 191 0.00
−1
067 42.86 112 25.00 147 11.11 192 72.73 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 160 161 162 163 164 165
Topic Identifier
068 0.00 113 34.18 148 9.09 193 23.53 Ad−Hoc Robust Monolingual Spanish track − Comparison to Median Mean Exact R−Precision by Topic (Topics 166 to 200)
1
069 83.33 114 0.00 149 69.23 194 75.00 humES06Rtde
0.5
080 44.08 115 21.05 160 50.00 195 0.00
Difference
081 79.31 116 0.00 161 80.00 196 66.67 0
082 76.32 117 35.71 162 73.64 197 65.83 −0.5
083 55.56 118 40.00 163 60.00 198 25.00 −1
166 167 168 169 170 171 172 173 174 175 176 177 178 179 190 191 192 193 194 195 196 197 198 199 200
084 25.00 119 72.38 164 38.89 199 84.71 Topic Identifier
085 75.70 130 69.23 165 66.67 200 78.72
160
hummingbird humES06Rtd AH-ROBUST-MONO-ES-CLEF2006
Overall statistics for 100 queries : Priority 1
Total number of documents over all queries Query Construction AUTOMATIC
Retrieved 100,000 Source Language Spanish; Castilian
Relevant 5,008 Topic Fields title, description
Relevant retrieved 4,467 Pooled false
Geometric Mean Average Precision 0.2215 plain SearchServer stemming run
Binary Preference (BPREF) 0.5126
Interploated Recall (%) Precision Averages (%)
Ad−Hoc Robust Monolingual Spanish track − Interpolated Recall vs Average Precision
100%
0 72.70
humES06Rtd
10 62.35 90%
20 57.34
80%
30 50.92
40 48.69 70%
50 45.59
Average Precision
60%
60 40.53
70 35.50 50%
80 29.91 40%
90 21.01
30%
100 11.36
Average precision (non-interpolated) for all 20%
relevant documents (averaged over queries)
42.00 10%
0%
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
Interpolated Recall
Mean Average Precision Ad−Hoc Robust Monolingual Spanish track − Box plot of the Topics of the Experiment
Maximum 1.0000
Minimum 0.0000
First Quartile 0.1433
Second Quartile 0.3790
Third Quartile 0.6940
Interquartile range 0.5506
Mean 0.4200
Standard Deviation 0.2966
Lower Outlier Threshold 0.0000
Upper Outlier Threshold 1.0000 0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Mean Average Precision
Mean With No Outliers 0.4200
Std With No Outliers 0.2966
Ad−Hoc Robust Monolingual Spanish track − Distribution of the Topics of the Experiment
15
Number of Topics of the Experiment
10
5
0
0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Mean Average Precision
Precision averages (%) for individual Ad−Hoc Robust Monolingual Spanish track − Comparison to Median Mean Average Precision by Topic (Topics 041 to 085)
queries 1
humES06Rtd
0.5
041 51.69 086 27.10 131 35.45 166 5.44
Difference
0
042 70.52 087 60.09 132 64.36 167 41.17
−0.5
043 20.61 088 49.56 133 0.57 168 11.23
044 29.17 089 44.25 134 14.24 169 0.00 −1
041 042 043 044 045 046 047 048 049 060 061 062 063 064 065 066 067 068 069 080 081 082 083 084 085
Topic Identifier
045 54.69 090 5.18 135 59.85 170 81.03 Ad−Hoc Robust Monolingual Spanish track − Comparison to Median Mean Average Precision by Topic (Topics 086 to 130)
1
046 4.36 091 34.10 136 41.19 171 25.33 humES06Rtd
047 10.43 092 24.05 137 19.05 172 7.29 0.5
Difference
048 10.12 093 17.44 138 9.59 173 63.25 0
049 22.83 094 75.11 139 11.99 174 82.69 −0.5
060 57.56 095 16.24 140 5.91 175 100.00 −1
086 087 088 089 090 091 092 093 094 095 096 097 098 099 110 111 112 113 114 115 116 117 118 119 130
061 0.00 096 19.50 141 90.70 176 95.41 Topic Identifier
Ad−Hoc Robust Monolingual Spanish track − Comparison to Median Mean Average Precision by Topic (Topics 131 to 165)
062 22.86 097 70.88 142 92.51 177 25.48 1
humES06Rtd
063 82.15 098 79.58 143 39.83 178 32.70 0.5
Difference
064 100.00 099 13.01 144 53.69 179 57.13
0
065 13.49 110 61.18 145 63.64 190 2.40
−0.5
066 94.15 111 0.68 146 11.44 191 14.29
−1
067 12.89 112 40.21 147 20.77 192 48.04 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 160 161 162 163 164 165
Topic Identifier
068 6.37 113 34.46 148 15.00 193 17.35 Ad−Hoc Robust Monolingual Spanish track − Comparison to Median Mean Average Precision by Topic (Topics 166 to 200)
1
069 88.60 114 9.08 149 68.85 194 84.33 humES06Rtd
0.5
080 43.46 115 14.21 160 41.67 195 0.00
Difference
081 82.80 116 14.38 161 81.00 196 71.47 0
082 81.50 117 19.93 162 71.15 197 69.94 −0.5
083 56.18 118 35.98 163 64.60 198 30.38 −1
166 167 168 169 170 171 172 173 174 175 176 177 178 179 190 191 192 193 194 195 196 197 198 199 200
084 20.80 119 71.93 164 19.51 199 82.75 Topic Identifier
085 76.96 130 73.95 165 68.57 200 59.15
161
hummingbird humES06Rtd AH-ROBUST-MONO-ES-CLEF2006
Docs Cutoff Levels Precision at DCL (%)
Ad−Hoc Robust Monolingual Spanish track − Retrieved documents vs Precision
100%
5 docs 49.80
humES06Rtd
10 docs 45.90 90%
15 docs 43.00
80%
20 docs 39.60
30 docs 34.77 70%
100 docs 22.68
60%
200 docs 15.25
R−Precision
500 docs 7.97 50%
1000 docs 4.47 40%
R-Precision (precision after R document retrieved,
where R = Relevant retrieved) 30%
41.41
20%
10%
0%
5 10 15 20 30 100 200 500 1000
Retrieved Documents (logarithmic scale)
Exact R-Precision Ad−Hoc Robust Monolingual Spanish track − Box plot of the Topics of the Experiment
Maximum 1.0000
Minimum 0.0000
First Quartile 0.1716
Second Quartile 0.4193
Third Quartile 0.6422
Interquartile range 0.4706
Mean 0.4141
Standard Deviation 0.2806
Lower Outlier Threshold 0.0000
Upper Outlier Threshold 1.0000 0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Exact R−Precision
Mean With No Outliers 0.4141
Std With No Outliers 0.2806
Ad−Hoc Robust Monolingual Spanish track − Distribution of the Topics of the Experiment
15
Number of Topics of the Experiment
10
5
0
0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Exact R−Precision
Precision averages (%) for individual Ad−Hoc Robust Monolingual Spanish track − Comparison to Median Mean Exact R−Precision by Topic (Topics 041 to 085)
queries 1
humES06Rtd
0.5
041 43.75 086 28.57 131 41.82 166 0.00
Difference
0
042 61.69 087 61.36 132 53.85 167 40.00
−0.5
043 22.22 088 42.86 133 0.00 168 15.38
044 30.00 089 50.00 134 0.00 169 0.00 −1
041 042 043 044 045 046 047 048 049 060 061 062 063 064 065 066 067 068 069 080 081 082 083 084 085
Topic Identifier
045 56.54 090 6.67 135 64.10 170 60.00 Ad−Hoc Robust Monolingual Spanish track − Comparison to Median Mean Exact R−Precision by Topic (Topics 086 to 130)
1
046 6.67 091 37.63 136 50.00 171 25.00 humES06Rtd
047 18.52 092 29.07 137 30.00 172 7.69 0.5
Difference
048 13.54 093 35.71 138 0.00 173 66.67 0
049 31.58 094 81.82 139 14.37 174 81.82 −0.5
060 56.00 095 28.04 140 11.76 175 100.00 −1
086 087 088 089 090 091 092 093 094 095 096 097 098 099 110 111 112 113 114 115 116 117 118 119 130
061 0.00 096 26.88 141 84.62 176 85.71 Topic Identifier
Ad−Hoc Robust Monolingual Spanish track − Comparison to Median Mean Exact R−Precision by Topic (Topics 131 to 165)
062 16.67 097 76.00 142 82.14 177 26.09 1
humES06Rtd
063 71.19 098 60.00 143 43.06 178 42.05 0.5
Difference
064 100.00 099 17.86 144 68.18 179 57.52
0
065 27.03 110 57.14 145 61.70 190 0.00
−0.5
066 85.86 111 0.00 146 0.00 191 0.00
−1
067 16.33 112 50.00 147 22.22 192 50.00 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 160 161 162 163 164 165
Topic Identifier
068 0.00 113 38.61 148 9.09 193 17.65 Ad−Hoc Robust Monolingual Spanish track − Comparison to Median Mean Exact R−Precision by Topic (Topics 166 to 200)
1
069 83.33 114 4.44 149 61.54 194 87.50 humES06Rtd
0.5
080 48.57 115 26.32 160 50.00 195 0.00
Difference
081 78.16 116 14.29 161 80.00 196 66.67 0
082 68.42 117 28.57 162 64.34 197 65.42 −0.5
083 55.56 118 20.00 163 66.67 198 12.50 −1
166 167 168 169 170 171 172 173 174 175 176 177 178 179 190 191 192 193 194 195 196 197 198 199 200
084 25.00 119 67.62 164 29.17 199 78.82 Topic Identifier
085 78.50 130 61.54 165 66.67 200 53.19
162
reina reinaEStdtest AH-ROBUST-MONO-ES-CLEF2006
Overall statistics for 100 queries : Priority 1
Total number of documents over all queries Query Construction AUTOMATIC
Retrieved 100,000 Source Language Spanish; Castilian
Relevant 5,008 Topic Fields title, description
Relevant retrieved 4,442 Pooled false
Geometric Mean Average Precision 0.2265 local expansion with term windows, title+desc
Binary Preference (BPREF) 0.5367 fields
Interploated Recall (%) Precision Averages (%)
Ad−Hoc Robust Monolingual Spanish track − Interpolated Recall vs Average Precision
100%
0 76.36
reinaEStdtest
10 65.98 90%
20 58.66
80%
30 55.57
40 52.15 70%
50 48.48
Average Precision
60%
60 41.28
70 36.40 50%
80 30.21 40%
90 21.00
30%
100 11.58
Average precision (non-interpolated) for all 20%
relevant documents (averaged over queries)
44.01 10%
0%
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
Interpolated Recall
Mean Average Precision Ad−Hoc Robust Monolingual Spanish track − Box plot of the Topics of the Experiment
Maximum 1.0000
Minimum 0.0000
First Quartile 0.1571
Second Quartile 0.4076
Third Quartile 0.7135
Interquartile range 0.5564
Mean 0.4401
Standard Deviation 0.3044
Lower Outlier Threshold 0.0000
Upper Outlier Threshold 1.0000 0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Mean Average Precision
Mean With No Outliers 0.4401
Std With No Outliers 0.3044
Ad−Hoc Robust Monolingual Spanish track − Distribution of the Topics of the Experiment
10
Number of Topics of the Experiment
8
6
4
2
0
0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Mean Average Precision
Precision averages (%) for individual Ad−Hoc Robust Monolingual Spanish track − Comparison to Median Mean Average Precision by Topic (Topics 041 to 085)
queries 1
reinaEStdtest
0.5
041 72.00 086 28.26 131 34.55 166 2.58
Difference
0
042 47.45 087 61.58 132 55.53 167 58.88
−0.5
043 21.71 088 34.99 133 0.70 168 10.70
044 19.78 089 51.90 134 60.62 169 0.00 −1
041 042 043 044 045 046 047 048 049 060 061 062 063 064 065 066 067 068 069 080 081 082 083 084 085
Topic Identifier
045 62.55 090 5.52 135 70.51 170 84.33 Ad−Hoc Robust Monolingual Spanish track − Comparison to Median Mean Average Precision by Topic (Topics 086 to 130)
1
046 11.11 091 40.53 136 42.12 171 40.08 reinaEStdtest
047 12.86 092 27.06 137 20.89 172 4.07 0.5
Difference
048 4.29 093 34.91 138 8.72 173 67.20 0
049 10.12 094 89.04 139 7.43 174 84.69 −0.5
060 39.88 095 23.95 140 10.52 175 100.00 −1
086 087 088 089 090 091 092 093 094 095 096 097 098 099 110 111 112 113 114 115 116 117 118 119 130
061 0.00 096 10.63 141 97.57 176 98.62 Topic Identifier
Ad−Hoc Robust Monolingual Spanish track − Comparison to Median Mean Average Precision by Topic (Topics 131 to 165)
062 27.27 097 71.17 142 88.47 177 26.13 1
reinaEStdtest
063 85.75 098 90.00 143 53.61 178 40.98 0.5
Difference
064 100.00 099 0.58 144 58.02 179 79.24
0
065 16.43 110 58.23 145 65.88 190 2.50
−0.5
066 88.53 111 2.07 146 14.84 191 50.00
−1
067 14.99 112 20.57 147 17.15 192 61.83 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 160 161 162 163 164 165
Topic Identifier
068 6.74 113 18.45 148 14.03 193 21.00 Ad−Hoc Robust Monolingual Spanish track − Comparison to Median Mean Average Precision by Topic (Topics 166 to 200)
1
069 85.42 114 6.85 149 67.19 194 72.24 reinaEStdtest
0.5
080 40.48 115 22.80 160 54.55 195 0.00
Difference
081 76.27 116 11.02 161 100.00 196 73.50 0
082 39.25 117 20.37 162 81.07 197 69.40 −0.5
083 45.82 118 42.54 163 58.42 198 43.22 −1
166 167 168 169 170 171 172 173 174 175 176 177 178 179 190 191 192 193 194 195 196 197 198 199 200
084 31.20 119 75.79 164 35.73 199 83.00 Topic Identifier
085 80.22 130 71.84 165 71.53 200 76.56
163
reina reinaEStdtest AH-ROBUST-MONO-ES-CLEF2006
Docs Cutoff Levels Precision at DCL (%)
Ad−Hoc Robust Monolingual Spanish track − Retrieved documents vs Precision
100%
5 docs 52.60
reinaEStdtest
10 docs 46.60 90%
15 docs 42.60
80%
20 docs 39.80
30 docs 35.57 70%
100 docs 22.57
60%
200 docs 15.16
R−Precision
500 docs 7.90 50%
1000 docs 4.44 40%
R-Precision (precision after R document retrieved,
where R = Relevant retrieved) 30%
42.96
20%
10%
0%
5 10 15 20 30 100 200 500 1000
Retrieved Documents (logarithmic scale)
Exact R-Precision Ad−Hoc Robust Monolingual Spanish track − Box plot of the Topics of the Experiment
Maximum 1.0000
Minimum 0.0000
First Quartile 0.1972
Second Quartile 0.4248
Third Quartile 0.6364
Interquartile range 0.4391
Mean 0.4296
Standard Deviation 0.2889
Lower Outlier Threshold 0.0000
Upper Outlier Threshold 1.0000 0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Exact R−Precision
Mean With No Outliers 0.4296
Std With No Outliers 0.2889
Ad−Hoc Robust Monolingual Spanish track − Distribution of the Topics of the Experiment
15
Number of Topics of the Experiment
10
5
0
0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Exact R−Precision
Precision averages (%) for individual Ad−Hoc Robust Monolingual Spanish track − Comparison to Median Mean Exact R−Precision by Topic (Topics 041 to 085)
queries 1
reinaEStdtest
0.5
041 62.50 086 28.57 131 40.00 166 0.00
Difference
0
042 49.04 087 54.55 132 53.85 167 40.00
−0.5
043 33.33 088 42.86 133 0.00 168 15.38
044 22.50 089 62.50 134 50.00 169 0.00 −1
041 042 043 044 045 046 047 048 049 060 061 062 063 064 065 066 067 068 069 080 081 082 083 084 085
Topic Identifier
045 63.35 090 6.67 135 63.46 170 80.00 Ad−Hoc Robust Monolingual Spanish track − Comparison to Median Mean Exact R−Precision by Topic (Topics 086 to 130)
1
046 13.33 091 36.56 136 37.50 171 50.00 reinaEStdtest
047 19.44 092 22.09 137 20.00 172 7.69 0.5
Difference
048 9.38 093 35.71 138 0.00 173 66.67 0
049 21.05 094 81.82 139 2.30 174 81.82 −0.5
060 46.67 095 33.33 140 15.97 175 100.00 −1
086 087 088 089 090 091 092 093 094 095 096 097 098 099 110 111 112 113 114 115 116 117 118 119 130
061 0.00 096 9.68 141 92.31 176 96.10 Topic Identifier
Ad−Hoc Robust Monolingual Spanish track − Comparison to Median Mean Exact R−Precision by Topic (Topics 131 to 165)
062 36.67 097 72.00 142 82.14 177 30.43 1
reinaEStdtest
063 81.36 098 80.00 143 55.56 178 52.27 0.5
Difference
064 100.00 099 3.57 144 63.64 179 76.11
0
065 28.38 110 57.14 145 61.70 190 0.00
−0.5
066 78.79 111 0.00 146 0.00 191 0.00
−1
067 16.33 112 25.00 147 22.22 192 63.64 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 160 161 162 163 164 165
Topic Identifier
068 11.11 113 22.78 148 9.09 193 20.59 Ad−Hoc Robust Monolingual Spanish track − Comparison to Median Mean Exact R−Precision by Topic (Topics 166 to 200)
1
069 83.33 114 6.67 149 53.85 194 75.00 reinaEStdtest
0.5
080 45.31 115 26.32 160 50.00 195 0.00
Difference
081 75.86 116 0.00 161 100.00 196 73.33 0
082 42.11 117 23.21 162 77.52 197 62.08 −0.5
083 44.44 118 40.00 163 60.00 198 50.00 −1
166 167 168 169 170 171 172 173 174 175 176 177 178 179 190 191 192 193 194 195 196 197 198 199 200
084 37.50 119 66.67 164 41.67 199 80.00 Topic Identifier
085 72.90 130 61.54 165 66.67 200 61.70
164
reina reinaESttest AH-ROBUST-MONO-ES-CLEF2006
Overall statistics for 100 queries : Priority 2
Total number of documents over all queries Query Construction AUTOMATIC
Retrieved 100,000 Source Language Spanish; Castilian
Relevant 5,008 Topic Fields title
Relevant retrieved 4,155 Pooled false
Geometric Mean Average Precision 0.1782 local expansion with term windows, only title field
Binary Preference (BPREF) 0.4962
Interploated Recall (%) Precision Averages (%)
Ad−Hoc Robust Monolingual Spanish track − Interpolated Recall vs Average Precision
100%
0 69.11
reinaESttest
10 59.76 90%
20 54.19
80%
30 50.27
40 45.56 70%
50 41.97
Average Precision
60%
60 36.43
70 30.65 50%
80 26.48 40%
90 18.44
30%
100 9.88
Average precision (non-interpolated) for all 20%
relevant documents (averaged over queries)
38.93 10%
0%
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
Interpolated Recall
Mean Average Precision Ad−Hoc Robust Monolingual Spanish track − Box plot of the Topics of the Experiment
Maximum 1.0000
Minimum 0.0000
First Quartile 0.1041
Second Quartile 0.3180
Third Quartile 0.6373
Interquartile range 0.5332
Mean 0.3893
Standard Deviation 0.3037
Lower Outlier Threshold 0.0000
Upper Outlier Threshold 1.0000 0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Mean Average Precision
Mean With No Outliers 0.3893
Std With No Outliers 0.3037
Ad−Hoc Robust Monolingual Spanish track − Distribution of the Topics of the Experiment
15
Number of Topics of the Experiment
10
5
0
0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Mean Average Precision
Precision averages (%) for individual Ad−Hoc Robust Monolingual Spanish track − Comparison to Median Mean Average Precision by Topic (Topics 041 to 085)
queries 1
reinaESttest
0.5
041 71.03 086 30.55 131 31.37 166 4.67
Difference
0
042 22.14 087 58.22 132 56.21 167 46.21
−0.5
043 1.42 088 31.22 133 0.35 168 10.24
044 13.62 089 68.24 134 21.54 169 0.00 −1
041 042 043 044 045 046 047 048 049 060 061 062 063 064 065 066 067 068 069 080 081 082 083 084 085
Topic Identifier
045 69.00 090 2.04 135 75.41 170 47.54 Ad−Hoc Robust Monolingual Spanish track − Comparison to Median Mean Average Precision by Topic (Topics 086 to 130)
1
046 1.44 091 37.66 136 42.24 171 19.90 reinaESttest
047 14.86 092 26.67 137 25.26 172 6.96 0.5
Difference
048 1.04 093 26.20 138 4.44 173 70.04 0
049 10.06 094 88.54 139 7.03 174 52.60 −0.5
060 40.10 095 2.63 140 10.34 175 100.00 −1
086 087 088 089 090 091 092 093 094 095 096 097 098 099 110 111 112 113 114 115 116 117 118 119 130
061 0.00 096 7.54 141 93.62 176 98.46 Topic Identifier
Ad−Hoc Robust Monolingual Spanish track − Comparison to Median Mean Average Precision by Topic (Topics 131 to 165)
062 35.64 097 76.72 142 89.51 177 34.43 1
reinaESttest
063 86.41 098 80.54 143 50.41 178 26.96 0.5
Difference
064 100.00 099 10.55 144 61.69 179 61.30
0
065 16.06 110 9.62 145 64.62 190 3.56
−0.5
066 87.44 111 2.21 146 54.55 191 25.00
−1
067 17.83 112 11.76 147 1.39 192 10.48 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 160 161 162 163 164 165
Topic Identifier
068 24.73 113 8.87 148 32.22 193 22.32 Ad−Hoc Robust Monolingual Spanish track − Comparison to Median Mean Average Precision by Topic (Topics 166 to 200)
1
069 81.45 114 5.56 149 53.47 194 0.42 reinaESttest
0.5
080 45.62 115 18.69 160 60.00 195 0.00
Difference
081 63.15 116 14.48 161 100.00 196 62.79 0
082 49.65 117 7.72 162 75.18 197 44.07 −0.5
083 45.34 118 34.70 163 40.36 198 19.80 −1
166 167 168 169 170 171 172 173 174 175 176 177 178 179 190 191 192 193 194 195 196 197 198 199 200
084 31.15 119 64.69 164 27.79 199 84.95 Topic Identifier
085 83.22 130 76.75 165 64.31 200 81.85
165
reina reinaESttest AH-ROBUST-MONO-ES-CLEF2006
Docs Cutoff Levels Precision at DCL (%)
Ad−Hoc Robust Monolingual Spanish track − Retrieved documents vs Precision
100%
5 docs 47.60
reinaESttest
10 docs 42.00 90%
15 docs 37.60
80%
20 docs 35.30
30 docs 32.10 70%
100 docs 20.48
60%
200 docs 13.78
R−Precision
500 docs 7.37 50%
1000 docs 4.16 40%
R-Precision (precision after R document retrieved,
where R = Relevant retrieved) 30%
38.82
20%
10%
0%
5 10 15 20 30 100 200 500 1000
Retrieved Documents (logarithmic scale)
Exact R-Precision Ad−Hoc Robust Monolingual Spanish track − Box plot of the Topics of the Experiment
Maximum 1.0000
Minimum 0.0000
First Quartile 0.1429
Second Quartile 0.3591
Third Quartile 0.6011
Interquartile range 0.4583
Mean 0.3882
Standard Deviation 0.2840
Lower Outlier Threshold 0.0000
Upper Outlier Threshold 1.0000 0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Exact R−Precision
Mean With No Outliers 0.3882
Std With No Outliers 0.2840
Ad−Hoc Robust Monolingual Spanish track − Distribution of the Topics of the Experiment
15
Number of Topics of the Experiment
10
5
0
0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Exact R−Precision
Precision averages (%) for individual Ad−Hoc Robust Monolingual Spanish track − Comparison to Median Mean Exact R−Precision by Topic (Topics 041 to 085)
queries 1
reinaESttest
0.5
041 62.50 086 32.14 131 35.45 166 14.29
Difference
0
042 32.95 087 60.23 132 50.00 167 30.00
−0.5
043 5.56 088 28.57 133 0.00 168 12.82
044 12.50 089 56.25 134 25.00 169 0.00 −1
041 042 043 044 045 046 047 048 049 060 061 062 063 064 065 066 067 068 069 080 081 082 083 084 085
Topic Identifier
045 69.11 090 6.67 135 69.23 170 60.00 Ad−Hoc Robust Monolingual Spanish track − Comparison to Median Mean Exact R−Precision by Topic (Topics 086 to 130)
1
046 0.00 091 32.26 136 37.50 171 25.00 reinaESttest
047 25.00 092 26.74 137 30.00 172 15.38 0.5
Difference
048 1.04 093 42.86 138 0.00 173 66.67 0
049 15.79 094 86.36 139 3.45 174 45.45 −0.5
060 42.67 095 9.03 140 15.13 175 100.00 −1
086 087 088 089 090 091 092 093 094 095 096 097 098 099 110 111 112 113 114 115 116 117 118 119 130
061 0.00 096 15.05 141 84.62 176 94.81 Topic Identifier
Ad−Hoc Robust Monolingual Spanish track − Comparison to Median Mean Exact R−Precision by Topic (Topics 131 to 165)
062 43.33 097 72.00 142 82.14 177 34.78 1
reinaESttest
063 77.97 098 80.00 143 54.86 178 36.36 0.5
Difference
064 100.00 099 17.86 144 63.64 179 62.83
0
065 32.43 110 14.29 145 53.19 190 0.00
−0.5
066 77.78 111 0.00 146 50.00 191 0.00
−1
067 26.53 112 0.00 147 0.00 192 15.91 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 160 161 162 163 164 165
Topic Identifier
068 22.22 113 11.39 148 36.36 193 29.41 Ad−Hoc Robust Monolingual Spanish track − Comparison to Median Mean Exact R−Precision by Topic (Topics 166 to 200)
1
069 83.33 114 2.22 149 53.85 194 0.00 reinaESttest
0.5
080 50.20 115 21.05 160 50.00 195 0.00
Difference
081 65.52 116 14.29 161 100.00 196 60.00 0
082 55.26 117 12.50 162 75.19 197 49.58 −0.5
083 44.44 118 40.00 163 30.00 198 12.50 −1
166 167 168 169 170 171 172 173 174 175 176 177 178 179 190 191 192 193 194 195 196 197 198 199 200
084 37.50 119 56.19 164 38.89 199 81.18 Topic Identifier
085 77.57 130 69.23 165 55.56 200 72.34
166
colesir CoLesIRfrTst AH-ROBUST-MONO-FR-CLEF2006
Overall statistics for 100 queries : Priority 1
Total number of documents over all queries Query Construction AUTOMATIC
Retrieved 100,000 Source Language French
Relevant 2,190 Topic Fields title, description
Relevant retrieved 2,025 Pooled false
Geometric Mean Average Precision 0.1191 robust monolingual FR - TEST; automatic; TD
Binary Preference (BPREF) 0.4219
Interploated Recall (%) Precision Averages (%)
Ad−Hoc Robust Monolingual French track − Interpolated Recall vs Average Precision
100%
0 68.41
CoLesIRfrTst
10 60.81 90%
20 53.03
80%
30 47.19
40 43.86 70%
50 40.99
Average Precision
60%
60 35.97
70 32.68 50%
80 28.77 40%
90 23.71
30%
100 18.86
Average precision (non-interpolated) for all 20%
relevant documents (averaged over queries)
39.51 10%
0%
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
Interpolated Recall
Mean Average Precision Ad−Hoc Robust Monolingual French track − Box plot of the Topics of the Experiment
Maximum 1.0000
Minimum 0.0000
First Quartile 0.1532
Second Quartile 0.3435
Third Quartile 0.5934
Interquartile range 0.4402
Mean 0.3951
Standard Deviation 0.3025
Lower Outlier Threshold 0.0000
Upper Outlier Threshold 1.0000 0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Mean Average Precision
Mean With No Outliers 0.3951
Std With No Outliers 0.3025
Ad−Hoc Robust Monolingual French track − Distribution of the Topics of the Experiment
15
Number of Topics of the Experiment
10
5
0
0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Mean Average Precision
Precision averages (%) for individual Ad−Hoc Robust Monolingual French track − Comparison to Median Mean Average Precision by Topic (Topics 041 to 085)
queries 1
CoLesIRfrTst
0.5
041 64.29 086 15.28 131 28.31 166 0.00
Difference
0
042 49.67 087 34.42 132 49.93 167 66.67
−0.5
043 50.00 088 59.56 133 9.93 168 14.82
044 28.12 089 45.93 134 100.00 169 0.00 −1
041 042 043 044 045 046 047 048 049 060 061 062 063 064 065 066 067 068 069 080 081 082 083 084 085
Topic Identifier
045 27.35 090 8.96 135 32.98 170 33.33 Ad−Hoc Robust Monolingual French track − Comparison to Median Mean Average Precision by Topic (Topics 086 to 130)
1
046 5.49 091 21.59 136 100.00 171 36.26 CoLesIRfrTst
047 23.77 092 26.25 137 0.06 172 0.00 0.5
Difference
048 1.47 093 96.67 138 10.70 173 100.00 0
049 15.36 094 65.55 139 12.06 174 96.83 −0.5
060 15.42 095 24.49 140 60.04 175 50.00 −1
086 087 088 089 090 091 092 093 094 095 096 097 098 099 110 111 112 113 114 115 116 117 118 119 130
061 16.23 096 39.14 141 100.00 176 91.99 Topic Identifier
Ad−Hoc Robust Monolingual French track − Comparison to Median Mean Average Precision by Topic (Topics 131 to 165)
062 39.54 097 21.46 142 98.86 177 16.77 1
CoLesIRfrTst
063 51.55 098 82.23 143 32.34 178 38.89 0.5
Difference
064 0.00 099 17.79 144 100.00 179 25.65
0
065 16.94 110 58.22 145 13.07 190 32.73
−0.5
066 90.09 111 0.74 146 0.00 191 0.00
−1
067 39.36 112 38.35 147 48.75 192 63.03 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 160 161 162 163 164 165
Topic Identifier
068 32.75 113 14.04 148 2.07 193 48.04 Ad−Hoc Robust Monolingual French track − Comparison to Median Mean Average Precision by Topic (Topics 166 to 200)
1
069 38.57 114 42.76 149 87.62 194 0.00 CoLesIRfrTst
0.5
080 5.61 115 54.67 160 0.00 195 25.16
Difference
081 60.02 116 34.29 161 0.00 196 100.00 0
082 61.88 117 28.38 162 78.63 197 83.58 −0.5
083 58.33 118 7.36 163 44.10 198 6.73 −1
166 167 168 169 170 171 172 173 174 175 176 177 178 179 190 191 192 193 194 195 196 197 198 199 200
084 50.00 119 76.58 164 40.23 199 69.31 Topic Identifier
085 18.01 130 59.12 165 45.00 200 22.81
167
colesir CoLesIRfrTst AH-ROBUST-MONO-FR-CLEF2006
Docs Cutoff Levels Precision at DCL (%)
Ad−Hoc Robust Monolingual French track − Retrieved documents vs Precision
100%
5 docs 40.80
CoLesIRfrTst
10 docs 35.20 90%
15 docs 30.47
80%
20 docs 27.80
30 docs 23.87 70%
100 docs 12.95
60%
200 docs 8.00
R−Precision
500 docs 3.81 50%
1000 docs 2.03 40%
R-Precision (precision after R document retrieved,
where R = Relevant retrieved) 30%
37.56
20%
10%
0%
5 10 15 20 30 100 200 500 1000
Retrieved Documents (logarithmic scale)
Exact R-Precision Ad−Hoc Robust Monolingual French track − Box plot of the Topics of the Experiment
Maximum 1.0000
Minimum 0.0000
First Quartile 0.1484
Second Quartile 0.3488
Third Quartile 0.5294
Interquartile range 0.3811
Mean 0.3756
Standard Deviation 0.2903
Lower Outlier Threshold 0.0000
Upper Outlier Threshold 1.0000 0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Exact R−Precision
Mean With No Outliers 0.3756
Std With No Outliers 0.2903
Ad−Hoc Robust Monolingual French track − Distribution of the Topics of the Experiment
20
Number of Topics of the Experiment
15
10
5
0
0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Exact R−Precision
Precision averages (%) for individual Ad−Hoc Robust Monolingual French track − Comparison to Median Mean Exact R−Precision by Topic (Topics 041 to 085)
queries 1
CoLesIRfrTst
0.5
041 50.00 086 16.67 131 36.59 166 0.00
Difference
0
042 51.32 087 42.11 132 25.00 167 50.00
−0.5
043 0.00 088 51.61 133 21.88 168 23.08
044 33.33 089 52.17 134 100.00 169 0.00 −1
041 042 043 044 045 046 047 048 049 060 061 062 063 064 065 066 067 068 069 080 081 082 083 084 085
Topic Identifier
045 34.04 090 5.88 135 42.31 170 0.00 Ad−Hoc Robust Monolingual French track − Comparison to Median Mean Exact R−Precision by Topic (Topics 086 to 130)
1
046 14.29 091 38.46 136 100.00 171 33.33 CoLesIRfrTst
047 23.08 092 27.08 137 0.00 172 0.00 0.5
Difference
048 0.00 093 80.00 138 17.65 173 100.00 0
049 16.67 094 50.00 139 11.11 174 85.71 −0.5
060 18.89 095 33.90 140 55.47 175 0.00 −1
086 087 088 089 090 091 092 093 094 095 096 097 098 099 110 111 112 113 114 115 116 117 118 119 130
061 8.33 096 35.71 141 100.00 176 84.21 Topic Identifier
Ad−Hoc Robust Monolingual French track − Comparison to Median Mean Exact R−Precision by Topic (Topics 131 to 165)
062 37.50 097 14.29 142 92.31 177 14.29 1
CoLesIRfrTst
063 52.94 098 77.78 143 32.14 178 66.67 0.5
Difference
064 0.00 099 29.41 144 100.00 179 25.00
0
065 15.38 110 61.54 145 21.43 190 41.18
−0.5
066 86.27 111 0.00 146 0.00 191 0.00
−1
067 42.11 112 40.00 147 50.00 192 57.89 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 160 161 162 163 164 165
Topic Identifier
068 40.00 113 22.22 148 0.00 193 46.34 Ad−Hoc Robust Monolingual French track − Comparison to Median Mean Exact R−Precision by Topic (Topics 166 to 200)
1
069 43.75 114 43.33 149 60.00 194 0.00 CoLesIRfrTst
0.5
080 9.68 115 60.00 160 0.00 195 21.05
Difference
081 52.94 116 30.00 161 0.00 196 100.00 0
082 56.25 117 32.14 162 76.00 197 78.63 −0.5
083 50.00 118 16.67 163 45.45 198 0.00 −1
166 167 168 169 170 171 172 173 174 175 176 177 178 179 190 191 192 193 194 195 196 197 198 199 200
084 0.00 119 75.00 164 46.07 199 57.58 Topic Identifier
085 20.00 130 61.54 165 50.00 200 33.33
168
daedalus frFSfrR2S AH-ROBUST-MONO-FR-CLEF2006
Overall statistics for 100 queries : Priority 4
Total number of documents over all queries Query Construction AUTOMATIC
Retrieved 100,000 Source Language French
Relevant 2,190 Topic Fields title, description
Relevant retrieved 2,036 Pooled false
Geometric Mean Average Precision 0.1187 Mandatory TD run
Binary Preference (BPREF) 0.4151
Interploated Recall (%) Precision Averages (%)
Ad−Hoc Robust Monolingual French track − Interpolated Recall vs Average Precision
100%
0 67.72
frFSfrR2S
10 59.82 90%
20 53.27
80%
30 48.76
40 42.19 70%
50 39.31
Average Precision
60%
60 34.78
70 30.39 50%
80 27.20 40%
90 22.13
30%
100 18.24
Average precision (non-interpolated) for all 20%
relevant documents (averaged over queries)
38.49 10%
0%
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
Interpolated Recall
Mean Average Precision Ad−Hoc Robust Monolingual French track − Box plot of the Topics of the Experiment
Maximum 1.0000
Minimum 0.0000
First Quartile 0.1719
Second Quartile 0.3409
Third Quartile 0.5764
Interquartile range 0.4045
Mean 0.3849
Standard Deviation 0.2993
Lower Outlier Threshold 0.0000
Upper Outlier Threshold 1.0000 0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Mean Average Precision
Mean With No Outliers 0.3849
Std With No Outliers 0.2993
Ad−Hoc Robust Monolingual French track − Distribution of the Topics of the Experiment
20
Number of Topics of the Experiment
15
10
5
0
0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Mean Average Precision
Precision averages (%) for individual Ad−Hoc Robust Monolingual French track − Comparison to Median Mean Average Precision by Topic (Topics 041 to 085)
queries 1
frFSfrR2S
0.5
041 100.00 086 17.79 131 33.68 166 0.00
Difference
0
042 35.42 087 41.28 132 48.73 167 64.29
−0.5
043 100.00 088 65.54 133 18.55 168 18.34
044 27.57 089 48.58 134 91.67 169 0.00 −1
041 042 043 044 045 046 047 048 049 060 061 062 063 064 065 066 067 068 069 080 081 082 083 084 085
Topic Identifier
045 24.09 090 10.61 135 32.95 170 0.57 Ad−Hoc Robust Monolingual French track − Comparison to Median Mean Average Precision by Topic (Topics 086 to 130)
1
046 6.10 091 23.08 136 58.33 171 36.74 frFSfrR2S
047 19.51 092 18.75 137 14.29 172 0.00 0.5
Difference
048 0.52 093 96.67 138 26.39 173 100.00 0
049 25.04 094 56.46 139 12.06 174 100.00 −0.5
060 31.92 095 29.35 140 56.94 175 100.00 −1
086 087 088 089 090 091 092 093 094 095 096 097 098 099 110 111 112 113 114 115 116 117 118 119 130
061 4.37 096 30.02 141 100.00 176 95.01 Topic Identifier
Ad−Hoc Robust Monolingual French track − Comparison to Median Mean Average Precision by Topic (Topics 131 to 165)
062 64.83 097 19.69 142 60.68 177 56.19 1
frFSfrR2S
063 34.92 098 62.64 143 37.13 178 39.39 0.5
Difference
064 0.00 099 24.64 144 29.39 179 32.03
0
065 20.00 110 59.19 145 34.15 190 39.64
−0.5
066 92.16 111 11.90 146 0.00 191 0.00
−1
067 42.21 112 40.40 147 42.71 192 50.55 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 160 161 162 163 164 165
Topic Identifier
068 34.03 113 4.57 148 2.88 193 4.96 Ad−Hoc Robust Monolingual French track − Comparison to Median Mean Average Precision by Topic (Topics 166 to 200)
1
069 34.93 114 36.41 149 46.93 194 0.00 frFSfrR2S
0.5
080 16.34 115 68.85 160 0.00 195 26.52
Difference
081 60.83 116 35.48 161 0.00 196 100.00 0
082 16.60 117 22.19 162 8.51 197 89.46 −0.5
083 41.67 118 1.66 163 35.71 198 37.18 −1
166 167 168 169 170 171 172 173 174 175 176 177 178 179 190 191 192 193 194 195 196 197 198 199 200
084 100.00 119 76.28 164 38.86 199 70.93 Topic Identifier
085 29.14 130 58.36 165 26.67 200 7.68
169
daedalus frFSfrR2S AH-ROBUST-MONO-FR-CLEF2006
Docs Cutoff Levels Precision at DCL (%)
Ad−Hoc Robust Monolingual French track − Retrieved documents vs Precision
100%
5 docs 39.40
frFSfrR2S
10 docs 33.40 90%
15 docs 29.13
80%
20 docs 26.80
30 docs 23.13 70%
100 docs 12.76
60%
200 docs 8.19
R−Precision
500 docs 3.84 50%
1000 docs 2.04 40%
R-Precision (precision after R document retrieved,
where R = Relevant retrieved) 30%
37.59
20%
10%
0%
5 10 15 20 30 100 200 500 1000
Retrieved Documents (logarithmic scale)
Exact R-Precision Ad−Hoc Robust Monolingual French track − Box plot of the Topics of the Experiment
Maximum 1.0000
Minimum 0.0000
First Quartile 0.1716
Second Quartile 0.3333
Third Quartile 0.5520
Interquartile range 0.3804
Mean 0.3759
Standard Deviation 0.2918
Lower Outlier Threshold 0.0000
Upper Outlier Threshold 1.0000 0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Exact R−Precision
Mean With No Outliers 0.3759
Std With No Outliers 0.2918
Ad−Hoc Robust Monolingual French track − Distribution of the Topics of the Experiment
20
Number of Topics of the Experiment
15
10
5
0
0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Exact R−Precision
Precision averages (%) for individual Ad−Hoc Robust Monolingual French track − Comparison to Median Mean Exact R−Precision by Topic (Topics 041 to 085)
queries 1
frFSfrR2S
0.5
041 100.00 086 19.44 131 40.24 166 0.00
Difference
0
042 39.47 087 47.37 132 25.00 167 50.00
−0.5
043 100.00 088 54.84 133 18.75 168 23.08
044 16.67 089 52.17 134 66.67 169 0.00 −1
041 042 043 044 045 046 047 048 049 060 061 062 063 064 065 066 067 068 069 080 081 082 083 084 085
Topic Identifier
045 23.40 090 17.65 135 38.46 170 0.00 Ad−Hoc Robust Monolingual French track − Comparison to Median Mean Exact R−Precision by Topic (Topics 086 to 130)
1
046 14.29 091 23.08 136 50.00 171 33.33 frFSfrR2S
047 25.00 092 29.17 137 33.33 172 0.00 0.5
Difference
048 0.00 093 80.00 138 29.41 173 100.00 0
049 25.00 094 68.75 139 11.11 174 100.00 −0.5
060 38.89 095 39.55 140 58.59 175 100.00 −1
086 087 088 089 090 091 092 093 094 095 096 097 098 099 110 111 112 113 114 115 116 117 118 119 130
061 0.00 096 28.57 141 100.00 176 89.47 Topic Identifier
Ad−Hoc Robust Monolingual French track − Comparison to Median Mean Exact R−Precision by Topic (Topics 131 to 165)
062 62.50 097 28.57 142 61.54 177 57.14 1
frFSfrR2S
063 41.18 098 55.56 143 35.71 178 33.33 0.5
Difference
064 0.00 099 47.06 144 33.33 179 25.00
0
065 15.38 110 61.54 145 28.57 190 29.41
−0.5
066 88.24 111 16.67 146 0.00 191 0.00
−1
067 36.84 112 40.00 147 50.00 192 47.37 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 160 161 162 163 164 165
Topic Identifier
068 40.00 113 0.00 148 0.00 193 2.44 Ad−Hoc Robust Monolingual French track − Comparison to Median Mean Exact R−Precision by Topic (Topics 166 to 200)
1
069 43.75 114 36.67 149 60.00 194 0.00 frFSfrR2S
0.5
080 19.35 115 60.00 160 0.00 195 31.58
Difference
081 52.94 116 30.00 161 0.00 196 100.00 0
082 18.75 117 25.00 162 4.00 197 82.44 −0.5
083 0.00 118 0.00 163 36.36 198 33.33 −1
166 167 168 169 170 171 172 173 174 175 176 177 178 179 190 191 192 193 194 195 196 197 198 199 200
084 100.00 119 75.00 164 42.70 199 60.61 Topic Identifier
085 36.67 130 61.54 165 0.00 200 0.00
170
daedalus fryfrRFS3456 AH-ROBUST-MONO-FR-CLEF2006
Overall statistics for 100 queries : Priority 2
Total number of documents over all queries Query Construction AUTOMATIC
Retrieved 100,000 Source Language French
Relevant 2,190 Topic Fields title, description, narrative
Relevant retrieved 2,075 Pooled false
Geometric Mean Average Precision 0.1336 Average run
Binary Preference (BPREF) 0.4389
Interploated Recall (%) Precision Averages (%)
Ad−Hoc Robust Monolingual French track − Interpolated Recall vs Average Precision
100%
0 69.54
fryfrRFS3456
10 64.14 90%
20 55.06
80%
30 51.96
40 45.98 70%
50 42.51
Average Precision
60%
60 37.91
70 33.84 50%
80 30.04 40%
90 24.26
30%
100 19.81
Average precision (non-interpolated) for all 20%
relevant documents (averaged over queries)
41.54 10%
0%
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
Interpolated Recall
Mean Average Precision Ad−Hoc Robust Monolingual French track − Box plot of the Topics of the Experiment
Maximum 1.0000
Minimum 0.0000
First Quartile 0.1803
Second Quartile 0.3720
Third Quartile 0.5848
Interquartile range 0.4045
Mean 0.4154
Standard Deviation 0.3029
Lower Outlier Threshold 0.0000
Upper Outlier Threshold 1.0000 0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Mean Average Precision
Mean With No Outliers 0.4154
Std With No Outliers 0.3029
Ad−Hoc Robust Monolingual French track − Distribution of the Topics of the Experiment
15
Number of Topics of the Experiment
10
5
0
0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Mean Average Precision
Precision averages (%) for individual Ad−Hoc Robust Monolingual French track − Comparison to Median Mean Average Precision by Topic (Topics 041 to 085)
queries 1
fryfrRFS3456
0.5
041 100.00 086 20.53 131 35.38 166 0.00
Difference
0
042 45.74 087 43.10 132 54.05 167 64.29
−0.5
043 100.00 088 62.36 133 20.38 168 18.12
044 24.88 089 60.22 134 91.67 169 0.00 −1
041 042 043 044 045 046 047 048 049 060 061 062 063 064 065 066 067 068 069 080 081 082 083 084 085
Topic Identifier
045 33.03 090 9.81 135 33.38 170 0.59 Ad−Hoc Robust Monolingual French track − Comparison to Median Mean Average Precision by Topic (Topics 086 to 130)
1
046 5.09 091 22.95 136 100.00 171 37.00 fryfrRFS3456
047 20.45 092 18.02 137 14.45 172 0.00 0.5
Difference
048 1.24 093 100.00 138 56.22 173 100.00 0
049 21.59 094 55.38 139 18.04 174 100.00 −0.5
060 33.18 095 22.83 140 51.26 175 100.00 −1
086 087 088 089 090 091 092 093 094 095 096 097 098 099 110 111 112 113 114 115 116 117 118 119 130
061 20.09 096 27.25 141 100.00 176 94.29 Topic Identifier
Ad−Hoc Robust Monolingual French track − Comparison to Median Mean Average Precision by Topic (Topics 131 to 165)
062 54.63 097 35.13 142 58.62 177 48.45 1
fryfrRFS3456
063 31.64 098 40.66 143 48.94 178 15.74 0.5
Difference
064 0.00 099 25.58 144 51.95 179 37.39
0
065 27.62 110 57.62 145 43.51 190 48.01
−0.5
066 93.51 111 16.45 146 0.00 191 0.00
−1
067 50.46 112 47.87 147 48.22 192 68.93 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 160 161 162 163 164 165
Topic Identifier
068 45.93 113 10.83 148 2.81 193 5.37 Ad−Hoc Robust Monolingual French track − Comparison to Median Mean Average Precision by Topic (Topics 166 to 200)
1
069 35.02 114 46.16 149 68.69 194 0.00 fryfrRFS3456
0.5
080 16.32 115 66.90 160 0.00 195 31.32
Difference
081 64.98 116 46.93 161 0.00 196 100.00 0
082 16.09 117 32.29 162 7.24 197 89.53 −0.5
083 58.33 118 9.54 163 39.03 198 39.37 −1
166 167 168 169 170 171 172 173 174 175 176 177 178 179 190 191 192 193 194 195 196 197 198 199 200
084 100.00 119 72.50 164 48.19 199 74.99 Topic Identifier
085 31.10 130 62.12 165 32.50 200 11.94
171
daedalus fryfrRFS3456 AH-ROBUST-MONO-FR-CLEF2006
Docs Cutoff Levels Precision at DCL (%)
Ad−Hoc Robust Monolingual French track − Retrieved documents vs Precision
100%
5 docs 40.60
fryfrRFS3456
10 docs 35.10 90%
15 docs 31.00
80%
20 docs 28.15
30 docs 24.70 70%
100 docs 13.53
60%
200 docs 8.48
R−Precision
500 docs 3.93 50%
1000 docs 2.08 40%
R-Precision (precision after R document retrieved,
where R = Relevant retrieved) 30%
40.06
20%
10%
0%
5 10 15 20 30 100 200 500 1000
Retrieved Documents (logarithmic scale)
Exact R-Precision Ad−Hoc Robust Monolingual French track − Box plot of the Topics of the Experiment
Maximum 1.0000
Minimum 0.0000
First Quartile 0.1667
Second Quartile 0.3702
Third Quartile 0.5656
Interquartile range 0.3989
Mean 0.4006
Standard Deviation 0.2999
Lower Outlier Threshold 0.0000
Upper Outlier Threshold 1.0000 0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Exact R−Precision
Mean With No Outliers 0.4006
Std With No Outliers 0.2999
Ad−Hoc Robust Monolingual French track − Distribution of the Topics of the Experiment
20
Number of Topics of the Experiment
15
10
5
0
0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Exact R−Precision
Precision averages (%) for individual Ad−Hoc Robust Monolingual French track − Comparison to Median Mean Exact R−Precision by Topic (Topics 041 to 085)
queries 1
fryfrRFS3456
0.5
041 100.00 086 19.44 131 41.46 166 0.00
Difference
0
042 52.63 087 52.63 132 25.00 167 50.00
−0.5
043 100.00 088 54.84 133 18.75 168 23.08
044 16.67 089 52.17 134 66.67 169 0.00 −1
041 042 043 044 045 046 047 048 049 060 061 062 063 064 065 066 067 068 069 080 081 082 083 084 085
Topic Identifier
045 34.04 090 11.76 135 36.54 170 0.00 Ad−Hoc Robust Monolingual French track − Comparison to Median Mean Exact R−Precision by Topic (Topics 086 to 130)
1
046 14.29 091 23.08 136 100.00 171 33.33 fryfrRFS3456
047 25.00 092 27.08 137 33.33 172 0.00 0.5
Difference
048 0.00 093 100.00 138 56.86 173 100.00 0
049 16.67 094 56.25 139 26.67 174 100.00 −0.5
060 41.11 095 31.64 140 53.91 175 100.00 −1
086 087 088 089 090 091 092 093 094 095 096 097 098 099 110 111 112 113 114 115 116 117 118 119 130
061 8.33 096 21.43 141 100.00 176 89.47 Topic Identifier
Ad−Hoc Robust Monolingual French track − Comparison to Median Mean Exact R−Precision by Topic (Topics 131 to 165)
062 62.50 097 14.29 142 61.54 177 42.86 1
fryfrRFS3456
063 29.41 098 44.44 143 50.00 178 0.00 0.5
Difference
064 0.00 099 47.06 144 33.33 179 37.50
0
065 23.08 110 66.67 145 50.00 190 47.06
−0.5
066 86.27 111 0.00 146 0.00 191 0.00
−1
067 47.37 112 46.67 147 37.50 192 68.42 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 160 161 162 163 164 165
Topic Identifier
068 48.00 113 11.11 148 0.00 193 4.88 Ad−Hoc Robust Monolingual French track − Comparison to Median Mean Exact R−Precision by Topic (Topics 166 to 200)
1
069 50.00 114 43.33 149 60.00 194 0.00 fryfrRFS3456
0.5
080 19.35 115 60.00 160 0.00 195 31.58
Difference
081 58.82 116 40.00 161 0.00 196 100.00 0
082 25.00 117 28.57 162 4.00 197 83.97 −0.5
083 50.00 118 16.67 163 36.36 198 33.33 −1
166 167 168 169 170 171 172 173 174 175 176 177 178 179 190 191 192 193 194 195 196 197 198 199 200
084 100.00 119 70.00 164 49.44 199 63.64 Topic Identifier
085 33.33 130 69.23 165 0.00 200 5.56
172
daedalus frx011frRFW3FS3 AH-ROBUST-MONO-FR-CLEF2006
Overall statistics for 100 queries : Priority 3
Total number of documents over all queries Query Construction AUTOMATIC
Retrieved 100,000 Source Language French
Relevant 2,190 Topic Fields title, description, narrative
Relevant retrieved 2,097 Pooled false
Geometric Mean Average Precision 0.1350 Combined run
Binary Preference (BPREF) 0.4285
Interploated Recall (%) Precision Averages (%)
Ad−Hoc Robust Monolingual French track − Interpolated Recall vs Average Precision
100%
0 69.69
frx011frRFW3FS3
10 62.18 90%
20 55.16
80%
30 51.43
40 46.78 70%
50 42.88
Average Precision
60%
60 38.90
70 33.79 50%
80 30.24 40%
90 24.05
30%
100 18.86
Average precision (non-interpolated) for all 20%
relevant documents (averaged over queries)
41.41 10%
0%
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
Interpolated Recall
Mean Average Precision Ad−Hoc Robust Monolingual French track − Box plot of the Topics of the Experiment
Maximum 1.0000
Minimum 0.0000
First Quartile 0.1858
Second Quartile 0.3693
Third Quartile 0.5853
Interquartile range 0.3995
Mean 0.4141
Standard Deviation 0.2966
Lower Outlier Threshold 0.0000
Upper Outlier Threshold 1.0000 0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Mean Average Precision
Mean With No Outliers 0.4141
Std With No Outliers 0.2966
Ad−Hoc Robust Monolingual French track − Distribution of the Topics of the Experiment
15
Number of Topics of the Experiment
10
5
0
0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Mean Average Precision
Precision averages (%) for individual Ad−Hoc Robust Monolingual French track − Comparison to Median Mean Average Precision by Topic (Topics 041 to 085)
queries 1
frx011frRFW3FS3
0.5
041 100.00 086 21.01 131 35.63 166 0.00
Difference
0
042 46.20 087 45.49 132 54.37 167 64.29
−0.5
043 100.00 088 58.01 133 17.02 168 17.21
044 27.54 089 60.22 134 91.67 169 0.00 −1
041 042 043 044 045 046 047 048 049 060 061 062 063 064 065 066 067 068 069 080 081 082 083 084 085
Topic Identifier
045 30.49 090 9.38 135 33.68 170 0.75 Ad−Hoc Robust Monolingual French track − Comparison to Median Mean Average Precision by Topic (Topics 086 to 130)
1
046 5.16 091 19.96 136 100.00 171 35.85 frx011frRFW3FS3
047 23.22 092 23.30 137 13.80 172 0.00 0.5
Difference
048 1.63 093 100.00 138 67.92 173 100.00 0
049 21.65 094 59.05 139 16.04 174 100.00 −0.5
060 32.78 095 27.52 140 56.83 175 50.00 −1
086 087 088 089 090 091 092 093 094 095 096 097 098 099 110 111 112 113 114 115 116 117 118 119 130
061 23.18 096 31.30 141 100.00 176 96.56 Topic Identifier
Ad−Hoc Robust Monolingual French track − Comparison to Median Mean Average Precision by Topic (Topics 131 to 165)
062 54.50 097 39.42 142 59.72 177 55.97 1
frx011frRFW3FS3
063 28.71 098 60.47 143 53.39 178 13.10 0.5
Difference
064 0.00 099 31.62 144 50.76 179 34.19
0
065 30.96 110 53.24 145 47.70 190 53.67
−0.5
066 92.12 111 20.00 146 0.00 191 0.00
−1
067 36.12 112 44.23 147 46.83 192 66.96 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 160 161 162 163 164 165
Topic Identifier
068 40.09 113 13.59 148 2.62 193 5.69 Ad−Hoc Robust Monolingual French track − Comparison to Median Mean Average Precision by Topic (Topics 166 to 200)
1
069 37.98 114 52.64 149 69.12 194 0.00 frx011frRFW3FS3
0.5
080 15.81 115 45.37 160 0.00 195 39.66
Difference
081 63.25 116 45.46 161 0.00 196 100.00 0
082 12.13 117 35.39 162 8.69 197 86.89 −0.5
083 100.00 118 8.06 163 37.74 198 23.60 −1
166 167 168 169 170 171 172 173 174 175 176 177 178 179 190 191 192 193 194 195 196 197 198 199 200
084 50.00 119 73.15 164 47.70 199 81.29 Topic Identifier
085 30.34 130 62.31 165 32.50 200 27.61
173
daedalus frx011frRFW3FS3 AH-ROBUST-MONO-FR-CLEF2006
Docs Cutoff Levels Precision at DCL (%)
Ad−Hoc Robust Monolingual French track − Retrieved documents vs Precision
100%
5 docs 40.80
frx011frRFW3FS3
10 docs 36.30 90%
15 docs 32.33
80%
20 docs 28.85
30 docs 25.07 70%
100 docs 13.78
60%
200 docs 8.60
R−Precision
500 docs 3.97 50%
1000 docs 2.10 40%
R-Precision (precision after R document retrieved,
where R = Relevant retrieved) 30%
39.02
20%
10%
0%
5 10 15 20 30 100 200 500 1000
Retrieved Documents (logarithmic scale)
Exact R-Precision Ad−Hoc Robust Monolingual French track − Box plot of the Topics of the Experiment
Maximum 1.0000
Minimum 0.0000
First Quartile 0.1667
Second Quartile 0.3418
Third Quartile 0.5460
Interquartile range 0.3793
Mean 0.3902
Standard Deviation 0.2946
Lower Outlier Threshold 0.0000
Upper Outlier Threshold 1.0000 0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Exact R−Precision
Mean With No Outliers 0.3902
Std With No Outliers 0.2946
Ad−Hoc Robust Monolingual French track − Distribution of the Topics of the Experiment
20
Number of Topics of the Experiment
15
10
5
0
0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Exact R−Precision
Precision averages (%) for individual Ad−Hoc Robust Monolingual French track − Comparison to Median Mean Exact R−Precision by Topic (Topics 041 to 085)
queries 1
frx011frRFW3FS3
0.5
041 100.00 086 19.44 131 41.46 166 0.00
Difference
0
042 50.00 087 52.63 132 25.00 167 50.00
−0.5
043 100.00 088 48.39 133 21.88 168 15.38
044 16.67 089 52.17 134 66.67 169 0.00 −1
041 042 043 044 045 046 047 048 049 060 061 062 063 064 065 066 067 068 069 080 081 082 083 084 085
Topic Identifier
045 31.91 090 11.76 135 40.38 170 0.00 Ad−Hoc Robust Monolingual French track − Comparison to Median Mean Exact R−Precision by Topic (Topics 086 to 130)
1
046 7.14 091 23.08 136 100.00 171 33.33 frx011frRFW3FS3
047 28.85 092 31.25 137 33.33 172 0.00 0.5
Difference
048 0.00 093 100.00 138 62.75 173 100.00 0
049 16.67 094 56.25 139 26.67 174 100.00 −0.5
060 44.44 095 35.03 140 57.03 175 0.00 −1
086 087 088 089 090 091 092 093 094 095 096 097 098 099 110 111 112 113 114 115 116 117 118 119 130
061 16.67 096 21.43 141 100.00 176 89.47 Topic Identifier
Ad−Hoc Robust Monolingual French track − Comparison to Median Mean Exact R−Precision by Topic (Topics 131 to 165)
062 62.50 097 28.57 142 61.54 177 42.86 1
frx011frRFW3FS3
063 29.41 098 66.67 143 50.00 178 0.00 0.5
Difference
064 0.00 099 47.06 144 33.33 179 25.00
0
065 23.08 110 61.54 145 50.00 190 52.94
−0.5
066 86.27 111 16.67 146 0.00 191 0.00
−1
067 42.11 112 46.67 147 50.00 192 68.42 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 160 161 162 163 164 165
Topic Identifier
068 44.00 113 22.22 148 0.00 193 4.88 Ad−Hoc Robust Monolingual French track − Comparison to Median Mean Exact R−Precision by Topic (Topics 166 to 200)
1
069 43.75 114 50.00 149 60.00 194 0.00 frx011frRFW3FS3
0.5
080 19.35 115 40.00 160 0.00 195 31.58
Difference
081 52.94 116 40.00 161 0.00 196 100.00 0
082 12.50 117 32.14 162 4.00 197 80.92 −0.5
083 100.00 118 16.67 163 45.45 198 33.33 −1
166 167 168 169 170 171 172 173 174 175 176 177 178 179 190 191 192 193 194 195 196 197 198 199 200
084 0.00 119 71.25 164 48.31 199 66.67 Topic Identifier
085 26.67 130 61.54 165 0.00 200 22.22
174
daedalus frFSfrR3S AH-ROBUST-MONO-FR-CLEF2006
Overall statistics for 100 queries : Priority 1
Total number of documents over all queries Query Construction AUTOMATIC
Retrieved 100,000 Source Language French
Relevant 2,190 Topic Fields title, description, narrative
Relevant retrieved 2,097 Pooled false
Geometric Mean Average Precision 0.1369 TDN run
Binary Preference (BPREF) 0.4480
Interploated Recall (%) Precision Averages (%)
Ad−Hoc Robust Monolingual French track − Interpolated Recall vs Average Precision
100%
0 70.39
frFSfrR3S
10 63.99 90%
20 56.62
80%
30 52.55
40 47.82 70%
50 43.91
Average Precision
60%
60 39.93
70 34.79 50%
80 31.24 40%
90 25.05
30%
100 19.86
Average precision (non-interpolated) for all 20%
relevant documents (averaged over queries)
42.54 10%
0%
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
Interpolated Recall
Mean Average Precision Ad−Hoc Robust Monolingual French track − Box plot of the Topics of the Experiment
Maximum 1.0000
Minimum 0.0000
First Quartile 0.1909
Second Quartile 0.3732
Third Quartile 0.5997
Interquartile range 0.4088
Mean 0.4254
Standard Deviation 0.3082
Lower Outlier Threshold 0.0000
Upper Outlier Threshold 1.0000 0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Mean Average Precision
Mean With No Outliers 0.4254
Std With No Outliers 0.3082
Ad−Hoc Robust Monolingual French track − Distribution of the Topics of the Experiment
15
Number of Topics of the Experiment
10
5
0
0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Mean Average Precision
Precision averages (%) for individual Ad−Hoc Robust Monolingual French track − Comparison to Median Mean Average Precision by Topic (Topics 041 to 085)
queries 1
frFSfrR3S
0.5
041 100.00 086 22.40 131 36.64 166 0.00
Difference
0
042 46.20 087 45.49 132 54.37 167 64.29
−0.5
043 100.00 088 58.01 133 21.65 168 17.21
044 27.54 089 60.22 134 91.67 169 0.00 −1
041 042 043 044 045 046 047 048 049 060 061 062 063 064 065 066 067 068 069 080 081 082 083 084 085
Topic Identifier
045 31.61 090 9.38 135 36.91 170 0.75 Ad−Hoc Robust Monolingual French track − Comparison to Median Mean Average Precision by Topic (Topics 086 to 130)
1
046 5.16 091 19.96 136 100.00 171 19.18 frFSfrR3S
047 23.22 092 23.30 137 13.80 172 0.00 0.5
Difference
048 1.63 093 100.00 138 67.92 173 100.00 0
049 21.65 094 59.05 139 16.04 174 100.00 −0.5
060 32.78 095 27.56 140 56.72 175 100.00 −1
086 087 088 089 090 091 092 093 094 095 096 097 098 099 110 111 112 113 114 115 116 117 118 119 130
061 19.01 096 31.30 141 100.00 176 96.56 Topic Identifier
Ad−Hoc Robust Monolingual French track − Comparison to Median Mean Average Precision by Topic (Topics 131 to 165)
062 54.50 097 39.42 142 59.72 177 55.97 1
frFSfrR3S
063 28.71 098 60.47 143 52.67 178 15.08 0.5
Difference
064 0.00 099 27.70 144 50.76 179 34.19
0
065 30.96 110 53.24 145 47.70 190 53.67
−0.5
066 92.12 111 20.00 146 0.00 191 0.00
−1
067 44.06 112 44.23 147 46.83 192 66.96 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 160 161 162 163 164 165
Topic Identifier
068 41.20 113 15.44 148 2.62 193 5.69 Ad−Hoc Robust Monolingual French track − Comparison to Median Mean Average Precision by Topic (Topics 166 to 200)
1
069 34.86 114 52.64 149 69.12 194 0.00 frFSfrR3S
0.5
080 15.81 115 55.37 160 0.00 195 33.40
Difference
081 63.25 116 45.46 161 0.00 196 100.00 0
082 6.04 117 35.39 162 10.69 197 86.89 −0.5
083 100.00 118 8.06 163 37.74 198 40.27 −1
166 167 168 169 170 171 172 173 174 175 176 177 178 179 190 191 192 193 194 195 196 197 198 199 200
084 100.00 119 73.15 164 47.70 199 81.29 Topic Identifier
085 30.89 130 62.31 165 32.50 200 27.61
175
daedalus frFSfrR3S AH-ROBUST-MONO-FR-CLEF2006
Docs Cutoff Levels Precision at DCL (%)
Ad−Hoc Robust Monolingual French track − Retrieved documents vs Precision
100%
5 docs 41.40
frFSfrR3S
10 docs 36.10 90%
15 docs 32.33
80%
20 docs 28.85
30 docs 25.07 70%
100 docs 13.78
60%
200 docs 8.60
R−Precision
500 docs 3.97 50%
1000 docs 2.10 40%
R-Precision (precision after R document retrieved,
where R = Relevant retrieved) 30%
40.96
20%
10%
0%
5 10 15 20 30 100 200 500 1000
Retrieved Documents (logarithmic scale)
Exact R-Precision Ad−Hoc Robust Monolingual French track − Box plot of the Topics of the Experiment
Maximum 1.0000
Minimum 0.0000
First Quartile 0.1801
Second Quartile 0.4000
Third Quartile 0.5852
Interquartile range 0.4050
Mean 0.4096
Standard Deviation 0.3020
Lower Outlier Threshold 0.0000
Upper Outlier Threshold 1.0000 0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Exact R−Precision
Mean With No Outliers 0.4096
Std With No Outliers 0.3020
Ad−Hoc Robust Monolingual French track − Distribution of the Topics of the Experiment
20
Number of Topics of the Experiment
15
10
5
0
0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Exact R−Precision
Precision averages (%) for individual Ad−Hoc Robust Monolingual French track − Comparison to Median Mean Exact R−Precision by Topic (Topics 041 to 085)
queries 1
frFSfrR3S
0.5
041 100.00 086 19.44 131 41.46 166 0.00
Difference
0
042 50.00 087 52.63 132 25.00 167 50.00
−0.5
043 100.00 088 48.39 133 21.88 168 15.38
044 16.67 089 52.17 134 66.67 169 0.00 −1
041 042 043 044 045 046 047 048 049 060 061 062 063 064 065 066 067 068 069 080 081 082 083 084 085
Topic Identifier
045 31.91 090 11.76 135 40.38 170 0.00 Ad−Hoc Robust Monolingual French track − Comparison to Median Mean Exact R−Precision by Topic (Topics 086 to 130)
1
046 7.14 091 23.08 136 100.00 171 33.33 frFSfrR3S
047 28.85 092 31.25 137 33.33 172 0.00 0.5
Difference
048 0.00 093 100.00 138 62.75 173 100.00 0
049 16.67 094 56.25 139 26.67 174 100.00 −0.5
060 44.44 095 35.03 140 57.03 175 100.00 −1
086 087 088 089 090 091 092 093 094 095 096 097 098 099 110 111 112 113 114 115 116 117 118 119 130
061 16.67 096 21.43 141 100.00 176 89.47 Topic Identifier
Ad−Hoc Robust Monolingual French track − Comparison to Median Mean Exact R−Precision by Topic (Topics 131 to 165)
062 62.50 097 28.57 142 61.54 177 42.86 1
frFSfrR3S
063 29.41 098 66.67 143 50.00 178 0.00 0.5
Difference
064 0.00 099 47.06 144 33.33 179 25.00
0
065 23.08 110 61.54 145 50.00 190 52.94
−0.5
066 86.27 111 16.67 146 0.00 191 0.00
−1
067 42.11 112 46.67 147 50.00 192 68.42 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 160 161 162 163 164 165
Topic Identifier
068 44.00 113 22.22 148 0.00 193 4.88 Ad−Hoc Robust Monolingual French track − Comparison to Median Mean Exact R−Precision by Topic (Topics 166 to 200)
1
069 43.75 114 50.00 149 60.00 194 0.00 frFSfrR3S
0.5
080 19.35 115 40.00 160 0.00 195 31.58
Difference
081 52.94 116 40.00 161 0.00 196 100.00 0
082 6.25 117 32.14 162 4.00 197 80.92 −0.5
083 100.00 118 16.67 163 45.45 198 33.33 −1
166 167 168 169 170 171 172 173 174 175 176 177 178 179 190 191 192 193 194 195 196 197 198 199 200
084 100.00 119 71.25 164 48.31 199 66.67 Topic Identifier
085 26.67 130 61.54 165 0.00 200 22.22
176
dcu dcudescfr12075 AH-ROBUST-MONO-FR-CLEF2006
Overall statistics for 100 queries : Priority 1
Total number of documents over all queries Query Construction AUTOMATIC
Retrieved 100,000 Source Language French
Relevant 2,190 Topic Fields title, description
Relevant retrieved 2,074 Pooled false
Geometric Mean Average Precision 0.1212 french mono run
Binary Preference (BPREF) 0.4519
Interploated Recall (%) Precision Averages (%)
Ad−Hoc Robust Monolingual French track − Interpolated Recall vs Average Precision
100%
0 64.43
dcudescfr12075
10 58.17 90%
20 51.83
80%
30 47.85
40 44.53 70%
50 42.58
Average Precision
60%
60 39.04
70 35.73 50%
80 32.67 40%
90 27.19
30%
100 21.44
Average precision (non-interpolated) for all 20%
relevant documents (averaged over queries)
41.08 10%
0%
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
Interpolated Recall
Mean Average Precision Ad−Hoc Robust Monolingual French track − Box plot of the Topics of the Experiment
Maximum 1.0000
Minimum 0.0000
First Quartile 0.1140
Second Quartile 0.3567
Third Quartile 0.6195
Interquartile range 0.5055
Mean 0.4108
Standard Deviation 0.3286
Lower Outlier Threshold 0.0000
Upper Outlier Threshold 1.0000 0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Mean Average Precision
Mean With No Outliers 0.4108
Std With No Outliers 0.3286
Ad−Hoc Robust Monolingual French track − Distribution of the Topics of the Experiment
20
Number of Topics of the Experiment
15
10
5
0
0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Mean Average Precision
Precision averages (%) for individual Ad−Hoc Robust Monolingual French track − Comparison to Median Mean Average Precision by Topic (Topics 041 to 085)
queries 1
dcudescfr12075
0.5
041 100.00 086 15.33 131 9.59 166 0.00
Difference
0
042 58.47 087 40.68 132 47.13 167 83.33
−0.5
043 33.33 088 59.90 133 1.27 168 39.40
044 21.21 089 37.40 134 91.67 169 0.00 −1
041 042 043 044 045 046 047 048 049 060 061 062 063 064 065 066 067 068 069 080 081 082 083 084 085
Topic Identifier
045 58.37 090 10.97 135 82.23 170 25.00 Ad−Hoc Robust Monolingual French track − Comparison to Median Mean Average Precision by Topic (Topics 086 to 130)
1
046 5.03 091 10.72 136 100.00 171 35.30 dcudescfr12075
047 20.22 092 39.62 137 19.59 172 0.00 0.5
Difference
048 0.87 093 96.67 138 53.31 173 100.00 0
049 13.13 094 6.11 139 8.95 174 100.00 −0.5
060 28.89 095 30.50 140 61.57 175 100.00 −1
086 087 088 089 090 091 092 093 094 095 096 097 098 099 110 111 112 113 114 115 116 117 118 119 130
061 9.61 096 39.66 141 100.00 176 97.62 Topic Identifier
Ad−Hoc Robust Monolingual French track − Comparison to Median Mean Average Precision by Topic (Topics 131 to 165)
062 36.06 097 44.26 142 98.97 177 38.42 1
dcudescfr12075
063 58.05 098 96.58 143 30.79 178 12.12 0.5
Difference
064 0.00 099 23.52 144 100.00 179 37.64
0
065 27.78 110 65.06 145 21.80 190 9.82
−0.5
066 88.26 111 4.47 146 0.00 191 0.00
−1
067 47.79 112 35.60 147 36.89 192 83.90 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 160 161 162 163 164 165
Topic Identifier
068 47.45 113 2.80 148 2.78 193 42.09 Ad−Hoc Robust Monolingual French track − Comparison to Median Mean Average Precision by Topic (Topics 166 to 200)
1
069 27.07 114 33.64 149 94.29 194 0.00 dcudescfr12075
0.5
080 15.19 115 33.54 160 0.00 195 22.80
Difference
081 66.06 116 34.05 161 0.00 196 100.00 0
082 31.47 117 8.58 162 87.86 197 86.86 −0.5
083 62.50 118 3.23 163 35.74 198 11.83 −1
166 167 168 169 170 171 172 173 174 175 176 177 178 179 190 191 192 193 194 195 196 197 198 199 200
084 50.00 119 83.47 164 49.29 199 62.33 Topic Identifier
085 22.33 130 56.58 165 41.67 200 2.27
177
dcu dcudescfr12075 AH-ROBUST-MONO-FR-CLEF2006
Docs Cutoff Levels Precision at DCL (%)
Ad−Hoc Robust Monolingual French track − Retrieved documents vs Precision
100%
5 docs 39.60
dcudescfr12075
10 docs 33.70 90%
15 docs 30.80
80%
20 docs 28.50
30 docs 24.73 70%
100 docs 13.58
60%
200 docs 8.50
R−Precision
500 docs 3.89 50%
1000 docs 2.07 40%
R-Precision (precision after R document retrieved,
where R = Relevant retrieved) 30%
37.93
20%
10%
0%
5 10 15 20 30 100 200 500 1000
Retrieved Documents (logarithmic scale)
Exact R-Precision Ad−Hoc Robust Monolingual French track − Box plot of the Topics of the Experiment
Maximum 1.0000
Minimum 0.0000
First Quartile 0.0541
Second Quartile 0.3452
Third Quartile 0.5798
Interquartile range 0.5257
Mean 0.3793
Standard Deviation 0.3214
Lower Outlier Threshold 0.0000
Upper Outlier Threshold 1.0000 0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Exact R−Precision
Mean With No Outliers 0.3793
Std With No Outliers 0.3214
Ad−Hoc Robust Monolingual French track − Distribution of the Topics of the Experiment
25
Number of Topics of the Experiment
20
15
10
5
0
0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Exact R−Precision
Precision averages (%) for individual Ad−Hoc Robust Monolingual French track − Comparison to Median Mean Exact R−Precision by Topic (Topics 041 to 085)
queries 1
dcudescfr12075
0.5
041 100.00 086 16.67 131 13.41 166 0.00
Difference
0
042 56.58 087 36.84 132 25.00 167 50.00
−0.5
043 0.00 088 64.52 133 3.12 168 38.46
044 0.00 089 39.13 134 66.67 169 0.00 −1
041 042 043 044 045 046 047 048 049 060 061 062 063 064 065 066 067 068 069 080 081 082 083 084 085
Topic Identifier
045 51.06 090 11.76 135 75.00 170 0.00 Ad−Hoc Robust Monolingual French track − Comparison to Median Mean Exact R−Precision by Topic (Topics 086 to 130)
1
046 0.00 091 7.69 136 100.00 171 33.33 dcudescfr12075
047 26.92 092 31.25 137 33.33 172 0.00 0.5
Difference
048 0.00 093 80.00 138 50.98 173 100.00 0
049 16.67 094 0.00 139 15.56 174 100.00 −0.5
060 26.67 095 37.85 140 62.50 175 100.00 −1
086 087 088 089 090 091 092 093 094 095 096 097 098 099 110 111 112 113 114 115 116 117 118 119 130
061 0.00 096 35.71 141 100.00 176 89.47 Topic Identifier
Ad−Hoc Robust Monolingual French track − Comparison to Median Mean Exact R−Precision by Topic (Topics 131 to 165)
062 37.50 097 57.14 142 92.31 177 28.57 1
dcudescfr12075
063 52.94 098 88.89 143 25.00 178 0.00 0.5
Difference
064 0.00 099 41.18 144 100.00 179 25.00
0
065 23.08 110 66.67 145 21.43 190 17.65
−0.5
066 84.31 111 0.00 146 0.00 191 0.00
−1
067 47.37 112 33.33 147 50.00 192 73.68 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 160 161 162 163 164 165
Topic Identifier
068 52.00 113 0.00 148 0.00 193 31.71 Ad−Hoc Robust Monolingual French track − Comparison to Median Mean Exact R−Precision by Topic (Topics 166 to 200)
1
069 37.50 114 36.67 149 80.00 194 0.00 dcudescfr12075
0.5
080 22.58 115 40.00 160 0.00 195 21.05
Difference
081 58.82 116 30.00 161 0.00 196 100.00 0
082 43.75 117 10.71 162 80.00 197 79.39 −0.5
083 50.00 118 0.00 163 36.36 198 0.00 −1
166 167 168 169 170 171 172 173 174 175 176 177 178 179 190 191 192 193 194 195 196 197 198 199 200
084 0.00 119 77.50 164 51.69 199 51.52 Topic Identifier
085 20.00 130 69.23 165 50.00 200 0.00
178
dcu dcudescfr12075 AH-ROBUST-MONO-FR-CLEF2006
Overall statistics for 100 queries : Priority 4
Total number of documents over all queries Query Construction AUTOMATIC
Retrieved 100,000 Source Language French
Relevant 2,190 Topic Fields title, description
Relevant retrieved 2,074 Pooled false
Geometric Mean Average Precision 0.1212 French monolingual run with title and description
Binary Preference (BPREF) 0.4519 fields
Interploated Recall (%) Precision Averages (%)
Ad−Hoc Robust Monolingual French track − Interpolated Recall vs Average Precision
100%
0 64.43
dcudescfr12075
10 58.17 90%
20 51.83
80%
30 47.85
40 44.53 70%
50 42.58
Average Precision
60%
60 39.04
70 35.73 50%
80 32.67 40%
90 27.19
30%
100 21.44
Average precision (non-interpolated) for all 20%
relevant documents (averaged over queries)
41.08 10%
0%
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
Interpolated Recall
Mean Average Precision Ad−Hoc Robust Monolingual French track − Box plot of the Topics of the Experiment
Maximum 1.0000
Minimum 0.0000
First Quartile 0.1140
Second Quartile 0.3567
Third Quartile 0.6195
Interquartile range 0.5055
Mean 0.4108
Standard Deviation 0.3286
Lower Outlier Threshold 0.0000
Upper Outlier Threshold 1.0000 0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Mean Average Precision
Mean With No Outliers 0.4108
Std With No Outliers 0.3286
Ad−Hoc Robust Monolingual French track − Distribution of the Topics of the Experiment
20
Number of Topics of the Experiment
15
10
5
0
0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Mean Average Precision
Precision averages (%) for individual Ad−Hoc Robust Monolingual French track − Comparison to Median Mean Average Precision by Topic (Topics 041 to 085)
queries 1
dcudescfr12075
0.5
041 100.00 086 15.33 131 9.59 166 0.00
Difference
0
042 58.47 087 40.68 132 47.13 167 83.33
−0.5
043 33.33 088 59.90 133 1.27 168 39.40
044 21.21 089 37.40 134 91.67 169 0.00 −1
041 042 043 044 045 046 047 048 049 060 061 062 063 064 065 066 067 068 069 080 081 082 083 084 085
Topic Identifier
045 58.37 090 10.97 135 82.23 170 25.00 Ad−Hoc Robust Monolingual French track − Comparison to Median Mean Average Precision by Topic (Topics 086 to 130)
1
046 5.03 091 10.72 136 100.00 171 35.30 dcudescfr12075
047 20.22 092 39.62 137 19.59 172 0.00 0.5
Difference
048 0.87 093 96.67 138 53.31 173 100.00 0
049 13.13 094 6.11 139 8.95 174 100.00 −0.5
060 28.89 095 30.50 140 61.57 175 100.00 −1
086 087 088 089 090 091 092 093 094 095 096 097 098 099 110 111 112 113 114 115 116 117 118 119 130
061 9.61 096 39.66 141 100.00 176 97.62 Topic Identifier
Ad−Hoc Robust Monolingual French track − Comparison to Median Mean Average Precision by Topic (Topics 131 to 165)
062 36.06 097 44.26 142 98.97 177 38.42 1
dcudescfr12075
063 58.05 098 96.58 143 30.79 178 12.12 0.5
Difference
064 0.00 099 23.52 144 100.00 179 37.64
0
065 27.78 110 65.06 145 21.80 190 9.82
−0.5
066 88.26 111 4.47 146 0.00 191 0.00
−1
067 47.79 112 35.60 147 36.89 192 83.90 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 160 161 162 163 164 165
Topic Identifier
068 47.45 113 2.80 148 2.78 193 42.09 Ad−Hoc Robust Monolingual French track − Comparison to Median Mean Average Precision by Topic (Topics 166 to 200)
1
069 27.07 114 33.64 149 94.29 194 0.00 dcudescfr12075
0.5
080 15.19 115 33.54 160 0.00 195 22.80
Difference
081 66.06 116 34.05 161 0.00 196 100.00 0
082 31.47 117 8.58 162 87.86 197 86.86 −0.5
083 62.50 118 3.23 163 35.74 198 11.83 −1
166 167 168 169 170 171 172 173 174 175 176 177 178 179 190 191 192 193 194 195 196 197 198 199 200
084 50.00 119 83.47 164 49.29 199 62.33 Topic Identifier
085 22.33 130 56.58 165 41.67 200 2.27
179
dcu dcudescfr12075 AH-ROBUST-MONO-FR-CLEF2006
Docs Cutoff Levels Precision at DCL (%)
Ad−Hoc Robust Monolingual French track − Retrieved documents vs Precision
100%
5 docs 39.60
dcudescfr12075
10 docs 33.70 90%
15 docs 30.80
80%
20 docs 28.50
30 docs 24.73 70%
100 docs 13.58
60%
200 docs 8.50
R−Precision
500 docs 3.89 50%
1000 docs 2.07 40%
R-Precision (precision after R document retrieved,
where R = Relevant retrieved) 30%
37.93
20%
10%
0%
5 10 15 20 30 100 200 500 1000
Retrieved Documents (logarithmic scale)
Exact R-Precision Ad−Hoc Robust Monolingual French track − Box plot of the Topics of the Experiment
Maximum 1.0000
Minimum 0.0000
First Quartile 0.0541
Second Quartile 0.3452
Third Quartile 0.5798
Interquartile range 0.5257
Mean 0.3793
Standard Deviation 0.3214
Lower Outlier Threshold 0.0000
Upper Outlier Threshold 1.0000 0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Exact R−Precision
Mean With No Outliers 0.3793
Std With No Outliers 0.3214
Ad−Hoc Robust Monolingual French track − Distribution of the Topics of the Experiment
25
Number of Topics of the Experiment
20
15
10
5
0
0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Exact R−Precision
Precision averages (%) for individual Ad−Hoc Robust Monolingual French track − Comparison to Median Mean Exact R−Precision by Topic (Topics 041 to 085)
queries 1
dcudescfr12075
0.5
041 100.00 086 16.67 131 13.41 166 0.00
Difference
0
042 56.58 087 36.84 132 25.00 167 50.00
−0.5
043 0.00 088 64.52 133 3.12 168 38.46
044 0.00 089 39.13 134 66.67 169 0.00 −1
041 042 043 044 045 046 047 048 049 060 061 062 063 064 065 066 067 068 069 080 081 082 083 084 085
Topic Identifier
045 51.06 090 11.76 135 75.00 170 0.00 Ad−Hoc Robust Monolingual French track − Comparison to Median Mean Exact R−Precision by Topic (Topics 086 to 130)
1
046 0.00 091 7.69 136 100.00 171 33.33 dcudescfr12075
047 26.92 092 31.25 137 33.33 172 0.00 0.5
Difference
048 0.00 093 80.00 138 50.98 173 100.00 0
049 16.67 094 0.00 139 15.56 174 100.00 −0.5
060 26.67 095 37.85 140 62.50 175 100.00 −1
086 087 088 089 090 091 092 093 094 095 096 097 098 099 110 111 112 113 114 115 116 117 118 119 130
061 0.00 096 35.71 141 100.00 176 89.47 Topic Identifier
Ad−Hoc Robust Monolingual French track − Comparison to Median Mean Exact R−Precision by Topic (Topics 131 to 165)
062 37.50 097 57.14 142 92.31 177 28.57 1
dcudescfr12075
063 52.94 098 88.89 143 25.00 178 0.00 0.5
Difference
064 0.00 099 41.18 144 100.00 179 25.00
0
065 23.08 110 66.67 145 21.43 190 17.65
−0.5
066 84.31 111 0.00 146 0.00 191 0.00
−1
067 47.37 112 33.33 147 50.00 192 73.68 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 160 161 162 163 164 165
Topic Identifier
068 52.00 113 0.00 148 0.00 193 31.71 Ad−Hoc Robust Monolingual French track − Comparison to Median Mean Exact R−Precision by Topic (Topics 166 to 200)
1
069 37.50 114 36.67 149 80.00 194 0.00 dcudescfr12075
0.5
080 22.58 115 40.00 160 0.00 195 21.05
Difference
081 58.82 116 30.00 161 0.00 196 100.00 0
082 43.75 117 10.71 162 80.00 197 79.39 −0.5
083 50.00 118 0.00 163 36.36 198 0.00 −1
166 167 168 169 170 171 172 173 174 175 176 177 178 179 190 191 192 193 194 195 196 197 198 199 200
084 0.00 119 77.50 164 51.69 199 51.52 Topic Identifier
085 20.00 130 69.23 165 50.00 200 0.00
180
dcu dcunarrfr1005 AH-ROBUST-MONO-FR-CLEF2006
Overall statistics for 100 queries : Priority 2
Total number of documents over all queries Query Construction AUTOMATIC
Retrieved 100,000 Source Language French
Relevant 2,190 Topic Fields title, description, narrative
Relevant retrieved 2,091 Pooled false
Geometric Mean Average Precision 0.1524 french mono
Binary Preference (BPREF) 0.4954
Interploated Recall (%) Precision Averages (%)
Ad−Hoc Robust Monolingual French track − Interpolated Recall vs Average Precision
100%
0 70.89
dcunarrfr1005
10 65.10 90%
20 59.46
80%
30 54.97
40 49.82 70%
50 47.24
Average Precision
60%
60 43.21
70 39.44 50%
80 36.24 40%
90 29.79
30%
100 23.55
Average precision (non-interpolated) for all 20%
relevant documents (averaged over queries)
45.87 10%
0%
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
Interpolated Recall
Mean Average Precision Ad−Hoc Robust Monolingual French track − Box plot of the Topics of the Experiment
Maximum 1.0000
Minimum 0.0000
First Quartile 0.2049
Second Quartile 0.3891
Third Quartile 0.7031
Interquartile range 0.4982
Mean 0.4587
Standard Deviation 0.3254
Lower Outlier Threshold 0.0000
Upper Outlier Threshold 1.0000 0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Mean Average Precision
Mean With No Outliers 0.4587
Std With No Outliers 0.3254
Ad−Hoc Robust Monolingual French track − Distribution of the Topics of the Experiment
15
Number of Topics of the Experiment
10
5
0
0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Mean Average Precision
Precision averages (%) for individual Ad−Hoc Robust Monolingual French track − Comparison to Median Mean Average Precision by Topic (Topics 041 to 085)
queries 1
dcunarrfr1005
0.5
041 100.00 086 20.40 131 18.66 166 0.00
Difference
0
042 68.29 087 43.90 132 38.11 167 83.33
−0.5
043 100.00 088 60.61 133 33.02 168 30.00
044 25.81 089 42.65 134 91.67 169 0.00 −1
041 042 043 044 045 046 047 048 049 060 061 062 063 064 065 066 067 068 069 080 081 082 083 084 085
Topic Identifier
045 58.64 090 10.18 135 70.20 170 33.33 Ad−Hoc Robust Monolingual French track − Comparison to Median Mean Average Precision by Topic (Topics 086 to 130)
1
046 9.33 091 11.00 136 100.00 171 18.06 dcunarrfr1005
047 24.76 092 34.17 137 37.14 172 0.00 0.5
Difference
048 7.87 093 100.00 138 70.43 173 100.00 0
049 16.90 094 56.42 139 6.10 174 100.00 −0.5
060 31.71 095 25.16 140 58.69 175 100.00 −1
086 087 088 089 090 091 092 093 094 095 096 097 098 099 110 111 112 113 114 115 116 117 118 119 130
061 6.87 096 35.13 141 100.00 176 99.50 Topic Identifier
Ad−Hoc Robust Monolingual French track − Comparison to Median Mean Average Precision by Topic (Topics 131 to 165)
062 39.72 097 47.06 142 98.97 177 35.30 1
dcunarrfr1005
063 43.71 098 95.35 143 60.10 178 20.59 0.5
Difference
064 0.00 099 29.14 144 100.00 179 31.45
0
065 37.92 110 56.33 145 41.77 190 10.62
−0.5
066 91.10 111 23.74 146 0.00 191 0.00
−1
067 56.95 112 43.51 147 67.51 192 84.02 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 160 161 162 163 164 165
Topic Identifier
068 46.11 113 6.99 148 4.33 193 37.30 Ad−Hoc Robust Monolingual French track − Comparison to Median Mean Average Precision by Topic (Topics 166 to 200)
1
069 33.22 114 49.90 149 85.29 194 0.00 dcunarrfr1005
0.5
080 16.16 115 33.54 160 0.00 195 49.69
Difference
081 65.59 116 41.12 161 0.00 196 100.00 0
082 9.97 117 26.89 162 83.56 197 89.70 −0.5
083 83.33 118 8.87 163 44.81 198 37.38 −1
166 167 168 169 170 171 172 173 174 175 176 177 178 179 190 191 192 193 194 195 196 197 198 199 200
084 100.00 119 79.16 164 25.07 199 79.20 Topic Identifier
085 26.80 130 64.69 165 45.00 200 20.72
181
dcu dcunarrfr1005 AH-ROBUST-MONO-FR-CLEF2006
Docs Cutoff Levels Precision at DCL (%)
Ad−Hoc Robust Monolingual French track − Retrieved documents vs Precision
100%
5 docs 43.20
dcunarrfr1005
10 docs 38.70 90%
15 docs 34.93
80%
20 docs 31.90
30 docs 26.77 70%
100 docs 14.38
60%
200 docs 8.78
R−Precision
500 docs 3.97 50%
1000 docs 2.09 40%
R-Precision (precision after R document retrieved,
where R = Relevant retrieved) 30%
44.55
20%
10%
0%
5 10 15 20 30 100 200 500 1000
Retrieved Documents (logarithmic scale)
Exact R-Precision Ad−Hoc Robust Monolingual French track − Box plot of the Topics of the Experiment
Maximum 1.0000
Minimum 0.0000
First Quartile 0.1771
Second Quartile 0.4248
Third Quartile 0.6349
Interquartile range 0.4578
Mean 0.4455
Standard Deviation 0.3098
Lower Outlier Threshold 0.0000
Upper Outlier Threshold 1.0000 0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Exact R−Precision
Mean With No Outliers 0.4455
Std With No Outliers 0.3098
Ad−Hoc Robust Monolingual French track − Distribution of the Topics of the Experiment
15
Number of Topics of the Experiment
10
5
0
0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Exact R−Precision
Precision averages (%) for individual Ad−Hoc Robust Monolingual French track − Comparison to Median Mean Exact R−Precision by Topic (Topics 041 to 085)
queries 1
dcunarrfr1005
0.5
041 100.00 086 16.67 131 23.17 166 0.00
Difference
0
042 64.47 087 42.11 132 25.00 167 50.00
−0.5
043 100.00 088 51.61 133 31.25 168 38.46
044 33.33 089 52.17 134 66.67 169 0.00 −1
041 042 043 044 045 046 047 048 049 060 061 062 063 064 065 066 067 068 069 080 081 082 083 084 085
Topic Identifier
045 48.94 090 11.76 135 67.31 170 0.00 Ad−Hoc Robust Monolingual French track − Comparison to Median Mean Exact R−Precision by Topic (Topics 086 to 130)
1
046 0.00 091 7.69 136 100.00 171 33.33 dcunarrfr1005
047 32.69 092 33.33 137 33.33 172 0.00 0.5
Difference
048 14.29 093 100.00 138 58.82 173 100.00 0
049 8.33 094 56.25 139 8.89 174 100.00 −0.5
060 40.00 095 33.90 140 60.94 175 100.00 −1
086 087 088 089 090 091 092 093 094 095 096 097 098 099 110 111 112 113 114 115 116 117 118 119 130
061 8.33 096 28.57 141 100.00 176 94.74 Topic Identifier
Ad−Hoc Robust Monolingual French track − Comparison to Median Mean Exact R−Precision by Topic (Topics 131 to 165)
062 37.50 097 57.14 142 92.31 177 14.29 1
dcunarrfr1005
063 47.06 098 88.89 143 60.71 178 33.33 0.5
Difference
064 0.00 099 41.18 144 100.00 179 12.50
0
065 46.15 110 61.54 145 42.86 190 11.76
−0.5
066 80.39 111 33.33 146 0.00 191 0.00
−1
067 57.89 112 53.33 147 62.50 192 78.95 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 160 161 162 163 164 165
Topic Identifier
068 44.00 113 11.11 148 0.00 193 29.27 Ad−Hoc Robust Monolingual French track − Comparison to Median Mean Exact R−Precision by Topic (Topics 166 to 200)
1
069 43.75 114 50.00 149 80.00 194 0.00 dcunarrfr1005
0.5
080 16.13 115 20.00 160 0.00 195 47.37
Difference
081 58.82 116 50.00 161 0.00 196 100.00 0
082 18.75 117 28.57 162 76.00 197 81.68 −0.5
083 50.00 118 16.67 163 54.55 198 33.33 −1
166 167 168 169 170 171 172 173 174 175 176 177 178 179 190 191 192 193 194 195 196 197 198 199 200
084 100.00 119 72.50 164 30.34 199 69.70 Topic Identifier
085 30.00 130 69.23 165 50.00 200 33.33
182
hildesheim uhifrmo2 AH-ROBUST-MONO-FR-CLEF2006
Overall statistics for 100 queries : Priority 1
Total number of documents over all queries Query Construction AUTOMATIC
Retrieved 99,000 Source Language French
Relevant 2,190 Topic Fields title, description
Relevant retrieved 1,882 Pooled false
Geometric Mean Average Precision 0.0625 Lucene stemmer, BRF 15, 30
Binary Preference (BPREF) 0.3319
Interploated Recall (%) Precision Averages (%)
Ad−Hoc Robust Monolingual French track − Interpolated Recall vs Average Precision
100%
0 49.45
uhifrmo2
10 42.93 90%
20 40.03
80%
30 36.61
40 34.46 70%
50 31.93
Average Precision
60%
60 29.33
70 25.93 50%
80 22.54 40%
90 17.75
30%
100 14.44
Average precision (non-interpolated) for all 20%
relevant documents (averaged over queries)
29.85 10%
0%
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
Interpolated Recall
Mean Average Precision Ad−Hoc Robust Monolingual French track − Box plot of the Topics of the Experiment
Maximum 1.0000
Minimum 0.0000
First Quartile 0.0375
Second Quartile 0.2029
Third Quartile 0.4821
Interquartile range 0.4445
Mean 0.2985
Standard Deviation 0.3017
Lower Outlier Threshold 0.0000
Upper Outlier Threshold 1.0000 0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Mean Average Precision
Mean With No Outliers 0.2985
Std With No Outliers 0.3017
Ad−Hoc Robust Monolingual French track − Distribution of the Topics of the Experiment
30
Number of Topics of the Experiment
25
20
15
10
5
0
0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Mean Average Precision
Precision averages (%) for individual Ad−Hoc Robust Monolingual French track − Comparison to Median Mean Average Precision by Topic (Topics 041 to 085)
queries 1
uhifrmo2
0.5
041 9.90 086 10.90 131 19.49 166 0.00
Difference
0
042 43.16 087 0.25 132 56.67 167 12.80
−0.5
043 0.81 088 59.43 133 1.41 168 14.18
044 22.05 089 46.58 134 16.41 169 0.00 −1
041 042 043 044 045 046 047 048 049 060 061 062 063 064 065 066 067 068 069 080 081 082 083 084 085
Topic Identifier
045 25.85 090 0.50 135 11.68 170 9.09 Ad−Hoc Robust Monolingual French track − Comparison to Median Mean Average Precision by Topic (Topics 086 to 130)
1
046 1.25 091 5.04 136 100.00 171 35.56 uhifrmo2
047 16.54 092 21.07 137 0.46 172 0.00 0.5
Difference
048 1.62 093 92.50 138 11.71 173 100.00 0
049 0.46 094 60.24 139 4.49 174 100.00 −0.5
060 18.18 095 24.55 140 19.50 175 100.00 −1
086 087 088 089 090 091 092 093 094 095 096 097 098 099 110 111 112 113 114 115 116 117 118 119 130
061 21.41 096 15.81 141 25.00 176 90.87 Topic Identifier
Ad−Hoc Robust Monolingual French track − Comparison to Median Mean Average Precision by Topic (Topics 131 to 165)
062 25.32 097 32.00 142 100.00 177 22.93 1
uhifrmo2
063 22.28 098 8.49 143 21.24 178 6.25 0.5
Difference
064 0.00 099 8.59 144 75.76 179 0.00
0
065 0.26 110 68.82 145 45.80 190 47.73
−0.5
066 88.54 111 2.91 146 0.00 191 0.00
−1
067 53.12 112 48.37 147 2.70 192 48.04 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 160 161 162 163 164 165
Topic Identifier
068 49.98 113 7.28 148 2.36 193 41.98 Ad−Hoc Robust Monolingual French track − Comparison to Median Mean Average Precision by Topic (Topics 166 to 200)
1
069 45.80 114 38.79 149 82.62 194 0.00 uhifrmo2
0.5
080 8.10 115 53.82 160 0.00 195 6.15
Difference
081 54.34 116 26.61 161 0.00 196 100.00 0
082 39.31 117 0.84 162 54.60 197 84.12 −0.5
083 18.38 118 3.01 163 19.01 198 13.96 −1
166 167 168 169 170 171 172 173 174 175 176 177 178 179 190 191 192 193 194 195 196 197 198 199 200
084 11.11 119 59.86 164 26.38 199 51.59 Topic Identifier
085 27.46 130 63.31 165 36.67 200 0.80
183
hildesheim uhifrmo2 AH-ROBUST-MONO-FR-CLEF2006
Docs Cutoff Levels Precision at DCL (%)
Ad−Hoc Robust Monolingual French track − Retrieved documents vs Precision
100%
5 docs 28.20
uhifrmo2
10 docs 26.70 90%
15 docs 25.20
80%
20 docs 23.00
30 docs 20.53 70%
100 docs 11.41
60%
200 docs 7.18
R−Precision
500 docs 3.48 50%
1000 docs 1.88 40%
R-Precision (precision after R document retrieved,
where R = Relevant retrieved) 30%
28.45
20%
10%
0%
5 10 15 20 30 100 200 500 1000
Retrieved Documents (logarithmic scale)
Exact R-Precision Ad−Hoc Robust Monolingual French track − Box plot of the Topics of the Experiment
Maximum 1.0000
Minimum 0.0000
First Quartile 0.0000
Second Quartile 0.1923
Third Quartile 0.5000
Interquartile range 0.5000
Mean 0.2845
Standard Deviation 0.3033
Lower Outlier Threshold 0.0000
Upper Outlier Threshold 1.0000 0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Exact R−Precision
Mean With No Outliers 0.2845
Std With No Outliers 0.3033
Ad−Hoc Robust Monolingual French track − Distribution of the Topics of the Experiment
40
Number of Topics of the Experiment
35
30
25
20
15
10
5
0
0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Exact R−Precision
Precision averages (%) for individual Ad−Hoc Robust Monolingual French track − Comparison to Median Mean Exact R−Precision by Topic (Topics 041 to 085)
queries 1
uhifrmo2
0.5
041 0.00 086 16.67 131 23.17 166 0.00
Difference
0
042 51.32 087 0.00 132 50.00 167 0.00
−0.5
043 0.00 088 58.06 133 6.25 168 15.38
044 16.67 089 52.17 134 0.00 169 0.00 −1
041 042 043 044 045 046 047 048 049 060 061 062 063 064 065 066 067 068 069 080 081 082 083 084 085
Topic Identifier
045 36.17 090 0.00 135 19.23 170 0.00 Ad−Hoc Robust Monolingual French track − Comparison to Median Mean Exact R−Precision by Topic (Topics 086 to 130)
1
046 0.00 091 0.00 136 100.00 171 33.33 uhifrmo2
047 19.23 092 25.00 137 0.00 172 0.00 0.5
Difference
048 0.00 093 80.00 138 25.49 173 100.00 0
049 0.00 094 56.25 139 4.44 174 100.00 −0.5
060 26.67 095 32.20 140 27.34 175 100.00 −1
086 087 088 089 090 091 092 093 094 095 096 097 098 099 110 111 112 113 114 115 116 117 118 119 130
061 25.00 096 7.14 141 0.00 176 78.95 Topic Identifier
Ad−Hoc Robust Monolingual French track − Comparison to Median Mean Exact R−Precision by Topic (Topics 131 to 165)
062 37.50 097 28.57 142 100.00 177 0.00 1
uhifrmo2
063 17.65 098 11.11 143 17.86 178 0.00 0.5
Difference
064 0.00 099 11.76 144 66.67 179 0.00
0
065 0.00 110 71.79 145 42.86 190 41.18
−0.5
066 80.39 111 0.00 146 0.00 191 0.00
−1
067 57.89 112 46.67 147 0.00 192 52.63 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 160 161 162 163 164 165
Topic Identifier
068 60.00 113 0.00 148 0.00 193 39.02 Ad−Hoc Robust Monolingual French track − Comparison to Median Mean Exact R−Precision by Topic (Topics 166 to 200)
1
069 50.00 114 46.67 149 60.00 194 0.00 uhifrmo2
0.5
080 12.90 115 60.00 160 0.00 195 15.79
Difference
081 52.94 116 40.00 161 0.00 196 100.00 0
082 50.00 117 3.57 162 48.00 197 78.63 −0.5
083 0.00 118 0.00 163 27.27 198 0.00 −1
166 167 168 169 170 171 172 173 174 175 176 177 178 179 190 191 192 193 194 195 196 197 198 199 200
084 0.00 119 58.75 164 30.34 199 48.48 Topic Identifier
085 30.00 130 61.54 165 0.00 200 0.00
184
hildesheim uhifrmo1 AH-ROBUST-MONO-FR-CLEF2006
Overall statistics for 100 queries : Priority 2
Total number of documents over all queries Query Construction AUTOMATIC
Retrieved 99,000 Source Language French
Relevant 2,190 Topic Fields title, description
Relevant retrieved 1,833 Pooled false
Geometric Mean Average Precision 0.0576 Lucene stemmer, BRF 5, 30
Binary Preference (BPREF) 0.3192
Interploated Recall (%) Precision Averages (%)
Ad−Hoc Robust Monolingual French track − Interpolated Recall vs Average Precision
100%
0 46.53
uhifrmo1
10 42.06 90%
20 38.84
80%
30 35.91
40 32.77 70%
50 30.43
Average Precision
60%
60 27.64
70 24.17 50%
80 21.02 40%
90 16.96
30%
100 13.55
Average precision (non-interpolated) for all 20%
relevant documents (averaged over queries)
28.50 10%
0%
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
Interpolated Recall
Mean Average Precision Ad−Hoc Robust Monolingual French track − Box plot of the Topics of the Experiment
Maximum 1.0000
Minimum 0.0000
First Quartile 0.0340
Second Quartile 0.1814
Third Quartile 0.4550
Interquartile range 0.4211
Mean 0.2850
Standard Deviation 0.2993
Lower Outlier Threshold 0.0000
Upper Outlier Threshold 1.0000 0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Mean Average Precision
Mean With No Outliers 0.2850
Std With No Outliers 0.2993
Ad−Hoc Robust Monolingual French track − Distribution of the Topics of the Experiment
30
Number of Topics of the Experiment
25
20
15
10
5
0
0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Mean Average Precision
Precision averages (%) for individual Ad−Hoc Robust Monolingual French track − Comparison to Median Mean Average Precision by Topic (Topics 041 to 085)
queries 1
uhifrmo1
0.5
041 5.80 086 8.36 131 13.91 166 0.00
Difference
0
042 19.76 087 0.37 132 62.20 167 10.16
−0.5
043 0.81 088 68.63 133 1.77 168 9.24
044 18.48 089 59.47 134 21.78 169 0.00 −1
041 042 043 044 045 046 047 048 049 060 061 062 063 064 065 066 067 068 069 080 081 082 083 084 085
Topic Identifier
045 26.99 090 0.35 135 5.25 170 11.11 Ad−Hoc Robust Monolingual French track − Comparison to Median Mean Average Precision by Topic (Topics 086 to 130)
1
046 3.48 091 3.58 136 100.00 171 17.50 uhifrmo1
047 12.74 092 19.08 137 0.14 172 0.00 0.5
Difference
048 1.27 093 85.00 138 14.60 173 100.00 0
049 0.46 094 45.72 139 6.44 174 100.00 −0.5
060 17.48 095 24.74 140 16.23 175 100.00 −1
086 087 088 089 090 091 092 093 094 095 096 097 098 099 110 111 112 113 114 115 116 117 118 119 130
061 4.75 096 12.38 141 25.00 176 82.81 Topic Identifier
Ad−Hoc Robust Monolingual French track − Comparison to Median Mean Average Precision by Topic (Topics 131 to 165)
062 41.47 097 25.10 142 100.00 177 31.54 1
uhifrmo1
063 22.28 098 3.66 143 7.50 178 12.96 0.5
Difference
064 0.00 099 4.75 144 63.89 179 0.00
0
065 0.48 110 61.72 145 45.88 190 42.49
−0.5
066 89.61 111 2.66 146 0.00 191 0.00
−1
067 45.29 112 32.17 147 2.96 192 37.48 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 160 161 162 163 164 165
Topic Identifier
068 49.80 113 6.40 148 3.31 193 36.78 Ad−Hoc Robust Monolingual French track − Comparison to Median Mean Average Precision by Topic (Topics 166 to 200)
1
069 55.25 114 25.07 149 87.62 194 0.00 uhifrmo1
0.5
080 6.67 115 54.34 160 0.00 195 1.22
Difference
081 58.19 116 26.61 161 0.00 196 100.00 0
082 17.80 117 0.54 162 43.61 197 81.64 −0.5
083 36.67 118 2.49 163 15.71 198 50.61 −1
166 167 168 169 170 171 172 173 174 175 176 177 178 179 190 191 192 193 194 195 196 197 198 199 200
084 33.33 119 42.78 164 21.07 199 56.93 Topic Identifier
085 25.20 130 58.03 165 39.29 200 1.66
185
hildesheim uhifrmo1 AH-ROBUST-MONO-FR-CLEF2006
Docs Cutoff Levels Precision at DCL (%)
Ad−Hoc Robust Monolingual French track − Retrieved documents vs Precision
100%
5 docs 28.00
uhifrmo1
10 docs 24.70 90%
15 docs 22.73
80%
20 docs 21.20
30 docs 18.20 70%
100 docs 10.48
60%
200 docs 6.70
R−Precision
500 docs 3.31 50%
1000 docs 1.83 40%
R-Precision (precision after R document retrieved,
where R = Relevant retrieved) 30%
27.68
20%
10%
0%
5 10 15 20 30 100 200 500 1000
Retrieved Documents (logarithmic scale)
Exact R-Precision Ad−Hoc Robust Monolingual French track − Box plot of the Topics of the Experiment
Maximum 1.0000
Minimum 0.0000
First Quartile 0.0000
Second Quartile 0.2033
Third Quartile 0.4721
Interquartile range 0.4721
Mean 0.2768
Standard Deviation 0.3010
Lower Outlier Threshold 0.0000
Upper Outlier Threshold 1.0000 0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Exact R−Precision
Mean With No Outliers 0.2768
Std With No Outliers 0.3010
Ad−Hoc Robust Monolingual French track − Distribution of the Topics of the Experiment
40
Number of Topics of the Experiment
35
30
25
20
15
10
5
0
0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Exact R−Precision
Precision averages (%) for individual Ad−Hoc Robust Monolingual French track − Comparison to Median Mean Exact R−Precision by Topic (Topics 041 to 085)
queries 1
uhifrmo1
0.5
041 0.00 086 8.33 131 19.51 166 0.00
Difference
0
042 28.95 087 0.00 132 75.00 167 0.00
−0.5
043 0.00 088 58.06 133 0.00 168 7.69
044 0.00 089 65.22 134 33.33 169 0.00 −1
041 042 043 044 045 046 047 048 049 060 061 062 063 064 065 066 067 068 069 080 081 082 083 084 085
Topic Identifier
045 27.66 090 0.00 135 9.62 170 0.00 Ad−Hoc Robust Monolingual French track − Comparison to Median Mean Exact R−Precision by Topic (Topics 086 to 130)
1
046 0.00 091 0.00 136 100.00 171 33.33 uhifrmo1
047 21.15 092 27.08 137 0.00 172 0.00 0.5
Difference
048 0.00 093 80.00 138 25.49 173 100.00 0
049 0.00 094 43.75 139 8.89 174 100.00 −0.5
060 27.78 095 34.46 140 25.00 175 100.00 −1
086 087 088 089 090 091 092 093 094 095 096 097 098 099 110 111 112 113 114 115 116 117 118 119 130
061 8.33 096 0.00 141 0.00 176 84.21 Topic Identifier
Ad−Hoc Robust Monolingual French track − Comparison to Median Mean Exact R−Precision by Topic (Topics 131 to 165)
062 50.00 097 14.29 142 100.00 177 28.57 1
uhifrmo1
063 17.65 098 11.11 143 14.29 178 0.00 0.5
Difference
064 0.00 099 5.88 144 66.67 179 0.00
0
065 0.00 110 69.23 145 42.86 190 47.06
−0.5
066 82.35 111 0.00 146 0.00 191 0.00
−1
067 47.37 112 33.33 147 12.50 192 47.37 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 160 161 162 163 164 165
Topic Identifier
068 56.00 113 0.00 148 0.00 193 39.02 Ad−Hoc Robust Monolingual French track − Comparison to Median Mean Exact R−Precision by Topic (Topics 166 to 200)
1
069 50.00 114 26.67 149 60.00 194 0.00 uhifrmo1
0.5
080 3.23 115 40.00 160 0.00 195 5.26
Difference
081 52.94 116 40.00 161 0.00 196 100.00 0
082 31.25 117 0.00 162 40.00 197 77.10 −0.5
083 0.00 118 0.00 163 9.09 198 33.33 −1
166 167 168 169 170 171 172 173 174 175 176 177 178 179 190 191 192 193 194 195 196 197 198 199 200
084 0.00 119 43.75 164 29.21 199 48.48 Topic Identifier
085 30.00 130 53.85 165 50.00 200 5.56
186
hildesheim uhifrmo1 AH-ROBUST-MONO-FR-CLEF2006
Overall statistics for 100 queries : Priority 2
Total number of documents over all queries Query Construction AUTOMATIC
Retrieved 99,000 Source Language French
Relevant 2,190 Topic Fields title, description
Relevant retrieved 1,833 Pooled false
Geometric Mean Average Precision 0.0576 Lucene stemmer, BRF 5, 30
Binary Preference (BPREF) 0.3192
Interploated Recall (%) Precision Averages (%)
Ad−Hoc Robust Monolingual French track − Interpolated Recall vs Average Precision
100%
0 46.53
uhifrmo1
10 42.06 90%
20 38.84
80%
30 35.91
40 32.77 70%
50 30.43
Average Precision
60%
60 27.64
70 24.17 50%
80 21.02 40%
90 16.96
30%
100 13.55
Average precision (non-interpolated) for all 20%
relevant documents (averaged over queries)
28.50 10%
0%
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
Interpolated Recall
Mean Average Precision Ad−Hoc Robust Monolingual French track − Box plot of the Topics of the Experiment
Maximum 1.0000
Minimum 0.0000
First Quartile 0.0340
Second Quartile 0.1814
Third Quartile 0.4550
Interquartile range 0.4211
Mean 0.2850
Standard Deviation 0.2993
Lower Outlier Threshold 0.0000
Upper Outlier Threshold 1.0000 0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Mean Average Precision
Mean With No Outliers 0.2850
Std With No Outliers 0.2993
Ad−Hoc Robust Monolingual French track − Distribution of the Topics of the Experiment
30
Number of Topics of the Experiment
25
20
15
10
5
0
0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Mean Average Precision
Precision averages (%) for individual Ad−Hoc Robust Monolingual French track − Comparison to Median Mean Average Precision by Topic (Topics 041 to 085)
queries 1
uhifrmo1
0.5
041 5.80 086 8.36 131 13.91 166 0.00
Difference
0
042 19.76 087 0.37 132 62.20 167 10.16
−0.5
043 0.81 088 68.63 133 1.77 168 9.24
044 18.48 089 59.47 134 21.78 169 0.00 −1
041 042 043 044 045 046 047 048 049 060 061 062 063 064 065 066 067 068 069 080 081 082 083 084 085
Topic Identifier
045 26.99 090 0.35 135 5.25 170 11.11 Ad−Hoc Robust Monolingual French track − Comparison to Median Mean Average Precision by Topic (Topics 086 to 130)
1
046 3.48 091 3.58 136 100.00 171 17.50 uhifrmo1
047 12.74 092 19.08 137 0.14 172 0.00 0.5
Difference
048 1.27 093 85.00 138 14.60 173 100.00 0
049 0.46 094 45.72 139 6.44 174 100.00 −0.5
060 17.48 095 24.74 140 16.23 175 100.00 −1
086 087 088 089 090 091 092 093 094 095 096 097 098 099 110 111 112 113 114 115 116 117 118 119 130
061 4.75 096 12.38 141 25.00 176 82.81 Topic Identifier
Ad−Hoc Robust Monolingual French track − Comparison to Median Mean Average Precision by Topic (Topics 131 to 165)
062 41.47 097 25.10 142 100.00 177 31.54 1
uhifrmo1
063 22.28 098 3.66 143 7.50 178 12.96 0.5
Difference
064 0.00 099 4.75 144 63.89 179 0.00
0
065 0.48 110 61.72 145 45.88 190 42.49
−0.5
066 89.61 111 2.66 146 0.00 191 0.00
−1
067 45.29 112 32.17 147 2.96 192 37.48 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 160 161 162 163 164 165
Topic Identifier
068 49.80 113 6.40 148 3.31 193 36.78 Ad−Hoc Robust Monolingual French track − Comparison to Median Mean Average Precision by Topic (Topics 166 to 200)
1
069 55.25 114 25.07 149 87.62 194 0.00 uhifrmo1
0.5
080 6.67 115 54.34 160 0.00 195 1.22
Difference
081 58.19 116 26.61 161 0.00 196 100.00 0
082 17.80 117 0.54 162 43.61 197 81.64 −0.5
083 36.67 118 2.49 163 15.71 198 50.61 −1
166 167 168 169 170 171 172 173 174 175 176 177 178 179 190 191 192 193 194 195 196 197 198 199 200
084 33.33 119 42.78 164 21.07 199 56.93 Topic Identifier
085 25.20 130 58.03 165 39.29 200 1.66
187
hildesheim uhifrmo1 AH-ROBUST-MONO-FR-CLEF2006
Docs Cutoff Levels Precision at DCL (%)
Ad−Hoc Robust Monolingual French track − Retrieved documents vs Precision
100%
5 docs 28.00
uhifrmo1
10 docs 24.70 90%
15 docs 22.73
80%
20 docs 21.20
30 docs 18.20 70%
100 docs 10.48
60%
200 docs 6.70
R−Precision
500 docs 3.31 50%
1000 docs 1.83 40%
R-Precision (precision after R document retrieved,
where R = Relevant retrieved) 30%
27.68
20%
10%
0%
5 10 15 20 30 100 200 500 1000
Retrieved Documents (logarithmic scale)
Exact R-Precision Ad−Hoc Robust Monolingual French track − Box plot of the Topics of the Experiment
Maximum 1.0000
Minimum 0.0000
First Quartile 0.0000
Second Quartile 0.2033
Third Quartile 0.4721
Interquartile range 0.4721
Mean 0.2768
Standard Deviation 0.3010
Lower Outlier Threshold 0.0000
Upper Outlier Threshold 1.0000 0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Exact R−Precision
Mean With No Outliers 0.2768
Std With No Outliers 0.3010
Ad−Hoc Robust Monolingual French track − Distribution of the Topics of the Experiment
40
Number of Topics of the Experiment
35
30
25
20
15
10
5
0
0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Exact R−Precision
Precision averages (%) for individual Ad−Hoc Robust Monolingual French track − Comparison to Median Mean Exact R−Precision by Topic (Topics 041 to 085)
queries 1
uhifrmo1
0.5
041 0.00 086 8.33 131 19.51 166 0.00
Difference
0
042 28.95 087 0.00 132 75.00 167 0.00
−0.5
043 0.00 088 58.06 133 0.00 168 7.69
044 0.00 089 65.22 134 33.33 169 0.00 −1
041 042 043 044 045 046 047 048 049 060 061 062 063 064 065 066 067 068 069 080 081 082 083 084 085
Topic Identifier
045 27.66 090 0.00 135 9.62 170 0.00 Ad−Hoc Robust Monolingual French track − Comparison to Median Mean Exact R−Precision by Topic (Topics 086 to 130)
1
046 0.00 091 0.00 136 100.00 171 33.33 uhifrmo1
047 21.15 092 27.08 137 0.00 172 0.00 0.5
Difference
048 0.00 093 80.00 138 25.49 173 100.00 0
049 0.00 094 43.75 139 8.89 174 100.00 −0.5
060 27.78 095 34.46 140 25.00 175 100.00 −1
086 087 088 089 090 091 092 093 094 095 096 097 098 099 110 111 112 113 114 115 116 117 118 119 130
061 8.33 096 0.00 141 0.00 176 84.21 Topic Identifier
Ad−Hoc Robust Monolingual French track − Comparison to Median Mean Exact R−Precision by Topic (Topics 131 to 165)
062 50.00 097 14.29 142 100.00 177 28.57 1
uhifrmo1
063 17.65 098 11.11 143 14.29 178 0.00 0.5
Difference
064 0.00 099 5.88 144 66.67 179 0.00
0
065 0.00 110 69.23 145 42.86 190 47.06
−0.5
066 82.35 111 0.00 146 0.00 191 0.00
−1
067 47.37 112 33.33 147 12.50 192 47.37 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 160 161 162 163 164 165
Topic Identifier
068 56.00 113 0.00 148 0.00 193 39.02 Ad−Hoc Robust Monolingual French track − Comparison to Median Mean Exact R−Precision by Topic (Topics 166 to 200)
1
069 50.00 114 26.67 149 60.00 194 0.00 uhifrmo1
0.5
080 3.23 115 40.00 160 0.00 195 5.26
Difference
081 52.94 116 40.00 161 0.00 196 100.00 0
082 31.25 117 0.00 162 40.00 197 77.10 −0.5
083 0.00 118 0.00 163 9.09 198 33.33 −1
166 167 168 169 170 171 172 173 174 175 176 177 178 179 190 191 192 193 194 195 196 197 198 199 200
084 0.00 119 43.75 164 29.21 199 48.48 Topic Identifier
085 30.00 130 53.85 165 50.00 200 5.56
188
hummingbird humFR06Rtde AH-ROBUST-MONO-FR-CLEF2006
Overall statistics for 100 queries : Priority 2
Total number of documents over all queries Query Construction AUTOMATIC
Retrieved 100,000 Source Language French
Relevant 2,190 Topic Fields title, description
Relevant retrieved 2,032 Pooled false
Geometric Mean Average Precision 0.1490 same as humFR06Rtd except expansion from blind
Binary Preference (BPREF) 0.4818 feedback also used
Interploated Recall (%) Precision Averages (%)
Ad−Hoc Robust Monolingual French track − Interpolated Recall vs Average Precision
100%
0 70.48
humFR06Rtde
10 63.94 90%
20 59.37
80%
30 54.50
40 49.57 70%
50 47.13
Average Precision
60%
60 43.83
70 39.26 50%
80 35.08 40%
90 29.35
30%
100 23.65
Average precision (non-interpolated) for all 20%
relevant documents (averaged over queries)
45.43 10%
0%
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
Interpolated Recall
Mean Average Precision Ad−Hoc Robust Monolingual French track − Box plot of the Topics of the Experiment
Maximum 1.0000
Minimum 0.0000
First Quartile 0.2201
Second Quartile 0.4430
Third Quartile 0.6629
Interquartile range 0.4429
Mean 0.4543
Standard Deviation 0.3145
Lower Outlier Threshold 0.0000
Upper Outlier Threshold 1.0000 0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Mean Average Precision
Mean With No Outliers 0.4543
Std With No Outliers 0.3145
Ad−Hoc Robust Monolingual French track − Distribution of the Topics of the Experiment
15
Number of Topics of the Experiment
10
5
0
0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Mean Average Precision
Precision averages (%) for individual Ad−Hoc Robust Monolingual French track − Comparison to Median Mean Average Precision by Topic (Topics 041 to 085)
queries 1
humFR06Rtde
0.5
041 100.00 086 18.71 131 35.52 166 0.00
Difference
0
042 52.15 087 45.81 132 51.60 167 83.33
−0.5
043 100.00 088 67.43 133 31.08 168 24.09
044 19.60 089 46.66 134 86.67 169 0.00 −1
041 042 043 044 045 046 047 048 049 060 061 062 063 064 065 066 067 068 069 080 081 082 083 084 085
Topic Identifier
045 43.01 090 13.18 135 62.90 170 33.33 Ad−Hoc Robust Monolingual French track − Comparison to Median Mean Average Precision by Topic (Topics 086 to 130)
1
046 13.67 091 12.72 136 100.00 171 22.56 humFR06Rtde
047 29.22 092 30.15 137 9.78 172 0.00 0.5
Difference
048 1.51 093 100.00 138 45.53 173 100.00 0
049 22.18 094 56.05 139 25.72 174 100.00 −0.5
060 32.53 095 26.17 140 65.61 175 100.00 −1
086 087 088 089 090 091 092 093 094 095 096 097 098 099 110 111 112 113 114 115 116 117 118 119 130
061 6.85 096 36.82 141 50.00 176 97.69 Topic Identifier
Ad−Hoc Robust Monolingual French track − Comparison to Median Mean Average Precision by Topic (Topics 131 to 165)
062 58.34 097 28.14 142 100.00 177 66.98 1
humFR06Rtde
063 51.40 098 93.61 143 46.73 178 46.67 0.5
Difference
064 0.00 099 21.83 144 100.00 179 20.21
0
065 30.19 110 57.72 145 28.39 190 38.00
−0.5
066 94.22 111 3.54 146 0.00 191 0.00
−1
067 44.85 112 45.02 147 43.54 192 71.81 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 160 161 162 163 164 165
Topic Identifier
068 35.51 113 2.00 148 6.55 193 48.52 Ad−Hoc Robust Monolingual French track − Comparison to Median Mean Average Precision by Topic (Topics 166 to 200)
1
069 43.46 114 45.08 149 100.00 194 0.00 humFR06Rtde
0.5
080 14.64 115 62.75 160 0.00 195 33.29
Difference
081 67.50 116 43.75 161 0.00 196 100.00 0
082 89.13 117 26.83 162 76.63 197 85.14 −0.5
083 58.33 118 10.52 163 54.06 198 45.50 −1
166 167 168 169 170 171 172 173 174 175 176 177 178 179 190 191 192 193 194 195 196 197 198 199 200
084 50.00 119 81.49 164 39.54 199 83.89 Topic Identifier
085 34.13 130 63.93 165 45.00 200 5.12
189
hummingbird humFR06Rtde AH-ROBUST-MONO-FR-CLEF2006
Docs Cutoff Levels Precision at DCL (%)
Ad−Hoc Robust Monolingual French track − Retrieved documents vs Precision
100%
5 docs 46.40
humFR06Rtde
10 docs 39.10 90%
15 docs 35.27
80%
20 docs 31.85
30 docs 26.93 70%
100 docs 14.55
60%
200 docs 9.02
R−Precision
500 docs 4.00 50%
1000 docs 2.03 40%
R-Precision (precision after R document retrieved,
where R = Relevant retrieved) 30%
42.77
20%
10%
0%
5 10 15 20 30 100 200 500 1000
Retrieved Documents (logarithmic scale)
Exact R-Precision Ad−Hoc Robust Monolingual French track − Box plot of the Topics of the Experiment
Maximum 1.0000
Minimum 0.0000
First Quartile 0.1716
Second Quartile 0.4089
Third Quartile 0.6283
Interquartile range 0.4567
Mean 0.4277
Standard Deviation 0.3109
Lower Outlier Threshold 0.0000
Upper Outlier Threshold 1.0000 0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Exact R−Precision
Mean With No Outliers 0.4277
Std With No Outliers 0.3109
Ad−Hoc Robust Monolingual French track − Distribution of the Topics of the Experiment
20
Number of Topics of the Experiment
15
10
5
0
0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Exact R−Precision
Precision averages (%) for individual Ad−Hoc Robust Monolingual French track − Comparison to Median Mean Exact R−Precision by Topic (Topics 041 to 085)
queries 1
humFR06Rtde
0.5
041 100.00 086 13.89 131 37.80 166 0.00
Difference
0
042 51.32 087 57.89 132 25.00 167 50.00
−0.5
043 100.00 088 54.84 133 37.50 168 30.77
044 16.67 089 52.17 134 66.67 169 0.00 −1
041 042 043 044 045 046 047 048 049 060 061 062 063 064 065 066 067 068 069 080 081 082 083 084 085
Topic Identifier
045 44.68 090 17.65 135 65.38 170 0.00 Ad−Hoc Robust Monolingual French track − Comparison to Median Mean Exact R−Precision by Topic (Topics 086 to 130)
1
046 14.29 091 23.08 136 100.00 171 33.33 humFR06Rtde
047 28.85 092 35.42 137 0.00 172 0.00 0.5
Difference
048 0.00 093 100.00 138 45.10 173 100.00 0
049 33.33 094 62.50 139 28.89 174 100.00 −0.5
060 41.11 095 40.68 140 63.28 175 100.00 −1
086 087 088 089 090 091 092 093 094 095 096 097 098 099 110 111 112 113 114 115 116 117 118 119 130
061 8.33 096 35.71 141 0.00 176 89.47 Topic Identifier
Ad−Hoc Robust Monolingual French track − Comparison to Median Mean Exact R−Precision by Topic (Topics 131 to 165)
062 62.50 097 42.86 142 100.00 177 57.14 1
humFR06Rtde
063 58.82 098 77.78 143 46.43 178 33.33 0.5
Difference
064 0.00 099 35.29 144 100.00 179 12.50
0
065 23.08 110 66.67 145 21.43 190 47.06
−0.5
066 86.27 111 0.00 146 0.00 191 0.00
−1
067 52.63 112 46.67 147 25.00 192 63.16 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 160 161 162 163 164 165
Topic Identifier
068 36.00 113 0.00 148 0.00 193 41.46 Ad−Hoc Robust Monolingual French track − Comparison to Median Mean Exact R−Precision by Topic (Topics 166 to 200)
1
069 43.75 114 40.00 149 100.00 194 0.00 humFR06Rtde
0.5
080 9.68 115 60.00 160 0.00 195 36.84
Difference
081 58.82 116 40.00 161 0.00 196 100.00 0
082 81.25 117 25.00 162 76.00 197 75.57 −0.5
083 50.00 118 16.67 163 45.45 198 33.33 −1
166 167 168 169 170 171 172 173 174 175 176 177 178 179 190 191 192 193 194 195 196 197 198 199 200
084 0.00 119 73.75 164 37.08 199 69.70 Topic Identifier
085 43.33 130 69.23 165 50.00 200 0.00
190
hummingbird humFR06Rtd AH-ROBUST-MONO-FR-CLEF2006
Overall statistics for 100 queries : Priority 1
Total number of documents over all queries Query Construction AUTOMATIC
Retrieved 100,000 Source Language French
Relevant 2,190 Topic Fields title, description
Relevant retrieved 2,027 Pooled false
Geometric Mean Average Precision 0.1317 plain SearchServer stemming run
Binary Preference (BPREF) 0.4467
Interploated Recall (%) Precision Averages (%)
Ad−Hoc Robust Monolingual French track − Interpolated Recall vs Average Precision
100%
0 68.95
humFR06Rtd
10 62.58 90%
20 56.73
80%
30 51.29
40 46.72 70%
50 43.33
Average Precision
60%
60 39.42
70 35.65 50%
80 32.39 40%
90 27.34
30%
100 22.90
Average precision (non-interpolated) for all 20%
relevant documents (averaged over queries)
42.74 10%
0%
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
Interpolated Recall
Mean Average Precision Ad−Hoc Robust Monolingual French track − Box plot of the Topics of the Experiment
Maximum 1.0000
Minimum 0.0000
First Quartile 0.1861
Second Quartile 0.3474
Third Quartile 0.6445
Interquartile range 0.4584
Mean 0.4274
Standard Deviation 0.3318
Lower Outlier Threshold 0.0000
Upper Outlier Threshold 1.0000 0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Mean Average Precision
Mean With No Outliers 0.4274
Std With No Outliers 0.3318
Ad−Hoc Robust Monolingual French track − Distribution of the Topics of the Experiment
15
Number of Topics of the Experiment
10
5
0
0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Mean Average Precision
Precision averages (%) for individual Ad−Hoc Robust Monolingual French track − Comparison to Median Mean Average Precision by Topic (Topics 041 to 085)
queries 1
humFR06Rtd
0.5
041 100.00 086 17.83 131 34.99 166 0.00
Difference
0
042 45.01 087 40.38 132 39.09 167 83.33
−0.5
043 100.00 088 66.99 133 13.19 168 19.25
044 24.79 089 43.56 134 91.67 169 0.00 −1
041 042 043 044 045 046 047 048 049 060 061 062 063 064 065 066 067 068 069 080 081 082 083 084 085
Topic Identifier
045 48.70 090 12.01 135 41.66 170 33.33 Ad−Hoc Robust Monolingual French track − Comparison to Median Mean Average Precision by Topic (Topics 086 to 130)
1
046 4.91 091 7.18 136 100.00 171 23.81 humFR06Rtd
047 19.81 092 28.10 137 5.97 172 0.00 0.5
Difference
048 1.81 093 96.67 138 30.01 173 100.00 0
049 19.74 094 57.38 139 21.63 174 98.21 −0.5
060 32.08 095 21.13 140 57.75 175 100.00 −1
086 087 088 089 090 091 092 093 094 095 096 097 098 099 110 111 112 113 114 115 116 117 118 119 130
061 5.10 096 30.68 141 100.00 176 92.58 Topic Identifier
Ad−Hoc Robust Monolingual French track − Comparison to Median Mean Average Precision by Topic (Topics 131 to 165)
062 56.11 097 20.26 142 99.45 177 56.76 1
humFR06Rtd
063 35.73 098 92.80 143 42.82 178 41.67 0.5
Difference
064 0.00 099 17.97 144 100.00 179 25.84
0
065 23.71 110 60.15 145 19.73 190 25.60
−0.5
066 94.71 111 2.17 146 0.00 191 0.00
−1
067 32.29 112 41.26 147 40.91 192 49.67 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 160 161 162 163 164 165
Topic Identifier
068 23.33 113 1.96 148 9.27 193 51.71 Ad−Hoc Robust Monolingual French track − Comparison to Median Mean Average Precision by Topic (Topics 166 to 200)
1
069 33.81 114 34.49 149 87.62 194 0.00 humFR06Rtd
0.5
080 14.87 115 56.37 160 0.00 195 13.96
Difference
081 61.90 116 37.74 161 0.00 196 100.00 0
082 77.06 117 24.92 162 77.17 197 86.73 −0.5
083 100.00 118 8.59 163 53.90 198 22.66 −1
166 167 168 169 170 171 172 173 174 175 176 177 178 179 190 191 192 193 194 195 196 197 198 199 200
084 100.00 119 74.66 164 21.94 199 80.54 Topic Identifier
085 24.47 130 60.24 165 41.67 200 1.96
191
hummingbird humFR06Rtd AH-ROBUST-MONO-FR-CLEF2006
Docs Cutoff Levels Precision at DCL (%)
Ad−Hoc Robust Monolingual French track − Retrieved documents vs Precision
100%
5 docs 44.00
humFR06Rtd
10 docs 37.50 90%
15 docs 33.40
80%
20 docs 28.90
30 docs 24.33 70%
100 docs 13.01
60%
200 docs 8.24
R−Precision
500 docs 3.79 50%
1000 docs 2.03 40%
R-Precision (precision after R document retrieved,
where R = Relevant retrieved) 30%
41.97
20%
10%
0%
5 10 15 20 30 100 200 500 1000
Retrieved Documents (logarithmic scale)
Exact R-Precision Ad−Hoc Robust Monolingual French track − Box plot of the Topics of the Experiment
Maximum 1.0000
Minimum 0.0000
First Quartile 0.1716
Second Quartile 0.3594
Third Quartile 0.6259
Interquartile range 0.4543
Mean 0.4197
Standard Deviation 0.3097
Lower Outlier Threshold 0.0000
Upper Outlier Threshold 1.0000 0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Exact R−Precision
Mean With No Outliers 0.4197
Std With No Outliers 0.3097
Ad−Hoc Robust Monolingual French track − Distribution of the Topics of the Experiment
20
Number of Topics of the Experiment
15
10
5
0
0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Exact R−Precision
Precision averages (%) for individual Ad−Hoc Robust Monolingual French track − Comparison to Median Mean Exact R−Precision by Topic (Topics 041 to 085)
queries 1
humFR06Rtd
0.5
041 100.00 086 16.67 131 36.59 166 0.00
Difference
0
042 42.11 087 52.63 132 25.00 167 50.00
−0.5
043 100.00 088 58.06 133 18.75 168 23.08
044 16.67 089 52.17 134 66.67 169 0.00 −1
041 042 043 044 045 046 047 048 049 060 061 062 063 064 065 066 067 068 069 080 081 082 083 084 085
Topic Identifier
045 53.19 090 17.65 135 44.23 170 0.00 Ad−Hoc Robust Monolingual French track − Comparison to Median Mean Exact R−Precision by Topic (Topics 086 to 130)
1
046 14.29 091 15.38 136 100.00 171 33.33 humFR06Rtd
047 23.08 092 31.25 137 0.00 172 0.00 0.5
Difference
048 0.00 093 80.00 138 33.33 173 100.00 0
049 33.33 094 68.75 139 28.89 174 85.71 −0.5
060 38.89 095 31.07 140 55.47 175 100.00 −1
086 087 088 089 090 091 092 093 094 095 096 097 098 099 110 111 112 113 114 115 116 117 118 119 130
061 8.33 096 28.57 141 100.00 176 84.21 Topic Identifier
Ad−Hoc Robust Monolingual French track − Comparison to Median Mean Exact R−Precision by Topic (Topics 131 to 165)
062 50.00 097 28.57 142 92.31 177 42.86 1
humFR06Rtd
063 35.29 098 77.78 143 32.14 178 33.33 0.5
Difference
064 0.00 099 41.18 144 100.00 179 25.00
0
065 23.08 110 69.23 145 14.29 190 29.41
−0.5
066 88.24 111 0.00 146 0.00 191 0.00
−1
067 42.11 112 46.67 147 37.50 192 52.63 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 160 161 162 163 164 165
Topic Identifier
068 28.00 113 0.00 148 0.00 193 46.34 Ad−Hoc Robust Monolingual French track − Comparison to Median Mean Exact R−Precision by Topic (Topics 166 to 200)
1
069 43.75 114 33.33 149 60.00 194 0.00 humFR06Rtd
0.5
080 16.13 115 40.00 160 0.00 195 15.79
Difference
081 58.82 116 50.00 161 0.00 196 100.00 0
082 68.75 117 25.00 162 76.00 197 83.97 −0.5
083 100.00 118 16.67 163 63.64 198 33.33 −1
166 167 168 169 170 171 172 173 174 175 176 177 178 179 190 191 192 193 194 195 196 197 198 199 200
084 100.00 119 68.75 164 22.47 199 72.73 Topic Identifier
085 33.33 130 61.54 165 50.00 200 0.00
192
reina reinaFRtdtest AH-ROBUST-MONO-FR-CLEF2006
Overall statistics for 100 queries : Priority 1
Total number of documents over all queries Query Construction AUTOMATIC
Retrieved 100,000 Source Language French
Relevant 2,190 Topic Fields title, description
Relevant retrieved 2,088 Pooled false
Geometric Mean Average Precision 0.1432 local expansion with term windows, title+desc
Binary Preference (BPREF) 0.4708 fields
Interploated Recall (%) Precision Averages (%)
Ad−Hoc Robust Monolingual French track − Interpolated Recall vs Average Precision
100%
0 69.53
reinaFRtdtest
10 62.23 90%
20 58.20
80%
30 54.47
40 49.51 70%
50 46.32
Average Precision
60%
60 42.33
70 37.27 50%
80 33.26 40%
90 27.75
30%
100 22.44
Average precision (non-interpolated) for all 20%
relevant documents (averaged over queries)
44.58 10%
0%
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
Interpolated Recall
Mean Average Precision Ad−Hoc Robust Monolingual French track − Box plot of the Topics of the Experiment
Maximum 1.0000
Minimum 0.0000
First Quartile 0.1818
Second Quartile 0.4108
Third Quartile 0.6787
Interquartile range 0.4970
Mean 0.4458
Standard Deviation 0.3292
Lower Outlier Threshold 0.0000
Upper Outlier Threshold 1.0000 0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Mean Average Precision
Mean With No Outliers 0.4458
Std With No Outliers 0.3292
Ad−Hoc Robust Monolingual French track − Distribution of the Topics of the Experiment
15
Number of Topics of the Experiment
10
5
0
0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Mean Average Precision
Precision averages (%) for individual Ad−Hoc Robust Monolingual French track − Comparison to Median Mean Average Precision by Topic (Topics 041 to 085)
queries 1
reinaFRtdtest
0.5
041 100.00 086 18.88 131 32.31 166 0.00
Difference
0
042 46.01 087 54.73 132 47.50 167 64.29
−0.5
043 100.00 088 69.08 133 9.10 168 26.86
044 13.55 089 49.00 134 91.67 169 0.00 −1
041 042 043 044 045 046 047 048 049 060 061 062 063 064 065 066 067 068 069 080 081 082 083 084 085
Topic Identifier
045 47.93 090 13.34 135 55.54 170 20.00 Ad−Hoc Robust Monolingual French track − Comparison to Median Mean Average Precision by Topic (Topics 086 to 130)
1
046 8.50 091 5.25 136 100.00 171 18.73 reinaFRtdtest
047 20.41 092 27.58 137 11.94 172 0.00 0.5
Difference
048 1.94 093 100.00 138 22.16 173 100.00 0
049 23.68 094 59.36 139 23.46 174 100.00 −0.5
060 20.41 095 27.96 140 66.45 175 100.00 −1
086 087 088 089 090 091 092 093 094 095 096 097 098 099 110 111 112 113 114 115 116 117 118 119 130
061 3.31 096 38.80 141 100.00 176 98.04 Topic Identifier
Ad−Hoc Robust Monolingual French track − Comparison to Median Mean Average Precision by Topic (Topics 131 to 165)
062 69.26 097 28.86 142 98.56 177 25.31 1
reinaFRtdtest
063 40.04 098 85.62 143 33.02 178 66.67 0.5
Difference
064 0.00 099 12.19 144 100.00 179 23.26
0
065 28.01 110 55.15 145 51.10 190 22.20
−0.5
066 89.46 111 17.63 146 0.00 191 0.00
−1
067 43.35 112 42.06 147 48.08 192 61.07 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 160 161 162 163 164 165
Topic Identifier
068 17.08 113 6.81 148 6.88 193 49.29 Ad−Hoc Robust Monolingual French track − Comparison to Median Mean Average Precision by Topic (Topics 166 to 200)
1
069 43.16 114 33.33 149 92.67 194 0.00 reinaFRtdtest
0.5
080 13.75 115 53.57 160 0.00 195 40.63
Difference
081 64.73 116 32.42 161 0.00 196 100.00 0
082 82.14 117 32.39 162 79.98 197 84.05 −0.5
083 100.00 118 7.26 163 46.53 198 47.15 −1
166 167 168 169 170 171 172 173 174 175 176 177 178 179 190 191 192 193 194 195 196 197 198 199 200
084 100.00 119 81.03 164 41.54 199 74.09 Topic Identifier
085 31.34 130 62.80 165 45.00 200 9.37
193
reina reinaFRtdtest AH-ROBUST-MONO-FR-CLEF2006
Docs Cutoff Levels Precision at DCL (%)
Ad−Hoc Robust Monolingual French track − Retrieved documents vs Precision
100%
5 docs 44.00
reinaFRtdtest
10 docs 37.40 90%
15 docs 33.67
80%
20 docs 30.20
30 docs 25.83 70%
100 docs 13.81
60%
200 docs 8.49
R−Precision
500 docs 3.94 50%
1000 docs 2.09 40%
R-Precision (precision after R document retrieved,
where R = Relevant retrieved) 30%
44.23
20%
10%
0%
5 10 15 20 30 100 200 500 1000
Retrieved Documents (logarithmic scale)
Exact R-Precision Ad−Hoc Robust Monolingual French track − Box plot of the Topics of the Experiment
Maximum 1.0000
Minimum 0.0000
First Quartile 0.2265
Second Quartile 0.4277
Third Quartile 0.6202
Interquartile range 0.3937
Mean 0.4423
Standard Deviation 0.3091
Lower Outlier Threshold 0.0000
Upper Outlier Threshold 1.0000 0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Exact R−Precision
Mean With No Outliers 0.4423
Std With No Outliers 0.3091
Ad−Hoc Robust Monolingual French track − Distribution of the Topics of the Experiment
15
Number of Topics of the Experiment
10
5
0
0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Exact R−Precision
Precision averages (%) for individual Ad−Hoc Robust Monolingual French track − Comparison to Median Mean Exact R−Precision by Topic (Topics 041 to 085)
queries 1
reinaFRtdtest
0.5
041 100.00 086 16.67 131 42.68 166 0.00
Difference
0
042 44.74 087 52.63 132 50.00 167 50.00
−0.5
043 100.00 088 54.84 133 21.88 168 30.77
044 0.00 089 43.48 134 66.67 169 0.00 −1
041 042 043 044 045 046 047 048 049 060 061 062 063 064 065 066 067 068 069 080 081 082 083 084 085
Topic Identifier
045 46.81 090 23.53 135 59.62 170 0.00 Ad−Hoc Robust Monolingual French track − Comparison to Median Mean Exact R−Precision by Topic (Topics 086 to 130)
1
046 14.29 091 7.69 136 100.00 171 33.33 reinaFRtdtest
047 23.08 092 35.42 137 33.33 172 0.00 0.5
Difference
048 0.00 093 100.00 138 25.49 173 100.00 0
049 33.33 094 56.25 139 31.11 174 100.00 −0.5
060 31.11 095 35.59 140 63.28 175 100.00 −1
086 087 088 089 090 091 092 093 094 095 096 097 098 099 110 111 112 113 114 115 116 117 118 119 130
061 8.33 096 50.00 141 100.00 176 89.47 Topic Identifier
Ad−Hoc Robust Monolingual French track − Comparison to Median Mean Exact R−Precision by Topic (Topics 131 to 165)
062 62.50 097 28.57 142 92.31 177 28.57 1
reinaFRtdtest
063 47.06 098 77.78 143 28.57 178 66.67 0.5
Difference
064 0.00 099 17.65 144 100.00 179 12.50
0
065 30.77 110 61.54 145 42.86 190 29.41
−0.5
066 82.35 111 16.67 146 0.00 191 0.00
−1
067 47.37 112 46.67 147 50.00 192 57.89 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 160 161 162 163 164 165
Topic Identifier
068 24.00 113 0.00 148 0.00 193 46.34 Ad−Hoc Robust Monolingual French track − Comparison to Median Mean Exact R−Precision by Topic (Topics 166 to 200)
1
069 43.75 114 36.67 149 80.00 194 0.00 reinaFRtdtest
0.5
080 12.90 115 40.00 160 0.00 195 47.37
Difference
081 58.82 116 40.00 161 0.00 196 100.00 0
082 75.00 117 28.57 162 76.00 197 76.34 −0.5
083 100.00 118 16.67 163 36.36 198 33.33 −1
166 167 168 169 170 171 172 173 174 175 176 177 178 179 190 191 192 193 194 195 196 197 198 199 200
084 100.00 119 77.50 164 46.07 199 57.58 Topic Identifier
085 33.33 130 61.54 165 50.00 200 22.22
194
reina reinaFRttest AH-ROBUST-MONO-FR-CLEF2006
Overall statistics for 100 queries : Priority 2
Total number of documents over all queries Query Construction AUTOMATIC
Retrieved 100,000 Source Language French
Relevant 2,190 Topic Fields title
Relevant retrieved 2,073 Pooled false
Geometric Mean Average Precision 0.1404 local expansion with term windows, only title field
Binary Preference (BPREF) 0.4668
Interploated Recall (%) Precision Averages (%)
Ad−Hoc Robust Monolingual French track − Interpolated Recall vs Average Precision
100%
0 70.06
reinaFRttest
10 62.31 90%
20 58.52
80%
30 52.76
40 47.42 70%
50 44.69
Average Precision
60%
60 41.36
70 37.12 50%
80 33.75 40%
90 27.98
30%
100 23.44
Average precision (non-interpolated) for all 20%
relevant documents (averaged over queries)
44.15 10%
0%
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
Interpolated Recall
Mean Average Precision Ad−Hoc Robust Monolingual French track − Box plot of the Topics of the Experiment
Maximum 1.0000
Minimum 0.0000
First Quartile 0.1633
Second Quartile 0.3782
Third Quartile 0.6626
Interquartile range 0.4993
Mean 0.4415
Standard Deviation 0.3307
Lower Outlier Threshold 0.0000
Upper Outlier Threshold 1.0000 0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Mean Average Precision
Mean With No Outliers 0.4415
Std With No Outliers 0.3307
Ad−Hoc Robust Monolingual French track − Distribution of the Topics of the Experiment
15
Number of Topics of the Experiment
10
5
0
0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Mean Average Precision
Precision averages (%) for individual Ad−Hoc Robust Monolingual French track − Comparison to Median Mean Average Precision by Topic (Topics 041 to 085)
queries 1
reinaFRttest
0.5
041 100.00 086 18.31 131 30.83 166 0.00
Difference
0
042 46.02 087 49.12 132 66.67 167 83.33
−0.5
043 100.00 088 65.49 133 9.01 168 22.32
044 15.99 089 50.70 134 83.33 169 0.00 −1
041 042 043 044 045 046 047 048 049 060 061 062 063 064 065 066 067 068 069 080 081 082 083 084 085
Topic Identifier
045 51.24 090 11.75 135 49.27 170 16.67 Ad−Hoc Robust Monolingual French track − Comparison to Median Mean Average Precision by Topic (Topics 086 to 130)
1
046 8.39 091 4.49 136 100.00 171 15.37 reinaFRttest
047 20.91 092 29.55 137 12.06 172 0.00 0.5
Difference
048 2.14 093 100.00 138 23.92 173 100.00 0
049 24.46 094 65.13 139 20.93 174 100.00 −0.5
060 20.14 095 28.02 140 60.64 175 100.00 −1
086 087 088 089 090 091 092 093 094 095 096 097 098 099 110 111 112 113 114 115 116 117 118 119 130
061 5.16 096 37.32 141 100.00 176 97.89 Topic Identifier
Ad−Hoc Robust Monolingual French track − Comparison to Median Mean Average Precision by Topic (Topics 131 to 165)
062 64.50 097 29.62 142 98.97 177 43.93 1
reinaFRttest
063 39.98 098 95.00 143 34.71 178 55.56 0.5
Difference
064 0.00 099 12.97 144 100.00 179 22.04
0
065 27.50 110 54.79 145 41.14 190 22.41
−0.5
066 90.44 111 18.45 146 0.00 191 0.00
−1
067 32.46 112 40.89 147 35.95 192 54.17 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 160 161 162 163 164 165
Topic Identifier
068 14.55 113 6.32 148 6.14 193 50.18 Ad−Hoc Robust Monolingual French track − Comparison to Median Mean Average Precision by Topic (Topics 166 to 200)
1
069 45.24 114 34.18 149 92.67 194 0.00 reinaFRttest
0.5
080 13.91 115 52.64 160 0.00 195 37.87
Difference
081 65.85 116 33.12 161 0.00 196 100.00 0
082 69.02 117 30.98 162 82.28 197 85.29 −0.5
083 100.00 118 5.93 163 45.33 198 36.64 −1
166 167 168 169 170 171 172 173 174 175 176 177 178 179 190 191 192 193 194 195 196 197 198 199 200
084 100.00 119 77.96 164 41.07 199 75.78 Topic Identifier
085 37.77 130 63.05 165 45.00 200 3.73
195
reina reinaFRttest AH-ROBUST-MONO-FR-CLEF2006
Docs Cutoff Levels Precision at DCL (%)
Ad−Hoc Robust Monolingual French track − Retrieved documents vs Precision
100%
5 docs 44.00
reinaFRttest
10 docs 36.60 90%
15 docs 32.27
80%
20 docs 29.20
30 docs 25.40 70%
100 docs 13.80
60%
200 docs 8.37
R−Precision
500 docs 3.93 50%
1000 docs 2.07 40%
R-Precision (precision after R document retrieved,
where R = Relevant retrieved) 30%
43.52
20%
10%
0%
5 10 15 20 30 100 200 500 1000
Retrieved Documents (logarithmic scale)
Exact R-Precision Ad−Hoc Robust Monolingual French track − Box plot of the Topics of the Experiment
Maximum 1.0000
Minimum 0.0000
First Quartile 0.1882
Second Quartile 0.3890
Third Quartile 0.6387
Interquartile range 0.4505
Mean 0.4352
Standard Deviation 0.3160
Lower Outlier Threshold 0.0000
Upper Outlier Threshold 1.0000 0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Exact R−Precision
Mean With No Outliers 0.4352
Std With No Outliers 0.3160
Ad−Hoc Robust Monolingual French track − Distribution of the Topics of the Experiment
20
Number of Topics of the Experiment
15
10
5
0
0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Exact R−Precision
Precision averages (%) for individual Ad−Hoc Robust Monolingual French track − Comparison to Median Mean Exact R−Precision by Topic (Topics 041 to 085)
queries 1
reinaFRttest
0.5
041 100.00 086 16.67 131 37.80 166 0.00
Difference
0
042 44.74 087 47.37 132 50.00 167 50.00
−0.5
043 100.00 088 54.84 133 21.88 168 30.77
044 0.00 089 43.48 134 66.67 169 0.00 −1
041 042 043 044 045 046 047 048 049 060 061 062 063 064 065 066 067 068 069 080 081 082 083 084 085
Topic Identifier
045 51.06 090 17.65 135 53.85 170 0.00 Ad−Hoc Robust Monolingual French track − Comparison to Median Mean Exact R−Precision by Topic (Topics 086 to 130)
1
046 14.29 091 7.69 136 100.00 171 0.00 reinaFRttest
047 28.85 092 35.42 137 33.33 172 0.00 0.5
Difference
048 0.00 093 100.00 138 25.49 173 100.00 0
049 33.33 094 62.50 139 26.67 174 100.00 −0.5
060 30.00 095 36.72 140 55.47 175 100.00 −1
086 087 088 089 090 091 092 093 094 095 096 097 098 099 110 111 112 113 114 115 116 117 118 119 130
061 8.33 096 42.86 141 100.00 176 89.47 Topic Identifier
Ad−Hoc Robust Monolingual French track − Comparison to Median Mean Exact R−Precision by Topic (Topics 131 to 165)
062 62.50 097 42.86 142 92.31 177 28.57 1
reinaFRttest
063 47.06 098 77.78 143 32.14 178 66.67 0.5
Difference
064 0.00 099 17.65 144 100.00 179 25.00
0
065 30.77 110 64.10 145 35.71 190 35.29
−0.5
066 82.35 111 16.67 146 0.00 191 0.00
−1
067 36.84 112 40.00 147 50.00 192 57.89 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 160 161 162 163 164 165
Topic Identifier
068 20.00 113 0.00 148 0.00 193 46.34 Ad−Hoc Robust Monolingual French track − Comparison to Median Mean Exact R−Precision by Topic (Topics 166 to 200)
1
069 37.50 114 36.67 149 80.00 194 0.00 reinaFRttest
0.5
080 16.13 115 40.00 160 0.00 195 36.84
Difference
081 70.59 116 40.00 161 0.00 196 100.00 0
082 75.00 117 25.00 162 76.00 197 79.39 −0.5
083 100.00 118 0.00 163 54.55 198 33.33 −1
166 167 168 169 170 171 172 173 174 175 176 177 178 179 190 191 192 193 194 195 196 197 198 199 200
084 100.00 119 75.00 164 46.07 199 63.64 Topic Identifier
085 36.67 130 53.85 165 50.00 200 0.00
196
unine UniNEfrr1 AH-ROBUST-MONO-FR-CLEF2006
Overall statistics for 100 queries : Priority 1
Total number of documents over all queries Query Construction AUTOMATIC
Retrieved 100,000 Source Language French
Relevant 2,190 Topic Fields title, description
Relevant retrieved 2,122 Pooled false
Geometric Mean Average Precision 0.1502 data fusion of Okapi, DFR & LM
Binary Preference (BPREF) 0.4980
Interploated Recall (%) Precision Averages (%)
Ad−Hoc Robust Monolingual French track − Interpolated Recall vs Average Precision
100%
0 70.05
UniNEfrr1
10 64.03 90%
20 59.40
80%
30 54.84
40 51.24 70%
50 49.00
Average Precision
60%
60 45.87
70 42.08 50%
80 39.95 40%
90 34.97
30%
100 28.29
Average precision (non-interpolated) for all 20%
relevant documents (averaged over queries)
47.57 10%
0%
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
Interpolated Recall
Mean Average Precision Ad−Hoc Robust Monolingual French track − Box plot of the Topics of the Experiment
Maximum 1.0000
Minimum 0.0000
First Quartile 0.1981
Second Quartile 0.4335
Third Quartile 0.7863
Interquartile range 0.5882
Mean 0.4757
Standard Deviation 0.3357
Lower Outlier Threshold 0.0000
Upper Outlier Threshold 1.0000 0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Mean Average Precision
Mean With No Outliers 0.4757
Std With No Outliers 0.3357
Ad−Hoc Robust Monolingual French track − Distribution of the Topics of the Experiment
15
Number of Topics of the Experiment
10
5
0
0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Mean Average Precision
Precision averages (%) for individual Ad−Hoc Robust Monolingual French track − Comparison to Median Mean Average Precision by Topic (Topics 041 to 085)
queries 1
UniNEfrr1
0.5
041 100.00 086 19.87 131 38.70 166 0.00
Difference
0
042 48.64 087 43.35 132 43.86 167 100.00
−0.5
043 100.00 088 76.12 133 20.08 168 20.89
044 35.85 089 63.24 134 86.67 169 0.00 −1
041 042 043 044 045 046 047 048 049 060 061 062 063 064 065 066 067 068 069 080 081 082 083 084 085
Topic Identifier
045 56.22 090 11.78 135 83.47 170 50.00 Ad−Hoc Robust Monolingual French track − Comparison to Median Mean Average Precision by Topic (Topics 086 to 130)
1
046 9.12 091 1.10 136 100.00 171 19.74 UniNEfrr1
047 13.19 092 37.65 137 19.76 172 0.00 0.5
Difference
048 1.33 093 100.00 138 56.93 173 100.00 0
049 14.74 094 65.86 139 19.20 174 100.00 −0.5
060 24.95 095 26.89 140 72.64 175 100.00 −1
086 087 088 089 090 091 092 093 094 095 096 097 098 099 110 111 112 113 114 115 116 117 118 119 130
061 5.91 096 27.30 141 100.00 176 98.56 Topic Identifier
Ad−Hoc Robust Monolingual French track − Comparison to Median Mean Average Precision by Topic (Topics 131 to 165)
062 39.11 097 43.55 142 100.00 177 37.35 1
UniNEfrr1
063 50.92 098 94.60 143 34.34 178 40.26 0.5
Difference
064 0.00 099 24.09 144 100.00 179 27.50
0
065 26.84 110 63.87 145 34.53 190 44.30
−0.5
066 88.05 111 6.80 146 0.00 191 0.00
−1
067 48.57 112 43.34 147 48.42 192 87.67 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 160 161 162 163 164 165
Topic Identifier
068 43.23 113 2.30 148 4.04 193 42.36 Ad−Hoc Robust Monolingual French track − Comparison to Median Mean Average Precision by Topic (Topics 166 to 200)
1
069 44.11 114 59.98 149 78.62 194 0.00 UniNEfrr1
0.5
080 14.03 115 69.27 160 0.00 195 51.00
Difference
081 66.15 116 28.21 161 0.00 196 100.00 0
082 92.28 117 25.77 162 78.71 197 85.74 −0.5
083 100.00 118 10.20 163 48.26 198 39.74 −1
166 167 168 169 170 171 172 173 174 175 176 177 178 179 190 191 192 193 194 195 196 197 198 199 200
084 100.00 119 86.42 164 51.91 199 78.65 Topic Identifier
085 30.42 130 71.96 165 41.67 200 13.89
197
unine UniNEfrr1 AH-ROBUST-MONO-FR-CLEF2006
Docs Cutoff Levels Precision at DCL (%)
Ad−Hoc Robust Monolingual French track − Retrieved documents vs Precision
100%
5 docs 43.20
UniNEfrr1
10 docs 37.80 90%
15 docs 34.20
80%
20 docs 31.55
30 docs 27.43 70%
100 docs 14.82
60%
200 docs 9.22
R−Precision
500 docs 4.12 50%
1000 docs 2.12 40%
R-Precision (precision after R document retrieved,
where R = Relevant retrieved) 30%
45.45
20%
10%
0%
5 10 15 20 30 100 200 500 1000
Retrieved Documents (logarithmic scale)
Exact R-Precision Ad−Hoc Robust Monolingual French track − Box plot of the Topics of the Experiment
Maximum 1.0000
Minimum 0.0000
First Quartile 0.1716
Second Quartile 0.4202
Third Quartile 0.7235
Interquartile range 0.5519
Mean 0.4545
Standard Deviation 0.3380
Lower Outlier Threshold 0.0000
Upper Outlier Threshold 1.0000 0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Exact R−Precision
Mean With No Outliers 0.4545
Std With No Outliers 0.3380
Ad−Hoc Robust Monolingual French track − Distribution of the Topics of the Experiment
20
Number of Topics of the Experiment
15
10
5
0
0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Exact R−Precision
Precision averages (%) for individual Ad−Hoc Robust Monolingual French track − Comparison to Median Mean Exact R−Precision by Topic (Topics 041 to 085)
queries 1
UniNEfrr1
0.5
041 100.00 086 16.67 131 36.59 166 0.00
Difference
0
042 51.32 087 47.37 132 25.00 167 100.00
−0.5
043 100.00 088 67.74 133 31.25 168 15.38
044 33.33 089 60.87 134 66.67 169 0.00 −1
041 042 043 044 045 046 047 048 049 060 061 062 063 064 065 066 067 068 069 080 081 082 083 084 085
Topic Identifier
045 51.06 090 17.65 135 78.85 170 0.00 Ad−Hoc Robust Monolingual French track − Comparison to Median Mean Exact R−Precision by Topic (Topics 086 to 130)
1
046 0.00 091 0.00 136 100.00 171 33.33 UniNEfrr1
047 9.62 092 45.83 137 0.00 172 0.00 0.5
Difference
048 0.00 093 100.00 138 54.90 173 100.00 0
049 25.00 094 75.00 139 11.11 174 100.00 −0.5
060 35.56 095 38.98 140 65.62 175 100.00 −1
086 087 088 089 090 091 092 093 094 095 096 097 098 099 110 111 112 113 114 115 116 117 118 119 130
061 0.00 096 21.43 141 100.00 176 89.47 Topic Identifier
Ad−Hoc Robust Monolingual French track − Comparison to Median Mean Exact R−Precision by Topic (Topics 131 to 165)
062 62.50 097 28.57 142 100.00 177 28.57 1
UniNEfrr1
063 58.82 098 88.89 143 42.86 178 33.33 0.5
Difference
064 0.00 099 23.53 144 100.00 179 25.00
0
065 23.08 110 69.23 145 28.57 190 41.18
−0.5
066 76.47 111 0.00 146 0.00 191 0.00
−1
067 57.89 112 40.00 147 50.00 192 78.95 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 160 161 162 163 164 165
Topic Identifier
068 48.00 113 0.00 148 0.00 193 36.59 Ad−Hoc Robust Monolingual French track − Comparison to Median Mean Exact R−Precision by Topic (Topics 166 to 200)
1
069 43.75 114 60.00 149 80.00 194 0.00 UniNEfrr1
0.5
080 9.68 115 60.00 160 0.00 195 47.37
Difference
081 58.82 116 30.00 161 0.00 196 100.00 0
082 87.50 117 25.00 162 80.00 197 78.63 −0.5
083 100.00 118 0.00 163 36.36 198 33.33 −1
166 167 168 169 170 171 172 173 174 175 176 177 178 179 190 191 192 193 194 195 196 197 198 199 200
084 100.00 119 80.00 164 55.06 199 69.70 Topic Identifier
085 26.67 130 69.23 165 50.00 200 16.67
198
unine UniNEfrr3 AH-ROBUST-MONO-FR-CLEF2006
Overall statistics for 100 queries : Priority 2
Total number of documents over all queries Query Construction AUTOMATIC
Retrieved 100,000 Source Language French
Relevant 2,190 Topic Fields title, description
Relevant retrieved 1,949 Pooled false
Geometric Mean Average Precision 0.1163 DFR (with Yahoo)
Binary Preference (BPREF) 0.4439
Interploated Recall (%) Precision Averages (%)
Ad−Hoc Robust Monolingual French track − Interpolated Recall vs Average Precision
100%
0 63.24
UniNEfrr3
10 57.65 90%
20 53.31
80%
30 50.29
40 47.13 70%
50 45.08
Average Precision
60%
60 42.38
70 35.52 50%
80 32.77 40%
90 26.69
30%
100 21.93
Average precision (non-interpolated) for all 20%
relevant documents (averaged over queries)
41.92 10%
0%
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
Interpolated Recall
Mean Average Precision Ad−Hoc Robust Monolingual French track − Box plot of the Topics of the Experiment
Maximum 1.0000
Minimum 0.0000
First Quartile 0.1210
Second Quartile 0.3946
Third Quartile 0.6711
Interquartile range 0.5501
Mean 0.4192
Standard Deviation 0.3242
Lower Outlier Threshold 0.0000
Upper Outlier Threshold 1.0000 0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Mean Average Precision
Mean With No Outliers 0.4192
Std With No Outliers 0.3242
Ad−Hoc Robust Monolingual French track − Distribution of the Topics of the Experiment
20
Number of Topics of the Experiment
15
10
5
0
0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Mean Average Precision
Precision averages (%) for individual Ad−Hoc Robust Monolingual French track − Comparison to Median Mean Average Precision by Topic (Topics 041 to 085)
queries 1
UniNEfrr3
0.5
041 100.00 086 21.06 131 32.35 166 0.00
Difference
0
042 52.78 087 42.15 132 43.47 167 50.00
−0.5
043 100.00 088 47.90 133 6.72 168 38.95
044 9.46 089 77.69 134 83.33 169 0.00 −1
041 042 043 044 045 046 047 048 049 060 061 062 063 064 065 066 067 068 069 080 081 082 083 084 085
Topic Identifier
045 48.04 090 11.70 135 1.43 170 12.50 Ad−Hoc Robust Monolingual French track − Comparison to Median Mean Average Precision by Topic (Topics 086 to 130)
1
046 6.43 091 0.22 136 100.00 171 44.00 UniNEfrr3
047 18.33 092 25.71 137 34.55 172 0.00 0.5
Difference
048 0.80 093 94.29 138 34.74 173 100.00 0
049 13.90 094 76.56 139 18.79 174 100.00 −0.5
060 0.24 095 11.37 140 59.81 175 100.00 −1
086 087 088 089 090 091 092 093 094 095 096 097 098 099 110 111 112 113 114 115 116 117 118 119 130
061 52.54 096 25.27 141 50.00 176 97.53 Topic Identifier
Ad−Hoc Robust Monolingual French track − Comparison to Median Mean Average Precision by Topic (Topics 131 to 165)
062 29.55 097 33.25 142 99.45 177 31.55 1
UniNEfrr3
063 41.93 098 93.61 143 45.68 178 67.11 0.5
Difference
064 0.00 099 24.64 144 80.95 179 46.38
0
065 24.39 110 67.11 145 39.22 190 65.22
−0.5
066 90.33 111 6.44 146 0.00 191 0.00
−1
067 50.99 112 40.39 147 41.26 192 90.61 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 160 161 162 163 164 165
Topic Identifier
068 55.81 113 3.16 148 2.93 193 39.69 Ad−Hoc Robust Monolingual French track − Comparison to Median Mean Average Precision by Topic (Topics 166 to 200)
1
069 55.08 114 21.58 149 87.62 194 0.00 UniNEfrr3
0.5
080 14.72 115 56.29 160 0.00 195 5.69
Difference
081 67.70 116 53.91 161 0.00 196 100.00 0
082 83.02 117 4.74 162 63.93 197 80.81 −0.5
083 22.22 118 71.37 163 22.23 198 67.54 −1
166 167 168 169 170 171 172 173 174 175 176 177 178 179 190 191 192 193 194 195 196 197 198 199 200
084 33.33 119 10.46 164 19.72 199 74.62 Topic Identifier
085 28.30 130 60.98 165 29.17 200 0.85
199
unine UniNEfrr3 AH-ROBUST-MONO-FR-CLEF2006
Docs Cutoff Levels Precision at DCL (%)
Ad−Hoc Robust Monolingual French track − Retrieved documents vs Precision
100%
5 docs 40.00
UniNEfrr3
10 docs 33.40 90%
15 docs 30.20
80%
20 docs 27.25
30 docs 23.43 70%
100 docs 12.28
60%
200 docs 7.69
R−Precision
500 docs 3.62 50%
1000 docs 1.95 40%
R-Precision (precision after R document retrieved,
where R = Relevant retrieved) 30%
38.77
20%
10%
0%
5 10 15 20 30 100 200 500 1000
Retrieved Documents (logarithmic scale)
Exact R-Precision Ad−Hoc Robust Monolingual French track − Box plot of the Topics of the Experiment
Maximum 1.0000
Minimum 0.0000
First Quartile 0.0222
Second Quartile 0.3798
Third Quartile 0.6435
Interquartile range 0.6213
Mean 0.3877
Standard Deviation 0.3193
Lower Outlier Threshold 0.0000
Upper Outlier Threshold 1.0000 0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Exact R−Precision
Mean With No Outliers 0.3877
Std With No Outliers 0.3193
Ad−Hoc Robust Monolingual French track − Distribution of the Topics of the Experiment
30
Number of Topics of the Experiment
25
20
15
10
5
0
0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Exact R−Precision
Precision averages (%) for individual Ad−Hoc Robust Monolingual French track − Comparison to Median Mean Exact R−Precision by Topic (Topics 041 to 085)
queries 1
UniNEfrr3
0.5
041 100.00 086 27.78 131 35.37 166 0.00
Difference
0
042 56.58 087 47.37 132 25.00 167 50.00
−0.5
043 100.00 088 54.84 133 15.62 168 38.46
044 0.00 089 65.22 134 66.67 169 0.00 −1
041 042 043 044 045 046 047 048 049 060 061 062 063 064 065 066 067 068 069 080 081 082 083 084 085
Topic Identifier
045 42.55 090 11.76 135 0.00 170 0.00 Ad−Hoc Robust Monolingual French track − Comparison to Median Mean Exact R−Precision by Topic (Topics 086 to 130)
1
046 0.00 091 0.00 136 100.00 171 33.33 UniNEfrr3
047 21.15 092 29.17 137 33.33 172 0.00 0.5
Difference
048 0.00 093 80.00 138 39.22 173 100.00 0
049 16.67 094 56.25 139 2.22 174 100.00 −0.5
060 2.22 095 18.64 140 59.38 175 100.00 −1
086 087 088 089 090 091 092 093 094 095 096 097 098 099 110 111 112 113 114 115 116 117 118 119 130
061 41.67 096 14.29 141 0.00 176 89.47 Topic Identifier
Ad−Hoc Robust Monolingual French track − Comparison to Median Mean Exact R−Precision by Topic (Topics 131 to 165)
062 37.50 097 42.86 142 92.31 177 42.86 1
UniNEfrr3
063 47.06 098 77.78 143 50.00 178 66.67 0.5
Difference
064 0.00 099 35.29 144 66.67 179 50.00
0
065 15.38 110 69.23 145 35.71 190 64.71
−0.5
066 84.31 111 0.00 146 0.00 191 0.00
−1
067 52.63 112 40.00 147 50.00 192 84.21 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 160 161 162 163 164 165
Topic Identifier
068 60.00 113 0.00 148 0.00 193 31.71 Ad−Hoc Robust Monolingual French track − Comparison to Median Mean Exact R−Precision by Topic (Topics 166 to 200)
1
069 50.00 114 36.67 149 60.00 194 0.00 UniNEfrr3
0.5
080 12.90 115 80.00 160 0.00 195 0.00
Difference
081 52.94 116 40.00 161 0.00 196 100.00 0
082 81.25 117 10.71 162 64.00 197 71.76 −0.5
083 0.00 118 66.67 163 27.27 198 66.67 −1
166 167 168 169 170 171 172 173 174 175 176 177 178 179 190 191 192 193 194 195 196 197 198 199 200
084 0.00 119 13.75 164 16.85 199 66.67 Topic Identifier
085 26.67 130 61.54 165 0.00 200 0.00
200
unine UniNEfrr2 AH-ROBUST-MONO-FR-CLEF2006
Overall statistics for 100 queries : Priority 3
Total number of documents over all queries Query Construction AUTOMATIC
Retrieved 100,000 Source Language French
Relevant 2,190 Topic Fields title
Relevant retrieved 1,921 Pooled false
Geometric Mean Average Precision 0.0938 data fusion of DFR, LM & Okapi
Binary Preference (BPREF) 0.4344
Interploated Recall (%) Precision Averages (%)
Ad−Hoc Robust Monolingual French track − Interpolated Recall vs Average Precision
100%
0 59.07
UniNEfrr2
10 56.12 90%
20 51.64
80%
30 47.57
40 42.73 70%
50 41.53
Average Precision
60%
60 39.04
70 34.41 50%
80 31.55 40%
90 27.36
30%
100 22.20
Average precision (non-interpolated) for all 20%
relevant documents (averaged over queries)
40.00 10%
0%
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
Interpolated Recall
Mean Average Precision Ad−Hoc Robust Monolingual French track − Box plot of the Topics of the Experiment
Maximum 1.0000
Minimum 0.0000
First Quartile 0.1190
Second Quartile 0.3821
Third Quartile 0.6577
Interquartile range 0.5387
Mean 0.4000
Standard Deviation 0.3289
Lower Outlier Threshold 0.0000
Upper Outlier Threshold 1.0000 0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Mean Average Precision
Mean With No Outliers 0.4000
Std With No Outliers 0.3289
Ad−Hoc Robust Monolingual French track − Distribution of the Topics of the Experiment
20
Number of Topics of the Experiment
15
10
5
0
0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Mean Average Precision
Precision averages (%) for individual Ad−Hoc Robust Monolingual French track − Comparison to Median Mean Average Precision by Topic (Topics 041 to 085)
queries 1
UniNEfrr2
0.5
041 100.00 086 19.85 131 19.32 166 0.00
Difference
0
042 52.05 087 45.87 132 48.07 167 45.00
−0.5
043 1.10 088 67.66 133 7.83 168 20.44
044 14.44 089 55.88 134 67.33 169 0.00 −1
041 042 043 044 045 046 047 048 049 060 061 062 063 064 065 066 067 068 069 080 081 082 083 084 085
Topic Identifier
045 63.29 090 7.94 135 1.12 170 5.00 Ad−Hoc Robust Monolingual French track − Comparison to Median Mean Average Precision by Topic (Topics 086 to 130)
1
046 11.79 091 0.08 136 100.00 171 43.13 UniNEfrr2
047 13.88 092 23.55 137 26.19 172 0.00 0.5
Difference
048 0.32 093 100.00 138 58.36 173 100.00 0
049 19.41 094 71.81 139 19.39 174 100.00 −0.5
060 0.00 095 12.01 140 25.54 175 100.00 −1
086 087 088 089 090 091 092 093 094 095 096 097 098 099 110 111 112 113 114 115 116 117 118 119 130
061 19.20 096 27.02 141 10.00 176 95.47 Topic Identifier
Ad−Hoc Robust Monolingual French track − Comparison to Median Mean Average Precision by Topic (Topics 131 to 165)
062 19.14 097 42.00 142 99.45 177 22.70 1
UniNEfrr2
063 51.46 098 94.60 143 19.59 178 40.89 0.5
Difference
064 0.00 099 23.79 144 80.95 179 30.04
0
065 23.26 110 65.43 145 44.34 190 48.55
−0.5
066 93.80 111 1.25 146 0.00 191 0.00
−1
067 58.20 112 44.85 147 40.77 192 92.84 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 160 161 162 163 164 165
Topic Identifier
068 51.60 113 2.86 148 2.99 193 48.44 Ad−Hoc Robust Monolingual French track − Comparison to Median Mean Average Precision by Topic (Topics 166 to 200)
1
069 34.37 114 13.40 149 90.29 194 0.00 UniNEfrr2
0.5
080 13.79 115 66.12 160 0.00 195 38.67
Difference
081 46.48 116 38.62 161 0.00 196 100.00 0
082 77.73 117 0.94 162 37.80 197 80.53 −0.5
083 100.00 118 9.81 163 24.28 198 69.52 −1
166 167 168 169 170 171 172 173 174 175 176 177 178 179 190 191 192 193 194 195 196 197 198 199 200
084 50.00 119 83.86 164 59.76 199 68.95 Topic Identifier
085 20.04 130 67.18 165 45.00 200 0.05
201
unine UniNEfrr2 AH-ROBUST-MONO-FR-CLEF2006
Docs Cutoff Levels Precision at DCL (%)
Ad−Hoc Robust Monolingual French track − Retrieved documents vs Precision
100%
5 docs 37.00
UniNEfrr2
10 docs 32.20 90%
15 docs 29.33
80%
20 docs 26.55
30 docs 23.47 70%
100 docs 12.69
60%
200 docs 7.84
R−Precision
500 docs 3.60 50%
1000 docs 1.92 40%
R-Precision (precision after R document retrieved,
where R = Relevant retrieved) 30%
38.34
20%
10%
0%
5 10 15 20 30 100 200 500 1000
Retrieved Documents (logarithmic scale)
Exact R-Precision Ad−Hoc Robust Monolingual French track − Box plot of the Topics of the Experiment
Maximum 1.0000
Minimum 0.0000
First Quartile 0.0972
Second Quartile 0.3333
Third Quartile 0.6155
Interquartile range 0.5183
Mean 0.3834
Standard Deviation 0.3232
Lower Outlier Threshold 0.0000
Upper Outlier Threshold 1.0000 0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Exact R−Precision
Mean With No Outliers 0.3834
Std With No Outliers 0.3232
Ad−Hoc Robust Monolingual French track − Distribution of the Topics of the Experiment
25
Number of Topics of the Experiment
20
15
10
5
0
0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Exact R−Precision
Precision averages (%) for individual Ad−Hoc Robust Monolingual French track − Comparison to Median Mean Exact R−Precision by Topic (Topics 041 to 085)
queries 1
UniNEfrr2
0.5
041 100.00 086 22.22 131 29.27 166 0.00
Difference
0
042 51.32 087 52.63 132 25.00 167 50.00
−0.5
043 0.00 088 58.06 133 12.50 168 30.77
044 0.00 089 52.17 134 66.67 169 0.00 −1
041 042 043 044 045 046 047 048 049 060 061 062 063 064 065 066 067 068 069 080 081 082 083 084 085
Topic Identifier
045 48.94 090 11.76 135 0.00 170 0.00 Ad−Hoc Robust Monolingual French track − Comparison to Median Mean Exact R−Precision by Topic (Topics 086 to 130)
1
046 14.29 091 0.00 136 100.00 171 33.33 UniNEfrr2
047 17.31 092 27.08 137 33.33 172 0.00 0.5
Difference
048 0.00 093 100.00 138 54.90 173 100.00 0
049 25.00 094 62.50 139 11.11 174 100.00 −0.5
060 0.00 095 22.03 140 27.34 175 100.00 −1
086 087 088 089 090 091 092 093 094 095 096 097 098 099 110 111 112 113 114 115 116 117 118 119 130
061 8.33 096 28.57 141 0.00 176 89.47 Topic Identifier
Ad−Hoc Robust Monolingual French track − Comparison to Median Mean Exact R−Precision by Topic (Topics 131 to 165)
062 25.00 097 42.86 142 92.31 177 28.57 1
UniNEfrr2
063 52.94 098 88.89 143 17.86 178 33.33 0.5
Difference
064 0.00 099 35.29 144 66.67 179 37.50
0
065 23.08 110 69.23 145 35.71 190 41.18
−0.5
066 86.27 111 0.00 146 0.00 191 0.00
−1
067 63.16 112 33.33 147 50.00 192 78.95 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 160 161 162 163 164 165
Topic Identifier
068 48.00 113 0.00 148 0.00 193 43.90 Ad−Hoc Robust Monolingual French track − Comparison to Median Mean Exact R−Precision by Topic (Topics 166 to 200)
1
069 37.50 114 16.67 149 80.00 194 0.00 UniNEfrr2
0.5
080 12.90 115 80.00 160 0.00 195 36.84
Difference
081 47.06 116 30.00 161 0.00 196 100.00 0
082 87.50 117 0.00 162 44.00 197 76.34 −0.5
083 100.00 118 0.00 163 27.27 198 66.67 −1
166 167 168 169 170 171 172 173 174 175 176 177 178 179 190 191 192 193 194 195 196 197 198 199 200
084 0.00 119 77.50 164 57.30 199 60.61 Topic Identifier
085 16.67 130 69.23 165 50.00 200 0.00
202
colesir CoLesIRitTst AH-ROBUST-MONO-IT-CLEF2006
Overall statistics for 100 queries : Priority 1
Total number of documents over all queries Query Construction AUTOMATIC
Retrieved 100,000 Source Language Italian
Relevant 1,930 Topic Fields title, description
Relevant retrieved 1,758 Pooled false
Geometric Mean Average Precision 0.0823 robust monolingual IT - TEST; automatic; TD
Binary Preference (BPREF) 0.3638
Interploated Recall (%) Precision Averages (%)
Ad−Hoc Robust Monolingual Italian track − Interpolated Recall vs Average Precision
100%
0 61.40
CoLesIRitTst
10 52.75 90%
20 45.71
80%
30 40.23
40 35.97 70%
50 33.01
Average Precision
60%
60 28.36
70 24.94 50%
80 21.38 40%
90 17.71
30%
100 13.03
Average precision (non-interpolated) for all 20%
relevant documents (averaged over queries)
32.23 10%
0%
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
Interpolated Recall
Mean Average Precision Ad−Hoc Robust Monolingual Italian track − Box plot of the Topics of the Experiment
Maximum 1.0000
Minimum 0.0000
First Quartile 0.0774
Second Quartile 0.2805
Third Quartile 0.4818
Interquartile range 0.4045
Mean 0.3223
Standard Deviation 0.2706
Lower Outlier Threshold 0.0000
Upper Outlier Threshold 1.0000 0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Mean Average Precision
Mean With No Outliers 0.3223
Std With No Outliers 0.2706
Ad−Hoc Robust Monolingual Italian track − Distribution of the Topics of the Experiment
20
Number of Topics of the Experiment
15
10
5
0
0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Mean Average Precision
Precision averages (%) for individual Ad−Hoc Robust Monolingual Italian track − Comparison to Median Mean Average Precision by Topic (Topics 041 to 085)
queries 1
CoLesIRitTst
0.5
041 37.99 086 16.45 131 6.15 166 0.15
Difference
0
042 33.48 087 43.73 132 45.42 167 58.33
−0.5
043 0.00 088 41.73 133 19.78 168 7.59
044 14.58 089 76.19 134 96.67 169 0.00 −1
041 042 043 044 045 046 047 048 049 060 061 062 063 064 065 066 067 068 069 080 081 082 083 084 085
Topic Identifier
045 12.25 090 11.73 135 45.43 170 0.00 Ad−Hoc Robust Monolingual Italian track − Comparison to Median Mean Average Precision by Topic (Topics 086 to 130)
1
046 11.89 091 18.63 136 57.92 171 23.44 CoLesIRitTst
047 2.54 092 21.12 137 30.21 172 0.00 0.5
Difference
048 7.89 093 65.71 138 35.40 173 64.29 0
049 16.15 094 51.50 139 7.20 174 69.12 −0.5
060 16.39 095 3.06 140 28.21 175 0.00 −1
086 087 088 089 090 091 092 093 094 095 096 097 098 099 110 111 112 113 114 115 116 117 118 119 130
061 28.67 096 11.89 141 96.83 176 50.63 Topic Identifier
Ad−Hoc Robust Monolingual Italian track − Comparison to Median Mean Average Precision by Topic (Topics 131 to 165)
062 38.49 097 68.38 142 90.32 177 26.65 1
CoLesIRitTst
063 45.37 098 75.36 143 43.27 178 28.63 0.5
Difference
064 0.00 099 24.19 144 0.00 179 21.99
0
065 30.35 110 83.10 145 100.00 190 34.58
−0.5
066 68.03 111 6.40 146 0.00 191 0.00
−1
067 27.19 112 52.52 147 6.87 192 55.76 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 160 161 162 163 164 165
Topic Identifier
068 29.99 113 7.55 148 6.12 193 17.08 Ad−Hoc Robust Monolingual Italian track − Comparison to Median Mean Average Precision by Topic (Topics 166 to 200)
1
069 6.52 114 28.26 149 26.87 194 39.41 CoLesIRitTst
0.5
080 17.60 115 27.89 160 0.00 195 45.32
Difference
081 34.44 116 35.99 161 1.32 196 100.00 0
082 56.26 117 43.32 162 51.94 197 45.73 −0.5
083 18.33 118 21.71 163 21.76 198 6.66 −1
166 167 168 169 170 171 172 173 174 175 176 177 178 179 190 191 192 193 194 195 196 197 198 199 200
084 0.00 119 66.28 164 4.66 199 53.12 Topic Identifier
085 50.87 130 30.73 165 100.00 200 13.89
203
colesir CoLesIRitTst AH-ROBUST-MONO-IT-CLEF2006
Docs Cutoff Levels Precision at DCL (%)
Ad−Hoc Robust Monolingual Italian track − Retrieved documents vs Precision
100%
5 docs 36.40
CoLesIRitTst
10 docs 31.60 90%
15 docs 27.93
80%
20 docs 25.50
30 docs 21.83 70%
100 docs 11.14
60%
200 docs 6.91
R−Precision
500 docs 3.25 50%
1000 docs 1.76 40%
R-Precision (precision after R document retrieved,
where R = Relevant retrieved) 30%
32.06
20%
10%
0%
5 10 15 20 30 100 200 500 1000
Retrieved Documents (logarithmic scale)
Exact R-Precision Ad−Hoc Robust Monolingual Italian track − Box plot of the Topics of the Experiment
Maximum 1.0000
Minimum 0.0000
First Quartile 0.0869
Second Quartile 0.3101
Third Quartile 0.4662
Interquartile range 0.3793
Mean 0.3206
Standard Deviation 0.2601
Lower Outlier Threshold 0.0000
Upper Outlier Threshold 1.0000 0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Exact R−Precision
Mean With No Outliers 0.3206
Std With No Outliers 0.2601
Ad−Hoc Robust Monolingual Italian track − Distribution of the Topics of the Experiment
25
Number of Topics of the Experiment
20
15
10
5
0
0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Exact R−Precision
Precision averages (%) for individual Ad−Hoc Robust Monolingual Italian track − Comparison to Median Mean Exact R−Precision by Topic (Topics 041 to 085)
queries 1
CoLesIRitTst
0.5
041 42.86 086 22.58 131 12.20 166 0.00
Difference
0
042 39.76 087 30.00 132 33.33 167 50.00
−0.5
043 0.00 088 46.67 133 31.25 168 9.38
044 0.00 089 73.08 134 80.00 169 0.00 −1
041 042 043 044 045 046 047 048 049 060 061 062 063 064 065 066 067 068 069 080 081 082 083 084 085
Topic Identifier
045 11.90 090 33.33 135 49.35 170 0.00 Ad−Hoc Robust Monolingual Italian track − Comparison to Median Mean Exact R−Precision by Topic (Topics 086 to 130)
1
046 12.50 091 25.00 136 66.67 171 28.57 CoLesIRitTst
047 0.00 092 19.44 137 25.00 172 0.00 0.5
Difference
048 8.00 093 77.78 138 30.77 173 50.00 0
049 24.14 094 40.00 139 6.25 174 62.50 −0.5
060 25.35 095 7.14 140 34.00 175 0.00 −1
086 087 088 089 090 091 092 093 094 095 096 097 098 099 110 111 112 113 114 115 116 117 118 119 130
061 37.50 096 0.00 141 85.71 176 43.75 Topic Identifier
Ad−Hoc Robust Monolingual Italian track − Comparison to Median Mean Exact R−Precision by Topic (Topics 131 to 165)
062 46.15 097 50.00 142 85.71 177 33.33 1
CoLesIRitTst
063 50.00 098 75.00 143 46.15 178 25.00 0.5
Difference
064 0.00 099 30.77 144 0.00 179 27.50
0
065 33.33 110 78.26 145 100.00 190 33.33
−0.5
066 59.46 111 0.00 146 0.00 191 0.00
−1
067 29.41 112 50.00 147 20.00 192 48.39 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 160 161 162 163 164 165
Topic Identifier
068 23.33 113 5.56 148 0.00 193 0.00 Ad−Hoc Robust Monolingual Italian track − Comparison to Median Mean Exact R−Precision by Topic (Topics 166 to 200)
1
069 0.00 114 46.15 149 33.33 194 38.46 CoLesIRitTst
0.5
080 22.62 115 40.74 160 0.00 195 41.67
Difference
081 40.74 116 40.00 161 0.00 196 100.00 0
082 48.48 117 52.94 162 50.00 197 44.55 −0.5
083 0.00 118 25.00 163 17.39 198 14.29 −1
166 167 168 169 170 171 172 173 174 175 176 177 178 179 190 191 192 193 194 195 196 197 198 199 200
084 0.00 119 65.00 164 9.68 199 44.44 Topic Identifier
085 46.58 130 40.00 165 100.00 200 17.14
204
daedalus itFSitR4S AH-ROBUST-MONO-IT-CLEF2006
Overall statistics for 100 queries : Priority 3
Total number of documents over all queries Query Construction AUTOMATIC
Retrieved 100,000 Source Language Italian
Relevant 1,930 Topic Fields title, description, narrative
Relevant retrieved 1,746 Pooled false
Geometric Mean Average Precision 0.1135 TDN run
Binary Preference (BPREF) 0.4116
Interploated Recall (%) Precision Averages (%)
Ad−Hoc Robust Monolingual Italian track − Interpolated Recall vs Average Precision
100%
0 67.98
itFSitR4S
10 60.56 90%
20 53.77
80%
30 47.16
40 42.49 70%
50 39.42
Average Precision
60%
60 34.00
70 29.74 50%
80 23.77 40%
90 20.07
30%
100 14.90
Average precision (non-interpolated) for all 20%
relevant documents (averaged over queries)
37.61 10%
0%
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
Interpolated Recall
Mean Average Precision Ad−Hoc Robust Monolingual Italian track − Box plot of the Topics of the Experiment
Maximum 1.0000
Minimum 0.0000
First Quartile 0.1410
Second Quartile 0.3480
Third Quartile 0.5392
Interquartile range 0.3982
Mean 0.3761
Standard Deviation 0.2734
Lower Outlier Threshold 0.0000
Upper Outlier Threshold 1.0000 0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Mean Average Precision
Mean With No Outliers 0.3761
Std With No Outliers 0.2734
Ad−Hoc Robust Monolingual Italian track − Distribution of the Topics of the Experiment
15
Number of Topics of the Experiment
10
5
0
0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Mean Average Precision
Precision averages (%) for individual Ad−Hoc Robust Monolingual Italian track − Comparison to Median Mean Average Precision by Topic (Topics 041 to 085)
queries 1
itFSitR4S
0.5
041 63.16 086 19.46 131 9.58 166 0.45
Difference
0
042 31.70 087 41.14 132 52.78 167 100.00
−0.5
043 0.00 088 37.31 133 38.06 168 24.10
044 14.65 089 74.34 134 77.31 169 0.00 −1
041 042 043 044 045 046 047 048 049 060 061 062 063 064 065 066 067 068 069 080 081 082 083 084 085
Topic Identifier
045 20.05 090 9.20 135 6.64 170 0.00 Ad−Hoc Robust Monolingual Italian track − Comparison to Median Mean Average Precision by Topic (Topics 086 to 130)
1
046 11.72 091 5.48 136 62.59 171 36.04 itFSitR4S
047 9.83 092 22.83 137 40.38 172 0.00 0.5
Difference
048 12.97 093 64.36 138 69.77 173 83.33 0
049 25.44 094 33.56 139 11.78 174 74.36 −0.5
060 20.26 095 2.37 140 14.65 175 0.00 −1
086 087 088 089 090 091 092 093 094 095 096 097 098 099 110 111 112 113 114 115 116 117 118 119 130
061 28.06 096 24.92 141 61.59 176 51.97 Topic Identifier
Ad−Hoc Robust Monolingual Italian track − Comparison to Median Mean Average Precision by Topic (Topics 131 to 165)
062 53.04 097 63.26 142 75.82 177 47.52 1
itFSitR4S
063 56.76 098 13.96 143 30.17 178 30.77 0.5
Difference
064 0.00 099 40.85 144 0.00 179 53.87
0
065 31.47 110 83.52 145 100.00 190 48.26
−0.5
066 80.47 111 14.25 146 0.00 191 0.00
−1
067 42.65 112 39.15 147 26.86 192 47.93 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 160 161 162 163 164 165
Topic Identifier
068 42.29 113 12.86 148 9.38 193 20.44 Ad−Hoc Robust Monolingual Italian track − Comparison to Median Mean Average Precision by Topic (Topics 166 to 200)
1
069 7.13 114 49.71 149 32.78 194 27.32 itFSitR4S
0.5
080 42.02 115 41.62 160 0.00 195 61.54
Difference
081 33.01 116 39.83 161 100.00 196 100.00 0
082 80.15 117 19.44 162 64.82 197 66.46 −0.5
083 53.97 118 26.21 163 46.15 198 28.86 −1
166 167 168 169 170 171 172 173 174 175 176 177 178 179 190 191 192 193 194 195 196 197 198 199 200
084 11.88 119 52.39 164 28.94 199 42.55 Topic Identifier
085 52.42 130 37.91 165 100.00 200 60.39
205
daedalus itFSitR4S AH-ROBUST-MONO-IT-CLEF2006
Docs Cutoff Levels Precision at DCL (%)
Ad−Hoc Robust Monolingual Italian track − Retrieved documents vs Precision
100%
5 docs 41.40
itFSitR4S
10 docs 35.20 90%
15 docs 32.20
80%
20 docs 29.10
30 docs 24.30 70%
100 docs 12.09
60%
200 docs 7.36
R−Precision
500 docs 3.36 50%
1000 docs 1.75 40%
R-Precision (precision after R document retrieved,
where R = Relevant retrieved) 30%
36.03
20%
10%
0%
5 10 15 20 30 100 200 500 1000
Retrieved Documents (logarithmic scale)
Exact R-Precision Ad−Hoc Robust Monolingual Italian track − Box plot of the Topics of the Experiment
Maximum 1.0000
Minimum 0.0000
First Quartile 0.1667
Second Quartile 0.3600
Third Quartile 0.5000
Interquartile range 0.3333
Mean 0.3603
Standard Deviation 0.2664
Lower Outlier Threshold 0.0000
Upper Outlier Threshold 0.7857 0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Exact R−Precision
Mean With No Outliers 0.3266
Std With No Outliers 0.2277
Ad−Hoc Robust Monolingual Italian track − Distribution of the Topics of the Experiment
20
Number of Topics of the Experiment
15
10
5
0
0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Exact R−Precision
Precision averages (%) for individual Ad−Hoc Robust Monolingual Italian track − Comparison to Median Mean Exact R−Precision by Topic (Topics 041 to 085)
queries 1
itFSitR4S
0.5
041 57.14 086 19.35 131 14.63 166 0.00
Difference
0
042 34.94 087 50.00 132 33.33 167 100.00
−0.5
043 0.00 088 33.33 133 37.50 168 21.88
044 0.00 089 76.92 134 60.00 169 0.00 −1
041 042 043 044 045 046 047 048 049 060 061 062 063 064 065 066 067 068 069 080 081 082 083 084 085
Topic Identifier
045 23.81 090 0.00 135 15.58 170 0.00 Ad−Hoc Robust Monolingual Italian track − Comparison to Median Mean Exact R−Precision by Topic (Topics 086 to 130)
1
046 18.75 091 0.00 136 50.00 171 42.86 itFSitR4S
047 11.54 092 22.22 137 50.00 172 0.00 0.5
Difference
048 16.00 093 77.78 138 61.54 173 50.00 0
049 34.48 094 45.00 139 0.00 174 62.50 −0.5
060 29.58 095 5.71 140 22.00 175 0.00 −1
086 087 088 089 090 091 092 093 094 095 096 097 098 099 110 111 112 113 114 115 116 117 118 119 130
061 37.50 096 25.00 141 57.14 176 56.25 Topic Identifier
Ad−Hoc Robust Monolingual Italian track − Comparison to Median Mean Exact R−Precision by Topic (Topics 131 to 165)
062 38.46 097 50.00 142 78.57 177 50.00 1
itFSitR4S
063 50.00 098 16.67 143 38.46 178 50.00 0.5
Difference
064 0.00 099 35.90 144 0.00 179 55.00
0
065 46.67 110 73.91 145 100.00 190 50.00
−0.5
066 75.68 111 27.27 146 0.00 191 0.00
−1
067 44.12 112 33.33 147 20.00 192 45.16 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 160 161 162 163 164 165
Topic Identifier
068 36.67 113 16.67 148 0.00 193 0.00 Ad−Hoc Robust Monolingual Italian track − Comparison to Median Mean Exact R−Precision by Topic (Topics 166 to 200)
1
069 16.67 114 38.46 149 0.00 194 23.08 itFSitR4S
0.5
080 44.05 115 48.15 160 0.00 195 58.33
Difference
081 37.04 116 30.00 161 100.00 196 100.00 0
082 66.67 117 17.65 162 71.43 197 60.00 −0.5
083 33.33 118 25.00 163 52.17 198 28.57 −1
166 167 168 169 170 171 172 173 174 175 176 177 178 179 190 191 192 193 194 195 196 197 198 199 200
084 0.00 119 46.67 164 32.26 199 36.11 Topic Identifier
085 50.68 130 40.00 165 100.00 200 60.00
206
daedalus ityitRFS3456 AH-ROBUST-MONO-IT-CLEF2006
Overall statistics for 100 queries : Priority 2
Total number of documents over all queries Query Construction AUTOMATIC
Retrieved 100,000 Source Language Italian
Relevant 1,930 Topic Fields title, description, narrative
Relevant retrieved 1,751 Pooled false
Geometric Mean Average Precision 0.1146 Average run
Binary Preference (BPREF) 0.4141
Interploated Recall (%) Precision Averages (%)
Ad−Hoc Robust Monolingual Italian track − Interpolated Recall vs Average Precision
100%
0 68.13
ityitRFS3456
10 60.55 90%
20 54.25
80%
30 47.45
40 42.69 70%
50 39.35
Average Precision
60%
60 33.98
70 29.93 50%
80 23.83 40%
90 20.11
30%
100 14.92
Average precision (non-interpolated) for all 20%
relevant documents (averaged over queries)
37.80 10%
0%
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
Interpolated Recall
Mean Average Precision Ad−Hoc Robust Monolingual Italian track − Box plot of the Topics of the Experiment
Maximum 1.0000
Minimum 0.0000
First Quartile 0.1392
Second Quartile 0.3526
Third Quartile 0.5473
Interquartile range 0.4081
Mean 0.3780
Standard Deviation 0.2726
Lower Outlier Threshold 0.0000
Upper Outlier Threshold 1.0000 0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Mean Average Precision
Mean With No Outliers 0.3780
Std With No Outliers 0.2726
Ad−Hoc Robust Monolingual Italian track − Distribution of the Topics of the Experiment
15
Number of Topics of the Experiment
10
5
0
0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Mean Average Precision
Precision averages (%) for individual Ad−Hoc Robust Monolingual Italian track − Comparison to Median Mean Average Precision by Topic (Topics 041 to 085)
queries 1
ityitRFS3456
0.5
041 63.06 086 18.95 131 10.44 166 0.47
Difference
0
042 31.48 087 40.96 132 50.76 167 100.00
−0.5
043 0.00 088 37.73 133 38.25 168 25.33
044 13.89 089 74.44 134 77.31 169 0.00 −1
041 042 043 044 045 046 047 048 049 060 061 062 063 064 065 066 067 068 069 080 081 082 083 084 085
Topic Identifier
045 20.47 090 11.97 135 6.68 170 0.00 Ad−Hoc Robust Monolingual Italian track − Comparison to Median Mean Average Precision by Topic (Topics 086 to 130)
1
046 11.08 091 5.35 136 65.21 171 35.95 ityitRFS3456
047 9.55 092 23.28 137 40.38 172 0.00 0.5
Difference
048 12.50 093 64.36 138 69.65 173 83.33 0
049 25.45 094 34.56 139 11.76 174 74.52 −0.5
060 20.44 095 2.70 140 14.53 175 0.00 −1
086 087 088 089 090 091 092 093 094 095 096 097 098 099 110 111 112 113 114 115 116 117 118 119 130
061 27.63 096 24.45 141 60.90 176 52.55 Topic Identifier
Ad−Hoc Robust Monolingual Italian track − Comparison to Median Mean Average Precision by Topic (Topics 131 to 165)
062 52.80 097 63.26 142 75.82 177 47.81 1
ityitRFS3456
063 56.96 098 13.94 143 31.91 178 31.25 0.5
Difference
064 0.00 099 42.60 144 0.00 179 54.52
0
065 30.80 110 82.55 145 100.00 190 48.05
−0.5
066 80.58 111 14.53 146 0.00 191 0.00
−1
067 42.56 112 39.57 147 30.22 192 48.09 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 160 161 162 163 164 165
Topic Identifier
068 40.49 113 13.52 148 9.38 193 20.44 Ad−Hoc Robust Monolingual Italian track − Comparison to Median Mean Average Precision by Topic (Topics 166 to 200)
1
069 7.12 114 49.68 149 32.78 194 29.27 ityitRFS3456
0.5
080 42.02 115 42.14 160 0.00 195 63.19
Difference
081 34.03 116 39.05 161 100.00 196 100.00 0
082 76.51 117 23.17 162 66.02 197 66.81 −0.5
083 53.97 118 26.10 163 47.80 198 26.81 −1
166 167 168 169 170 171 172 173 174 175 176 177 178 179 190 191 192 193 194 195 196 197 198 199 200
084 11.95 119 54.94 164 29.26 199 44.06 Topic Identifier
085 52.24 130 37.91 165 100.00 200 58.98
207
daedalus ityitRFS3456 AH-ROBUST-MONO-IT-CLEF2006
Docs Cutoff Levels Precision at DCL (%)
Ad−Hoc Robust Monolingual Italian track − Retrieved documents vs Precision
100%
5 docs 41.20
ityitRFS3456
10 docs 35.00 90%
15 docs 32.67
80%
20 docs 29.00
30 docs 24.40 70%
100 docs 12.15
60%
200 docs 7.41
R−Precision
500 docs 3.37 50%
1000 docs 1.75 40%
R-Precision (precision after R document retrieved,
where R = Relevant retrieved) 30%
36.89
20%
10%
0%
5 10 15 20 30 100 200 500 1000
Retrieved Documents (logarithmic scale)
Exact R-Precision Ad−Hoc Robust Monolingual Italian track − Box plot of the Topics of the Experiment
Maximum 1.0000
Minimum 0.0000
First Quartile 0.1905
Second Quartile 0.3685
Third Quartile 0.5093
Interquartile range 0.3187
Mean 0.3689
Standard Deviation 0.2615
Lower Outlier Threshold 0.0000
Upper Outlier Threshold 0.7857 0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Exact R−Precision
Mean With No Outliers 0.3356
Std With No Outliers 0.2229
Ad−Hoc Robust Monolingual Italian track − Distribution of the Topics of the Experiment
20
Number of Topics of the Experiment
15
10
5
0
0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Exact R−Precision
Precision averages (%) for individual Ad−Hoc Robust Monolingual Italian track − Comparison to Median Mean Exact R−Precision by Topic (Topics 041 to 085)
queries 1
ityitRFS3456
0.5
041 57.14 086 19.35 131 19.51 166 0.00
Difference
0
042 34.94 087 50.00 132 33.33 167 100.00
−0.5
043 0.00 088 40.00 133 37.50 168 25.00
044 0.00 089 76.92 134 60.00 169 0.00 −1
041 042 043 044 045 046 047 048 049 060 061 062 063 064 065 066 067 068 069 080 081 082 083 084 085
Topic Identifier
045 21.43 090 33.33 135 15.58 170 0.00 Ad−Hoc Robust Monolingual Italian track − Comparison to Median Mean Exact R−Precision by Topic (Topics 086 to 130)
1
046 18.75 091 0.00 136 58.33 171 42.86 ityitRFS3456
047 11.54 092 25.00 137 50.00 172 0.00 0.5
Difference
048 16.00 093 77.78 138 61.54 173 50.00 0
049 34.48 094 45.00 139 0.00 174 62.50 −0.5
060 28.17 095 7.14 140 22.00 175 0.00 −1
086 087 088 089 090 091 092 093 094 095 096 097 098 099 110 111 112 113 114 115 116 117 118 119 130
061 31.25 096 25.00 141 57.14 176 56.25 Topic Identifier
Ad−Hoc Robust Monolingual Italian track − Comparison to Median Mean Exact R−Precision by Topic (Topics 131 to 165)
062 38.46 097 50.00 142 78.57 177 50.00 1
ityitRFS3456
063 37.50 098 16.67 143 42.31 178 50.00 0.5
Difference
064 0.00 099 41.03 144 0.00 179 55.00
0
065 40.00 110 73.91 145 100.00 190 50.00
−0.5
066 75.68 111 27.27 146 0.00 191 0.00
−1
067 41.18 112 33.33 147 40.00 192 48.39 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 160 161 162 163 164 165
Topic Identifier
068 36.67 113 22.22 148 0.00 193 0.00 Ad−Hoc Robust Monolingual Italian track − Comparison to Median Mean Exact R−Precision by Topic (Topics 166 to 200)
1
069 16.67 114 38.46 149 0.00 194 30.77 ityitRFS3456
0.5
080 44.05 115 51.85 160 0.00 195 58.33
Difference
081 37.04 116 30.00 161 100.00 196 100.00 0
082 63.64 117 29.41 162 71.43 197 60.00 −0.5
083 33.33 118 25.00 163 52.17 198 28.57 −1
166 167 168 169 170 171 172 173 174 175 176 177 178 179 190 191 192 193 194 195 196 197 198 199 200
084 0.00 119 50.00 164 32.26 199 36.11 Topic Identifier
085 49.32 130 40.00 165 100.00 200 57.14
208
daedalus itFSitR2S AH-ROBUST-MONO-IT-CLEF2006
Overall statistics for 100 queries : Priority 4
Total number of documents over all queries Query Construction AUTOMATIC
Retrieved 100,000 Source Language Italian
Relevant 1,930 Topic Fields title, description
Relevant retrieved 1,750 Pooled false
Geometric Mean Average Precision 0.1050 Mandatory TD run
Binary Preference (BPREF) 0.3906
Interploated Recall (%) Precision Averages (%)
Ad−Hoc Robust Monolingual Italian track − Interpolated Recall vs Average Precision
100%
0 65.66
itFSitR2S
10 58.07 90%
20 52.31
80%
30 44.19
40 39.14 70%
50 36.07
Average Precision
60%
60 30.95
70 26.99 50%
80 21.40 40%
90 17.82
30%
100 12.68
Average precision (non-interpolated) for all 20%
relevant documents (averaged over queries)
35.11 10%
0%
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
Interpolated Recall
Mean Average Precision Ad−Hoc Robust Monolingual Italian track − Box plot of the Topics of the Experiment
Maximum 1.0000
Minimum 0.0000
First Quartile 0.1269
Second Quartile 0.3322
Third Quartile 0.5098
Interquartile range 0.3828
Mean 0.3511
Standard Deviation 0.2657
Lower Outlier Threshold 0.0000
Upper Outlier Threshold 1.0000 0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Mean Average Precision
Mean With No Outliers 0.3511
Std With No Outliers 0.2657
Ad−Hoc Robust Monolingual Italian track − Distribution of the Topics of the Experiment
15
Number of Topics of the Experiment
10
5
0
0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Mean Average Precision
Precision averages (%) for individual Ad−Hoc Robust Monolingual Italian track − Comparison to Median Mean Average Precision by Topic (Topics 041 to 085)
queries 1
itFSitR2S
0.5
041 65.05 086 12.17 131 15.41 166 0.63
Difference
0
042 27.92 087 39.20 132 25.78 167 100.00
−0.5
043 0.00 088 45.96 133 37.56 168 26.92
044 10.51 089 71.96 134 81.03 169 0.00 −1
041 042 043 044 045 046 047 048 049 060 061 062 063 064 065 066 067 068 069 080 081 082 083 084 085
Topic Identifier
045 21.09 090 11.53 135 6.25 170 0.00 Ad−Hoc Robust Monolingual Italian track − Comparison to Median Mean Average Precision by Topic (Topics 086 to 130)
1
046 8.64 091 4.48 136 59.72 171 46.74 itFSitR2S
047 4.43 092 24.19 137 38.71 172 0.00 0.5
Difference
048 7.99 093 65.68 138 42.95 173 62.50 0
049 24.74 094 41.75 139 10.73 174 74.89 −0.5
060 20.81 095 4.14 140 12.45 175 0.00 −1
086 087 088 089 090 091 092 093 094 095 096 097 098 099 110 111 112 113 114 115 116 117 118 119 130
061 24.68 096 19.33 141 32.99 176 50.85 Topic Identifier
Ad−Hoc Robust Monolingual Italian track − Comparison to Median Mean Average Precision by Topic (Topics 131 to 165)
062 46.89 097 43.33 142 75.82 177 45.37 1
itFSitR2S
063 54.27 098 14.29 143 33.46 178 29.52 0.5
Difference
064 0.00 099 44.70 144 0.00 179 57.08
0
065 16.38 110 84.88 145 100.00 190 36.73
−0.5
066 79.28 111 12.94 146 0.00 191 0.00
−1
067 36.11 112 41.54 147 21.00 192 36.07 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 160 161 162 163 164 165
Topic Identifier
068 36.57 113 7.17 148 11.78 193 20.59 Ad−Hoc Robust Monolingual Italian track − Comparison to Median Mean Average Precision by Topic (Topics 166 to 200)
1
069 3.17 114 29.98 149 27.78 194 51.10 itFSitR2S
0.5
080 40.95 115 37.26 160 0.00 195 57.30
Difference
081 34.04 116 31.45 161 100.00 196 100.00 0
082 59.71 117 35.76 162 56.38 197 63.73 −0.5
083 49.36 118 12.99 163 51.53 198 31.86 −1
166 167 168 169 170 171 172 173 174 175 176 177 178 179 190 191 192 193 194 195 196 197 198 199 200
084 13.22 119 59.85 164 21.47 199 46.14 Topic Identifier
085 51.47 130 30.96 165 100.00 200 45.47
209
daedalus itFSitR2S AH-ROBUST-MONO-IT-CLEF2006
Docs Cutoff Levels Precision at DCL (%)
Ad−Hoc Robust Monolingual Italian track − Retrieved documents vs Precision
100%
5 docs 39.00
itFSitR2S
10 docs 32.70 90%
15 docs 29.93
80%
20 docs 26.75
30 docs 23.23 70%
100 docs 11.90
60%
200 docs 7.28
R−Precision
500 docs 3.33 50%
1000 docs 1.75 40%
R-Precision (precision after R document retrieved,
where R = Relevant retrieved) 30%
35.72
20%
10%
0%
5 10 15 20 30 100 200 500 1000
Retrieved Documents (logarithmic scale)
Exact R-Precision Ad−Hoc Robust Monolingual Italian track − Box plot of the Topics of the Experiment
Maximum 1.0000
Minimum 0.0000
First Quartile 0.1920
Second Quartile 0.3417
Third Quartile 0.5000
Interquartile range 0.3080
Mean 0.3572
Standard Deviation 0.2586
Lower Outlier Threshold 0.0000
Upper Outlier Threshold 0.7857 0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Exact R−Precision
Mean With No Outliers 0.3234
Std With No Outliers 0.2175
Ad−Hoc Robust Monolingual Italian track − Distribution of the Topics of the Experiment
20
Number of Topics of the Experiment
15
10
5
0
0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Exact R−Precision
Precision averages (%) for individual Ad−Hoc Robust Monolingual Italian track − Comparison to Median Mean Exact R−Precision by Topic (Topics 041 to 085)
queries 1
itFSitR2S
0.5
041 57.14 086 19.35 131 21.95 166 0.00
Difference
0
042 32.53 087 50.00 132 33.33 167 100.00
−0.5
043 0.00 088 46.67 133 37.50 168 31.25
044 0.00 089 73.08 134 60.00 169 0.00 −1
041 042 043 044 045 046 047 048 049 060 061 062 063 064 065 066 067 068 069 080 081 082 083 084 085
Topic Identifier
045 19.05 090 33.33 135 15.58 170 0.00 Ad−Hoc Robust Monolingual Italian track − Comparison to Median Mean Exact R−Precision by Topic (Topics 086 to 130)
1
046 6.25 091 0.00 136 58.33 171 42.86 itFSitR2S
047 3.85 092 22.22 137 50.00 172 0.00 0.5
Difference
048 12.00 093 77.78 138 30.77 173 50.00 0
049 31.03 094 40.00 139 0.00 174 62.50 −0.5
060 29.58 095 11.43 140 22.00 175 0.00 −1
086 087 088 089 090 091 092 093 094 095 096 097 098 099 110 111 112 113 114 115 116 117 118 119 130
061 31.25 096 25.00 141 28.57 176 62.50 Topic Identifier
Ad−Hoc Robust Monolingual Italian track − Comparison to Median Mean Exact R−Precision by Topic (Topics 131 to 165)
062 38.46 097 50.00 142 78.57 177 50.00 1
itFSitR2S
063 50.00 098 25.00 143 30.77 178 25.00 0.5
Difference
064 0.00 099 43.59 144 0.00 179 55.00
0
065 26.67 110 78.26 145 100.00 190 41.67
−0.5
066 72.97 111 27.27 146 0.00 191 0.00
−1
067 35.29 112 33.33 147 40.00 192 35.48 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 160 161 162 163 164 165
Topic Identifier
068 40.00 113 16.67 148 0.00 193 0.00 Ad−Hoc Robust Monolingual Italian track − Comparison to Median Mean Exact R−Precision by Topic (Topics 166 to 200)
1
069 0.00 114 38.46 149 0.00 194 46.15 itFSitR2S
0.5
080 45.24 115 44.44 160 0.00 195 58.33
Difference
081 37.04 116 40.00 161 100.00 196 100.00 0
082 54.55 117 41.18 162 57.14 197 62.73 −0.5
083 33.33 118 25.00 163 56.52 198 28.57 −1
166 167 168 169 170 171 172 173 174 175 176 177 178 179 190 191 192 193 194 195 196 197 198 199 200
084 25.00 119 55.00 164 32.26 199 38.89 Topic Identifier
085 52.05 130 35.00 165 100.00 200 42.86
210
daedalus itFSitR6S AH-ROBUST-MONO-IT-CLEF2006
Overall statistics for 100 queries : Priority 1
Total number of documents over all queries Query Construction AUTOMATIC
Retrieved 100,000 Source Language Italian
Relevant 1,930 Topic Fields title, description, narrative
Relevant retrieved 1,756 Pooled false
Geometric Mean Average Precision 0.1153 TDN run
Binary Preference (BPREF) 0.4136
Interploated Recall (%) Precision Averages (%)
Ad−Hoc Robust Monolingual Italian track − Interpolated Recall vs Average Precision
100%
0 68.95
itFSitR6S
10 61.09 90%
20 55.18
80%
30 47.88
40 42.30 70%
50 38.91
Average Precision
60%
60 33.44
70 29.62 50%
80 23.55 40%
90 20.05
30%
100 14.80
Average precision (non-interpolated) for all 20%
relevant documents (averaged over queries)
37.82 10%
0%
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
Interpolated Recall
Mean Average Precision Ad−Hoc Robust Monolingual Italian track − Box plot of the Topics of the Experiment
Maximum 1.0000
Minimum 0.0000
First Quartile 0.1362
Second Quartile 0.3785
Third Quartile 0.5480
Interquartile range 0.4118
Mean 0.3782
Standard Deviation 0.2689
Lower Outlier Threshold 0.0000
Upper Outlier Threshold 1.0000 0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Mean Average Precision
Mean With No Outliers 0.3782
Std With No Outliers 0.2689
Ad−Hoc Robust Monolingual Italian track − Distribution of the Topics of the Experiment
15
Number of Topics of the Experiment
10
5
0
0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Mean Average Precision
Precision averages (%) for individual Ad−Hoc Robust Monolingual Italian track − Comparison to Median Mean Average Precision by Topic (Topics 041 to 085)
queries 1
itFSitR6S
0.5
041 63.06 086 18.83 131 11.38 166 0.53
Difference
0
042 30.55 087 40.12 132 49.83 167 100.00
−0.5
043 0.00 088 38.89 133 38.16 168 27.29
044 13.25 089 73.37 134 78.57 169 0.00 −1
041 042 043 044 045 046 047 048 049 060 061 062 063 064 065 066 067 068 069 080 081 082 083 084 085
Topic Identifier
045 20.68 090 11.98 135 6.62 170 0.00 Ad−Hoc Robust Monolingual Italian track − Comparison to Median Mean Average Precision by Topic (Topics 086 to 130)
1
046 10.36 091 5.29 136 62.98 171 42.53 itFSitR6S
047 10.96 092 22.74 137 40.00 172 0.00 0.5
Difference
048 12.17 093 61.98 138 62.42 173 75.00 0
049 25.87 094 35.29 139 11.68 174 74.84 −0.5
060 22.02 095 2.91 140 14.06 175 0.00 −1
086 087 088 089 090 091 092 093 094 095 096 097 098 099 110 111 112 113 114 115 116 117 118 119 130
061 27.20 096 23.29 141 53.66 176 52.69 Topic Identifier
Ad−Hoc Robust Monolingual Italian track − Comparison to Median Mean Average Precision by Topic (Topics 131 to 165)
062 51.46 097 62.50 142 75.82 177 48.23 1
itFSitR6S
063 57.22 098 13.98 143 31.93 178 42.19 0.5
Difference
064 0.00 099 43.91 144 0.00 179 55.49
0
065 29.67 110 84.51 145 100.00 190 46.39
−0.5
066 79.98 111 14.15 146 0.00 191 0.00
−1
067 42.17 112 39.30 147 33.58 192 46.67 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 160 161 162 163 164 165
Topic Identifier
068 40.51 113 12.30 148 9.38 193 20.28 Ad−Hoc Robust Monolingual Italian track − Comparison to Median Mean Average Precision by Topic (Topics 166 to 200)
1
069 6.55 114 46.67 149 30.36 194 31.71 itFSitR6S
0.5
080 41.10 115 42.25 160 0.00 195 64.34
Difference
081 33.77 116 41.85 161 100.00 196 100.00 0
082 75.68 117 35.23 162 64.43 197 66.52 −0.5
083 52.78 118 25.98 163 48.82 198 30.47 −1
166 167 168 169 170 171 172 173 174 175 176 177 178 179 190 191 192 193 194 195 196 197 198 199 200
084 11.94 119 55.53 164 28.84 199 43.92 Topic Identifier
085 54.11 130 37.53 165 100.00 200 56.79
211
daedalus itFSitR6S AH-ROBUST-MONO-IT-CLEF2006
Docs Cutoff Levels Precision at DCL (%)
Ad−Hoc Robust Monolingual Italian track − Retrieved documents vs Precision
100%
5 docs 39.80
itFSitR6S
10 docs 35.20 90%
15 docs 32.00
80%
20 docs 29.10
30 docs 24.67 70%
100 docs 12.31
60%
200 docs 7.45
R−Precision
500 docs 3.37 50%
1000 docs 1.76 40%
R-Precision (precision after R document retrieved,
where R = Relevant retrieved) 30%
36.49
20%
10%
0%
5 10 15 20 30 100 200 500 1000
Retrieved Documents (logarithmic scale)
Exact R-Precision Ad−Hoc Robust Monolingual Italian track − Box plot of the Topics of the Experiment
Maximum 1.0000
Minimum 0.0000
First Quartile 0.1667
Second Quartile 0.3657
Third Quartile 0.5000
Interquartile range 0.3333
Mean 0.3649
Standard Deviation 0.2587
Lower Outlier Threshold 0.0000
Upper Outlier Threshold 0.7857 0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Exact R−Precision
Mean With No Outliers 0.3315
Std With No Outliers 0.2189
Ad−Hoc Robust Monolingual Italian track − Distribution of the Topics of the Experiment
20
Number of Topics of the Experiment
15
10
5
0
0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Exact R−Precision
Precision averages (%) for individual Ad−Hoc Robust Monolingual Italian track − Comparison to Median Mean Exact R−Precision by Topic (Topics 041 to 085)
queries 1
itFSitR6S
0.5
041 57.14 086 19.35 131 19.51 166 0.00
Difference
0
042 34.94 087 50.00 132 33.33 167 100.00
−0.5
043 0.00 088 33.33 133 37.50 168 31.25
044 0.00 089 76.92 134 60.00 169 0.00 −1
041 042 043 044 045 046 047 048 049 060 061 062 063 064 065 066 067 068 069 080 081 082 083 084 085
Topic Identifier
045 21.43 090 33.33 135 15.58 170 0.00 Ad−Hoc Robust Monolingual Italian track − Comparison to Median Mean Exact R−Precision by Topic (Topics 086 to 130)
1
046 12.50 091 0.00 136 50.00 171 42.86 itFSitR6S
047 11.54 092 19.44 137 50.00 172 0.00 0.5
Difference
048 16.00 093 77.78 138 53.85 173 50.00 0
049 37.93 094 45.00 139 0.00 174 62.50 −0.5
060 32.39 095 10.00 140 22.00 175 0.00 −1
086 087 088 089 090 091 092 093 094 095 096 097 098 099 110 111 112 113 114 115 116 117 118 119 130
061 31.25 096 25.00 141 42.86 176 56.25 Topic Identifier
Ad−Hoc Robust Monolingual Italian track − Comparison to Median Mean Exact R−Precision by Topic (Topics 131 to 165)
062 38.46 097 50.00 142 78.57 177 50.00 1
itFSitR6S
063 37.50 098 16.67 143 38.46 178 50.00 0.5
Difference
064 0.00 099 43.59 144 0.00 179 55.00
0
065 33.33 110 73.91 145 100.00 190 50.00
−0.5
066 72.97 111 27.27 146 0.00 191 0.00
−1
067 44.12 112 33.33 147 40.00 192 48.39 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 160 161 162 163 164 165
Topic Identifier
068 40.00 113 16.67 148 0.00 193 0.00 Ad−Hoc Robust Monolingual Italian track − Comparison to Median Mean Exact R−Precision by Topic (Topics 166 to 200)
1
069 16.67 114 38.46 149 0.00 194 30.77 itFSitR6S
0.5
080 44.05 115 51.85 160 0.00 195 58.33
Difference
081 37.04 116 30.00 161 100.00 196 100.00 0
082 60.61 117 35.29 162 64.29 197 60.00 −0.5
083 33.33 118 25.00 163 56.52 198 28.57 −1
166 167 168 169 170 171 172 173 174 175 176 177 178 179 190 191 192 193 194 195 196 197 198 199 200
084 0.00 119 50.00 164 32.26 199 36.11 Topic Identifier
085 49.32 130 45.00 165 100.00 200 54.29
212
dcu dcu1500narrit1005.qy1 AH-ROBUST-MONO-IT-CLEF2006
Overall statistics for 100 queries : Priority 1
Total number of documents over all queries Query Construction AUTOMATIC
Retrieved 100,000 Source Language Italian
Relevant 1,930 Topic Fields title, description, narrative
Relevant retrieved 1,787 Pooled false
Geometric Mean Average Precision 0.1107 italian mono reranked
Binary Preference (BPREF) 0.4392
Interploated Recall (%) Precision Averages (%)
Ad−Hoc Robust Monolingual Italian track − Interpolated Recall vs Average Precision
100%
0 69.06
dcu1500narrit1005.qy1
10 61.53 90%
20 58.02
80%
30 51.94
40 46.48 70%
50 43.23
Average Precision
60%
60 38.29
70 33.21 50%
80 27.10 40%
90 21.20
30%
100 16.62
Average precision (non-interpolated) for all 20%
relevant documents (averaged over queries)
40.94 10%
0%
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
Interpolated Recall
Mean Average Precision Ad−Hoc Robust Monolingual Italian track − Box plot of the Topics of the Experiment
Maximum 1.0000
Minimum 0.0000
First Quartile 0.1822
Second Quartile 0.3886
Third Quartile 0.6313
Interquartile range 0.4491
Mean 0.4094
Standard Deviation 0.2878
Lower Outlier Threshold 0.0000
Upper Outlier Threshold 1.0000 0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Mean Average Precision
Mean With No Outliers 0.4094
Std With No Outliers 0.2878
Ad−Hoc Robust Monolingual Italian track − Distribution of the Topics of the Experiment
15
Number of Topics of the Experiment
10
5
0
0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Mean Average Precision
Precision averages (%) for individual Ad−Hoc Robust Monolingual Italian track − Comparison to Median Mean Average Precision by Topic (Topics 041 to 085)
queries 1
dcu1500narrit1005.qy1
0.5
041 56.80 086 25.54 131 4.80 166 0.18
Difference
0
042 18.59 087 47.71 132 47.67 167 100.00
−0.5
043 0.00 088 36.52 133 59.93 168 46.04
044 25.76 089 79.71 134 90.00 169 0.00 −1
041 042 043 044 045 046 047 048 049 060 061 062 063 064 065 066 067 068 069 080 081 082 083 084 085
Topic Identifier
045 23.35 090 6.11 135 74.13 170 0.00 Ad−Hoc Robust Monolingual Italian track − Comparison to Median Mean Average Precision by Topic (Topics 086 to 130)
1
046 17.84 091 51.03 136 68.09 171 28.49 dcu1500narrit1005.qy1
047 9.11 092 19.24 137 60.42 172 0.00 0.5
Difference
048 11.87 093 67.63 138 78.67 173 58.33 0
049 21.21 094 45.34 139 5.56 174 78.03 −0.5
060 19.35 095 4.09 140 48.21 175 0.00 −1
086 087 088 089 090 091 092 093 094 095 096 097 098 099 110 111 112 113 114 115 116 117 118 119 130
061 33.55 096 13.10 141 94.05 176 61.65 Topic Identifier
Ad−Hoc Robust Monolingual Italian track − Comparison to Median Mean Average Precision by Topic (Topics 131 to 165)
062 64.16 097 69.51 142 99.01 177 46.50 1
dcu1500narrit1005.qy1
063 70.22 098 48.54 143 30.91 178 29.84 0.5
Difference
064 0.00 099 48.85 144 0.00 179 69.67
0
065 31.54 110 78.05 145 100.00 190 52.74
−0.5
066 80.54 111 33.51 146 0.00 191 0.00
−1
067 38.77 112 63.14 147 24.93 192 36.14 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 160 161 162 163 164 165
Topic Identifier
068 6.92 113 10.12 148 10.27 193 18.64 Ad−Hoc Robust Monolingual Italian track − Comparison to Median Mean Average Precision by Topic (Topics 166 to 200)
1
069 9.88 114 51.33 149 37.30 194 52.19 dcu1500narrit1005.qy1
0.5
080 30.56 115 42.16 160 0.00 195 56.97
Difference
081 36.69 116 39.20 161 0.00 196 100.00 0
082 63.12 117 35.49 162 70.80 197 70.91 −0.5
083 66.67 118 26.88 163 68.35 198 24.55 −1
166 167 168 169 170 171 172 173 174 175 176 177 178 179 190 191 192 193 194 195 196 197 198 199 200
084 9.62 119 66.21 164 27.24 199 58.05 Topic Identifier
085 61.16 130 38.95 165 100.00 200 49.15
213
dcu dcu1500narrit1005.qy1 AH-ROBUST-MONO-IT-CLEF2006
Docs Cutoff Levels Precision at DCL (%)
Ad−Hoc Robust Monolingual Italian track − Retrieved documents vs Precision
100%
5 docs 46.00
dcu1500narrit1005.qy1
10 docs 38.60 90%
15 docs 35.20
80%
20 docs 32.05
30 docs 26.67 70%
100 docs 12.75
60%
200 docs 7.61
R−Precision
500 docs 3.42 50%
1000 docs 1.79 40%
R-Precision (precision after R document retrieved,
where R = Relevant retrieved) 30%
38.86
20%
10%
0%
5 10 15 20 30 100 200 500 1000
Retrieved Documents (logarithmic scale)
Exact R-Precision Ad−Hoc Robust Monolingual Italian track − Box plot of the Topics of the Experiment
Maximum 1.0000
Minimum 0.0000
First Quartile 0.1952
Second Quartile 0.4000
Third Quartile 0.5670
Interquartile range 0.3717
Mean 0.3886
Standard Deviation 0.2716
Lower Outlier Threshold 0.0000
Upper Outlier Threshold 1.0000 0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Exact R−Precision
Mean With No Outliers 0.3886
Std With No Outliers 0.2716
Ad−Hoc Robust Monolingual Italian track − Distribution of the Topics of the Experiment
20
Number of Topics of the Experiment
15
10
5
0
0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Exact R−Precision
Precision averages (%) for individual Ad−Hoc Robust Monolingual Italian track − Comparison to Median Mean Exact R−Precision by Topic (Topics 041 to 085)
queries 1
dcu1500narrit1005.qy1
0.5
041 57.14 086 25.81 131 12.20 166 0.00
Difference
0
042 27.71 087 40.00 132 33.33 167 100.00
−0.5
043 0.00 088 40.00 133 56.25 168 46.88
044 0.00 089 76.92 134 80.00 169 0.00 −1
041 042 043 044 045 046 047 048 049 060 061 062 063 064 065 066 067 068 069 080 081 082 083 084 085
Topic Identifier
045 19.05 090 0.00 135 68.83 170 0.00 Ad−Hoc Robust Monolingual Italian track − Comparison to Median Mean Exact R−Precision by Topic (Topics 086 to 130)
1
046 25.00 091 50.00 136 58.33 171 28.57 dcu1500narrit1005.qy1
047 7.69 092 22.22 137 75.00 172 0.00 0.5
Difference
048 20.00 093 77.78 138 69.23 173 50.00 0
049 34.48 094 50.00 139 0.00 174 62.50 −0.5
060 28.17 095 5.71 140 40.00 175 0.00 −1
086 087 088 089 090 091 092 093 094 095 096 097 098 099 110 111 112 113 114 115 116 117 118 119 130
061 50.00 096 0.00 141 85.71 176 56.25 Topic Identifier
Ad−Hoc Robust Monolingual Italian track − Comparison to Median Mean Exact R−Precision by Topic (Topics 131 to 165)
062 53.85 097 75.00 142 92.86 177 33.33 1
dcu1500narrit1005.qy1
063 62.50 098 50.00 143 34.62 178 25.00 0.5
Difference
064 0.00 099 46.15 144 0.00 179 62.50
0
065 40.00 110 65.22 145 100.00 190 41.67
−0.5
066 70.27 111 45.45 146 0.00 191 0.00
−1
067 38.24 112 50.00 147 20.00 192 41.94 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 160 161 162 163 164 165
Topic Identifier
068 16.67 113 16.67 148 0.00 193 25.00 Ad−Hoc Robust Monolingual Italian track − Comparison to Median Mean Exact R−Precision by Topic (Topics 166 to 200)
1
069 16.67 114 46.15 149 33.33 194 53.85 dcu1500narrit1005.qy1
0.5
080 32.14 115 44.44 160 0.00 195 41.67
Difference
081 40.74 116 40.00 161 0.00 196 100.00 0
082 51.52 117 35.29 162 64.29 197 67.27 −0.5
083 33.33 118 25.00 163 60.87 198 28.57 −1
166 167 168 169 170 171 172 173 174 175 176 177 178 179 190 191 192 193 194 195 196 197 198 199 200
084 0.00 119 60.00 164 29.03 199 52.78 Topic Identifier
085 57.53 130 35.00 165 100.00 200 48.57
214
dcu dcudescit1005 AH-ROBUST-MONO-IT-CLEF2006
Overall statistics for 100 queries : Priority 2
Total number of documents over all queries Query Construction AUTOMATIC
Retrieved 100,000 Source Language Italian
Relevant 1,930 Topic Fields title, description
Relevant retrieved 1,778 Pooled false
Geometric Mean Average Precision 0.0919 italian mono standard
Binary Preference (BPREF) 0.4006
Interploated Recall (%) Precision Averages (%)
Ad−Hoc Robust Monolingual Italian track − Interpolated Recall vs Average Precision
100%
0 62.97
dcudescit1005
10 59.41 90%
20 54.14
80%
30 47.34
40 42.33 70%
50 37.90
Average Precision
60%
60 34.14
70 29.64 50%
80 25.65 40%
90 20.88
30%
100 16.43
Average precision (non-interpolated) for all 20%
relevant documents (averaged over queries)
37.73 10%
0%
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
Interpolated Recall
Mean Average Precision Ad−Hoc Robust Monolingual Italian track − Box plot of the Topics of the Experiment
Maximum 1.0000
Minimum 0.0000
First Quartile 0.0919
Second Quartile 0.3686
Third Quartile 0.6015
Interquartile range 0.5095
Mean 0.3773
Standard Deviation 0.2991
Lower Outlier Threshold 0.0000
Upper Outlier Threshold 1.0000 0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Mean Average Precision
Mean With No Outliers 0.3773
Std With No Outliers 0.2991
Ad−Hoc Robust Monolingual Italian track − Distribution of the Topics of the Experiment
20
Number of Topics of the Experiment
15
10
5
0
0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Mean Average Precision
Precision averages (%) for individual Ad−Hoc Robust Monolingual Italian track − Comparison to Median Mean Average Precision by Topic (Topics 041 to 085)
queries 1
dcudescit1005
0.5
041 68.78 086 14.76 131 2.11 166 0.15
Difference
0
042 4.99 087 61.00 132 42.46 167 100.00
−0.5
043 0.00 088 43.56 133 49.13 168 41.61
044 8.93 089 90.21 134 77.67 169 0.00 −1
041 042 043 044 045 046 047 048 049 060 061 062 063 064 065 066 067 068 069 080 081 082 083 084 085
Topic Identifier
045 32.19 090 8.61 135 65.24 170 0.00 Ad−Hoc Robust Monolingual Italian track − Comparison to Median Mean Average Precision by Topic (Topics 086 to 130)
1
046 8.17 091 6.32 136 69.65 171 43.79 dcudescit1005
047 2.75 092 24.55 137 20.03 172 0.00 0.5
Difference
048 3.05 093 79.47 138 56.34 173 29.17 0
049 23.30 094 54.98 139 4.68 174 75.28 −0.5
060 54.57 095 3.22 140 17.36 175 0.00 −1
086 087 088 089 090 091 092 093 094 095 096 097 098 099 110 111 112 113 114 115 116 117 118 119 130
061 45.98 096 36.09 141 100.00 176 64.80 Topic Identifier
Ad−Hoc Robust Monolingual Italian track − Comparison to Median Mean Average Precision by Topic (Topics 131 to 165)
062 37.54 097 37.04 142 99.52 177 58.26 1
dcudescit1005
063 76.45 098 22.77 143 21.23 178 36.70 0.5
Difference
064 0.00 099 58.85 144 0.00 179 62.01
0
065 17.46 110 86.46 145 100.00 190 49.21
−0.5
066 76.70 111 16.03 146 0.00 191 0.00
−1
067 38.51 112 61.44 147 7.86 192 51.14 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 160 161 162 163 164 165
Topic Identifier
068 15.44 113 1.08 148 9.58 193 42.29 Ad−Hoc Robust Monolingual Italian track − Comparison to Median Mean Average Precision by Topic (Topics 166 to 200)
1
069 4.29 114 34.91 149 21.58 194 48.34 dcudescit1005
0.5
080 28.11 115 40.54 160 0.00 195 54.88
Difference
081 45.57 116 30.08 161 0.00 196 100.00 0
082 60.99 117 21.00 162 59.30 197 70.39 −0.5
083 73.33 118 9.71 163 67.84 198 9.46 −1
166 167 168 169 170 171 172 173 174 175 176 177 178 179 190 191 192 193 194 195 196 197 198 199 200
084 9.83 119 76.53 164 35.76 199 46.93 Topic Identifier
085 37.03 130 30.03 165 100.00 200 37.79
215
dcu dcudescit1005 AH-ROBUST-MONO-IT-CLEF2006
Docs Cutoff Levels Precision at DCL (%)
Ad−Hoc Robust Monolingual Italian track − Retrieved documents vs Precision
100%
5 docs 41.00
dcudescit1005
10 docs 35.80 90%
15 docs 32.73
80%
20 docs 29.00
30 docs 25.07 70%
100 docs 12.27
60%
200 docs 7.33
R−Precision
500 docs 3.33 50%
1000 docs 1.78 40%
R-Precision (precision after R document retrieved,
where R = Relevant retrieved) 30%
35.21
20%
10%
0%
5 10 15 20 30 100 200 500 1000
Retrieved Documents (logarithmic scale)
Exact R-Precision Ad−Hoc Robust Monolingual Italian track − Box plot of the Topics of the Experiment
Maximum 1.0000
Minimum 0.0000
First Quartile 0.0425
Second Quartile 0.3333
Third Quartile 0.5500
Interquartile range 0.5075
Mean 0.3521
Standard Deviation 0.2899
Lower Outlier Threshold 0.0000
Upper Outlier Threshold 1.0000 0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Exact R−Precision
Mean With No Outliers 0.3521
Std With No Outliers 0.2899
Ad−Hoc Robust Monolingual Italian track − Distribution of the Topics of the Experiment
30
Number of Topics of the Experiment
25
20
15
10
5
0
0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Exact R−Precision
Precision averages (%) for individual Ad−Hoc Robust Monolingual Italian track − Comparison to Median Mean Exact R−Precision by Topic (Topics 041 to 085)
queries 1
dcudescit1005
0.5
041 57.14 086 19.35 131 4.88 166 0.00
Difference
0
042 3.61 087 40.00 132 33.33 167 100.00
−0.5
043 0.00 088 33.33 133 43.75 168 40.62
044 0.00 089 84.62 134 80.00 169 0.00 −1
041 042 043 044 045 046 047 048 049 060 061 062 063 064 065 066 067 068 069 080 081 082 083 084 085
Topic Identifier
045 33.33 090 0.00 135 62.34 170 0.00 Ad−Hoc Robust Monolingual Italian track − Comparison to Median Mean Exact R−Precision by Topic (Topics 086 to 130)
1
046 12.50 091 0.00 136 66.67 171 42.86 dcudescit1005
047 7.69 092 27.78 137 25.00 172 0.00 0.5
Difference
048 8.00 093 77.78 138 46.15 173 0.00 0
049 24.14 094 55.00 139 0.00 174 75.00 −0.5
060 56.34 095 2.86 140 20.00 175 0.00 −1
086 087 088 089 090 091 092 093 094 095 096 097 098 099 110 111 112 113 114 115 116 117 118 119 130
061 43.75 096 25.00 141 100.00 176 56.25 Topic Identifier
Ad−Hoc Robust Monolingual Italian track − Comparison to Median Mean Exact R−Precision by Topic (Topics 131 to 165)
062 46.15 097 50.00 142 92.86 177 50.00 1
dcudescit1005
063 62.50 098 25.00 143 26.92 178 25.00 0.5
Difference
064 0.00 099 48.72 144 0.00 179 55.00
0
065 33.33 110 78.26 145 100.00 190 41.67
−0.5
066 70.27 111 9.09 146 0.00 191 0.00
−1
067 32.35 112 50.00 147 20.00 192 48.39 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 160 161 162 163 164 165
Topic Identifier
068 20.00 113 0.00 148 0.00 193 25.00 Ad−Hoc Robust Monolingual Italian track − Comparison to Median Mean Exact R−Precision by Topic (Topics 166 to 200)
1
069 0.00 114 30.77 149 0.00 194 30.77 dcudescit1005
0.5
080 32.14 115 48.15 160 0.00 195 58.33
Difference
081 44.44 116 40.00 161 0.00 196 100.00 0
082 57.58 117 23.53 162 42.86 197 64.55 −0.5
083 66.67 118 25.00 163 60.87 198 0.00 −1
166 167 168 169 170 171 172 173 174 175 176 177 178 179 190 191 192 193 194 195 196 197 198 199 200
084 0.00 119 71.67 164 41.94 199 50.00 Topic Identifier
085 39.73 130 30.00 165 100.00 200 42.86
216
hummingbird humIT06Rtd AH-ROBUST-MONO-IT-CLEF2006
Overall statistics for 100 queries : Priority 1
Total number of documents over all queries Query Construction AUTOMATIC
Retrieved 100,000 Source Language Italian
Relevant 1,930 Topic Fields title, description
Relevant retrieved 1,803 Pooled false
Geometric Mean Average Precision 0.1009 plain SearchServer stemming run
Binary Preference (BPREF) 0.3977
Interploated Recall (%) Precision Averages (%)
Ad−Hoc Robust Monolingual Italian track − Interpolated Recall vs Average Precision
100%
0 65.75
humIT06Rtd
10 58.93 90%
20 53.08
80%
30 47.05
40 43.13 70%
50 39.56
Average Precision
60%
60 32.12
70 27.43 50%
80 23.05 40%
90 19.15
30%
100 15.17
Average precision (non-interpolated) for all 20%
relevant documents (averaged over queries)
36.83 10%
0%
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
Interpolated Recall
Mean Average Precision Ad−Hoc Robust Monolingual Italian track − Box plot of the Topics of the Experiment
Maximum 1.0000
Minimum 0.0000
First Quartile 0.1541
Second Quartile 0.3451
Third Quartile 0.5398
Interquartile range 0.3858
Mean 0.3683
Standard Deviation 0.2680
Lower Outlier Threshold 0.0000
Upper Outlier Threshold 1.0000 0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Mean Average Precision
Mean With No Outliers 0.3683
Std With No Outliers 0.2680
Ad−Hoc Robust Monolingual Italian track − Distribution of the Topics of the Experiment
20
Number of Topics of the Experiment
15
10
5
0
0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Mean Average Precision
Precision averages (%) for individual Ad−Hoc Robust Monolingual Italian track − Comparison to Median Mean Average Precision by Topic (Topics 041 to 085)
queries 1
humIT06Rtd
0.5
041 64.91 086 16.90 131 32.03 166 0.75
Difference
0
042 37.18 087 44.29 132 27.49 167 83.33
−0.5
043 0.00 088 44.86 133 34.41 168 16.32
044 15.26 089 78.39 134 81.88 169 0.00 −1
041 042 043 044 045 046 047 048 049 060 061 062 063 064 065 066 067 068 069 080 081 082 083 084 085
Topic Identifier
045 21.78 090 2.19 135 40.94 170 0.00 Ad−Hoc Robust Monolingual Italian track − Comparison to Median Mean Average Precision by Topic (Topics 086 to 130)
1
046 12.59 091 50.73 136 67.54 171 49.23 humIT06Rtd
047 2.89 092 27.23 137 25.96 172 0.00 0.5
Difference
048 9.37 093 83.56 138 30.62 173 36.67 0
049 29.58 094 42.23 139 8.16 174 75.90 −0.5
060 26.94 095 3.57 140 20.78 175 0.00 −1
086 087 088 089 090 091 092 093 094 095 096 097 098 099 110 111 112 113 114 115 116 117 118 119 130
061 25.33 096 32.42 141 53.02 176 53.69 Topic Identifier
Ad−Hoc Robust Monolingual Italian track − Comparison to Median Mean Average Precision by Topic (Topics 131 to 165)
062 43.40 097 54.73 142 95.29 177 64.57 1
humIT06Rtd
063 51.16 098 70.49 143 35.78 178 20.45 0.5
Difference
064 0.00 099 42.58 144 0.00 179 57.92
0
065 16.51 110 77.89 145 100.00 190 43.00
−0.5
066 80.66 111 11.63 146 0.00 191 0.00
−1
067 38.67 112 53.89 147 23.92 192 37.05 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 160 161 162 163 164 165
Topic Identifier
068 28.78 113 6.89 148 15.56 193 34.62 Ad−Hoc Robust Monolingual Italian track − Comparison to Median Mean Average Precision by Topic (Topics 166 to 200)
1
069 2.45 114 30.60 149 33.13 194 45.36 humIT06Rtd
0.5
080 41.41 115 41.20 160 0.00 195 70.90
Difference
081 30.06 116 29.85 161 0.00 196 100.00 0
082 57.91 117 38.31 162 54.08 197 63.46 −0.5
083 75.76 118 13.24 163 49.89 198 15.26 −1
166 167 168 169 170 171 172 173 174 175 176 177 178 179 190 191 192 193 194 195 196 197 198 199 200
084 12.93 119 55.86 164 17.45 199 65.30 Topic Identifier
085 54.63 130 30.91 165 100.00 200 37.16
217
hummingbird humIT06Rtd AH-ROBUST-MONO-IT-CLEF2006
Docs Cutoff Levels Precision at DCL (%)
Ad−Hoc Robust Monolingual Italian track − Retrieved documents vs Precision
100%
5 docs 43.40
humIT06Rtd
10 docs 36.60 90%
15 docs 31.80
80%
20 docs 28.80
30 docs 24.83 70%
100 docs 12.40
60%
200 docs 7.60
R−Precision
500 docs 3.44 50%
1000 docs 1.80 40%
R-Precision (precision after R document retrieved,
where R = Relevant retrieved) 30%
36.34
20%
10%
0%
5 10 15 20 30 100 200 500 1000
Retrieved Documents (logarithmic scale)
Exact R-Precision Ad−Hoc Robust Monolingual Italian track − Box plot of the Topics of the Experiment
Maximum 1.0000
Minimum 0.0000
First Quartile 0.1606
Second Quartile 0.4000
Third Quartile 0.5000
Interquartile range 0.3394
Mean 0.3634
Standard Deviation 0.2551
Lower Outlier Threshold 0.0000
Upper Outlier Threshold 1.0000 0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Exact R−Precision
Mean With No Outliers 0.3634
Std With No Outliers 0.2551
Ad−Hoc Robust Monolingual Italian track − Distribution of the Topics of the Experiment
20
Number of Topics of the Experiment
15
10
5
0
0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Exact R−Precision
Precision averages (%) for individual Ad−Hoc Robust Monolingual Italian track − Comparison to Median Mean Exact R−Precision by Topic (Topics 041 to 085)
queries 1
humIT06Rtd
0.5
041 57.14 086 25.81 131 46.34 166 0.00
Difference
0
042 43.37 087 40.00 132 33.33 167 50.00
−0.5
043 0.00 088 40.00 133 50.00 168 28.12
044 0.00 089 76.92 134 80.00 169 0.00 −1
041 042 043 044 045 046 047 048 049 060 061 062 063 064 065 066 067 068 069 080 081 082 083 084 085
Topic Identifier
045 19.05 090 0.00 135 46.75 170 0.00 Ad−Hoc Robust Monolingual Italian track − Comparison to Median Mean Exact R−Precision by Topic (Topics 086 to 130)
1
046 12.50 091 50.00 136 50.00 171 42.86 humIT06Rtd
047 0.00 092 22.22 137 50.00 172 0.00 0.5
Difference
048 16.00 093 77.78 138 30.77 173 0.00 0
049 31.03 094 40.00 139 0.00 174 62.50 −0.5
060 28.17 095 5.71 140 18.00 175 0.00 −1
086 087 088 089 090 091 092 093 094 095 096 097 098 099 110 111 112 113 114 115 116 117 118 119 130
061 31.25 096 50.00 141 42.86 176 50.00 Topic Identifier
Ad−Hoc Robust Monolingual Italian track − Comparison to Median Mean Exact R−Precision by Topic (Topics 131 to 165)
062 38.46 097 50.00 142 92.86 177 50.00 1
humIT06Rtd
063 37.50 098 66.67 143 34.62 178 25.00 0.5
Difference
064 0.00 099 46.15 144 0.00 179 57.50
0
065 26.67 110 69.57 145 100.00 190 50.00
−0.5
066 72.97 111 27.27 146 0.00 191 0.00
−1
067 44.12 112 50.00 147 40.00 192 45.16 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 160 161 162 163 164 165
Topic Identifier
068 36.67 113 5.56 148 0.00 193 50.00 Ad−Hoc Robust Monolingual Italian track − Comparison to Median Mean Exact R−Precision by Topic (Topics 166 to 200)
1
069 0.00 114 38.46 149 33.33 194 38.46 humIT06Rtd
0.5
080 42.86 115 48.15 160 0.00 195 58.33
Difference
081 40.74 116 40.00 161 0.00 196 100.00 0
082 48.48 117 52.94 162 50.00 197 60.00 −0.5
083 66.67 118 25.00 163 47.83 198 14.29 −1
166 167 168 169 170 171 172 173 174 175 176 177 178 179 190 191 192 193 194 195 196 197 198 199 200
084 0.00 119 60.00 164 16.13 199 55.56 Topic Identifier
085 53.42 130 35.00 165 100.00 200 42.86
218
hummingbird humIT06Rtde AH-ROBUST-MONO-IT-CLEF2006
Overall statistics for 100 queries : Priority 2
Total number of documents over all queries Query Construction AUTOMATIC
Retrieved 100,000 Source Language Italian
Relevant 1,930 Topic Fields title, description
Relevant retrieved 1,804 Pooled false
Geometric Mean Average Precision 0.1147 same as humIT06Rtd except expansion from blind
Binary Preference (BPREF) 0.4501 feedback also used
Interploated Recall (%) Precision Averages (%)
Ad−Hoc Robust Monolingual Italian track − Interpolated Recall vs Average Precision
100%
0 66.67
humIT06Rtde
10 62.27 90%
20 57.38
80%
30 53.07
40 48.98 70%
50 44.60
Average Precision
60%
60 38.98
70 34.28 50%
80 29.01 40%
90 24.41
30%
100 18.44
Average precision (non-interpolated) for all 20%
relevant documents (averaged over queries)
41.94 10%
0%
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
Interpolated Recall
Mean Average Precision Ad−Hoc Robust Monolingual Italian track − Box plot of the Topics of the Experiment
Maximum 1.0000
Minimum 0.0000
First Quartile 0.1709
Second Quartile 0.4297
Third Quartile 0.6393
Interquartile range 0.4684
Mean 0.4194
Standard Deviation 0.2852
Lower Outlier Threshold 0.0000
Upper Outlier Threshold 1.0000 0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Mean Average Precision
Mean With No Outliers 0.4194
Std With No Outliers 0.2852
Ad−Hoc Robust Monolingual Italian track − Distribution of the Topics of the Experiment
15
Number of Topics of the Experiment
10
5
0
0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Mean Average Precision
Precision averages (%) for individual Ad−Hoc Robust Monolingual Italian track − Comparison to Median Mean Average Precision by Topic (Topics 041 to 085)
queries 1
humIT06Rtde
0.5
041 71.90 086 16.49 131 32.88 166 0.28
Difference
0
042 42.73 087 65.80 132 41.67 167 100.00
−0.5
043 0.00 088 48.19 133 50.54 168 23.49
044 17.69 089 81.43 134 90.00 169 0.00 −1
041 042 043 044 045 046 047 048 049 060 061 062 063 064 065 066 067 068 069 080 081 082 083 084 085
Topic Identifier
045 23.09 090 3.37 135 62.92 170 0.00 Ad−Hoc Robust Monolingual Italian track − Comparison to Median Mean Average Precision by Topic (Topics 086 to 130)
1
046 12.12 091 42.95 136 67.31 171 43.58 humIT06Rtde
047 3.66 092 29.66 137 29.41 172 0.00 0.5
Difference
048 11.04 093 87.09 138 44.34 173 50.00 0
049 32.23 094 47.87 139 7.30 174 88.87 −0.5
060 49.44 095 6.60 140 19.88 175 0.00 −1
086 087 088 089 090 091 092 093 094 095 096 097 098 099 110 111 112 113 114 115 116 117 118 119 130
061 33.28 096 26.78 141 75.90 176 56.64 Topic Identifier
Ad−Hoc Robust Monolingual Italian track − Comparison to Median Mean Average Precision by Topic (Topics 131 to 165)
062 53.90 097 65.48 142 97.22 177 64.58 1
humIT06Rtde
063 55.55 098 70.85 143 27.31 178 19.13 0.5
Difference
064 0.00 099 60.68 144 0.00 179 77.01
0
065 19.93 110 83.35 145 100.00 190 63.29
−0.5
066 76.66 111 15.92 146 0.00 191 0.00
−1
067 45.65 112 53.74 147 24.77 192 80.93 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 160 161 162 163 164 165
Topic Identifier
068 42.99 113 9.65 148 15.56 193 42.28 Ad−Hoc Robust Monolingual Italian track − Comparison to Median Mean Average Precision by Topic (Topics 166 to 200)
1
069 4.56 114 39.61 149 33.33 194 52.25 humIT06Rtde
0.5
080 36.63 115 51.64 160 0.00 195 77.28
Difference
081 40.22 116 33.59 161 0.00 196 100.00 0
082 78.04 117 45.99 162 67.20 197 70.93 −0.5
083 59.56 118 43.33 163 58.80 198 12.24 −1
166 167 168 169 170 171 172 173 174 175 176 177 178 179 190 191 192 193 194 195 196 197 198 199 200
084 14.73 119 66.39 164 23.33 199 72.22 Topic Identifier
085 60.70 130 35.32 165 50.00 200 61.09
219
hummingbird humIT06Rtde AH-ROBUST-MONO-IT-CLEF2006
Docs Cutoff Levels Precision at DCL (%)
Ad−Hoc Robust Monolingual Italian track − Retrieved documents vs Precision
100%
5 docs 45.20
humIT06Rtde
10 docs 40.20 90%
15 docs 36.60
80%
20 docs 33.10
30 docs 28.00 70%
100 docs 13.93
60%
200 docs 8.20
R−Precision
500 docs 3.55 50%
1000 docs 1.80 40%
R-Precision (precision after R document retrieved,
where R = Relevant retrieved) 30%
39.70
20%
10%
0%
5 10 15 20 30 100 200 500 1000
Retrieved Documents (logarithmic scale)
Exact R-Precision Ad−Hoc Robust Monolingual Italian track − Box plot of the Topics of the Experiment
Maximum 1.0000
Minimum 0.0000
First Quartile 0.1633
Second Quartile 0.4330
Third Quartile 0.5858
Interquartile range 0.4225
Mean 0.3970
Standard Deviation 0.2701
Lower Outlier Threshold 0.0000
Upper Outlier Threshold 1.0000 0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Exact R−Precision
Mean With No Outliers 0.3970
Std With No Outliers 0.2701
Ad−Hoc Robust Monolingual Italian track − Distribution of the Topics of the Experiment
20
Number of Topics of the Experiment
15
10
5
0
0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Exact R−Precision
Precision averages (%) for individual Ad−Hoc Robust Monolingual Italian track − Comparison to Median Mean Exact R−Precision by Topic (Topics 041 to 085)
queries 1
humIT06Rtde
0.5
041 71.43 086 22.58 131 46.34 166 0.00
Difference
0
042 45.78 087 60.00 132 33.33 167 100.00
−0.5
043 0.00 088 40.00 133 56.25 168 34.38
044 0.00 089 80.77 134 80.00 169 0.00 −1
041 042 043 044 045 046 047 048 049 060 061 062 063 064 065 066 067 068 069 080 081 082 083 084 085
Topic Identifier
045 21.43 090 0.00 135 62.34 170 0.00 Ad−Hoc Robust Monolingual Italian track − Comparison to Median Mean Exact R−Precision by Topic (Topics 086 to 130)
1
046 6.25 091 50.00 136 50.00 171 42.86 humIT06Rtde
047 3.85 092 33.33 137 50.00 172 0.00 0.5
Difference
048 16.00 093 77.78 138 46.15 173 50.00 0
049 31.03 094 45.00 139 0.00 174 75.00 −0.5
060 54.93 095 14.29 140 16.00 175 0.00 −1
086 087 088 089 090 091 092 093 094 095 096 097 098 099 110 111 112 113 114 115 116 117 118 119 130
061 43.75 096 37.50 141 57.14 176 56.25 Topic Identifier
Ad−Hoc Robust Monolingual Italian track − Comparison to Median Mean Exact R−Precision by Topic (Topics 131 to 165)
062 38.46 097 50.00 142 92.86 177 50.00 1
humIT06Rtde
063 50.00 098 58.33 143 30.77 178 25.00 0.5
Difference
064 0.00 099 56.41 144 0.00 179 65.00
0
065 26.67 110 73.91 145 100.00 190 58.33
−0.5
066 70.27 111 27.27 146 0.00 191 0.00
−1
067 44.12 112 58.33 147 40.00 192 70.97 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 160 161 162 163 164 165
Topic Identifier
068 46.67 113 5.56 148 0.00 193 25.00 Ad−Hoc Robust Monolingual Italian track − Comparison to Median Mean Exact R−Precision by Topic (Topics 166 to 200)
1
069 16.67 114 38.46 149 33.33 194 38.46 humIT06Rtde
0.5
080 39.29 115 51.85 160 0.00 195 66.67
Difference
081 48.15 116 30.00 161 0.00 196 100.00 0
082 66.67 117 58.82 162 64.29 197 65.45 −0.5
083 66.67 118 25.00 163 60.87 198 14.29 −1
166 167 168 169 170 171 172 173 174 175 176 177 178 179 190 191 192 193 194 195 196 197 198 199 200
084 0.00 119 63.33 164 29.03 199 61.11 Topic Identifier
085 54.79 130 35.00 165 0.00 200 65.71
220
reina reinaITttest AH-ROBUST-MONO-IT-CLEF2006
Overall statistics for 100 queries : Priority 2
Total number of documents over all queries Query Construction AUTOMATIC
Retrieved 100,000 Source Language Italian
Relevant 1,930 Topic Fields title
Relevant retrieved 1,758 Pooled false
Geometric Mean Average Precision 0.0833 local expansion with term windows, only title field
Binary Preference (BPREF) 0.4007
Interploated Recall (%) Precision Averages (%)
Ad−Hoc Robust Monolingual Italian track − Interpolated Recall vs Average Precision
100%
0 63.46
reinaITttest
10 56.81 90%
20 51.92
80%
30 45.97
40 40.69 70%
50 37.15
Average Precision
60%
60 31.69
70 27.14 50%
80 23.84 40%
90 18.40
30%
100 12.82
Average precision (non-interpolated) for all 20%
relevant documents (averaged over queries)
35.62 10%
0%
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
Interpolated Recall
Mean Average Precision Ad−Hoc Robust Monolingual Italian track − Box plot of the Topics of the Experiment
Maximum 1.0000
Minimum 0.0000
First Quartile 0.1118
Second Quartile 0.3523
Third Quartile 0.5559
Interquartile range 0.4440
Mean 0.3562
Standard Deviation 0.2812
Lower Outlier Threshold 0.0000
Upper Outlier Threshold 1.0000 0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Mean Average Precision
Mean With No Outliers 0.3562
Std With No Outliers 0.2812
Ad−Hoc Robust Monolingual Italian track − Distribution of the Topics of the Experiment
20
Number of Topics of the Experiment
15
10
5
0
0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Mean Average Precision
Precision averages (%) for individual Ad−Hoc Robust Monolingual Italian track − Comparison to Median Mean Average Precision by Topic (Topics 041 to 085)
queries 1
reinaITttest
0.5
041 74.97 086 18.47 131 18.70 166 0.36
Difference
0
042 44.14 087 50.56 132 58.12 167 100.00
−0.5
043 0.00 088 34.74 133 38.41 168 28.91
044 22.50 089 74.13 134 49.39 169 0.00 −1
041 042 043 044 045 046 047 048 049 060 061 062 063 064 065 066 067 068 069 080 081 082 083 084 085
Topic Identifier
045 27.59 090 0.64 135 59.19 170 0.00 Ad−Hoc Robust Monolingual Italian track − Comparison to Median Mean Average Precision by Topic (Topics 086 to 130)
1
046 11.18 091 0.10 136 56.47 171 48.17 reinaITttest
047 5.79 092 14.57 137 34.07 172 0.00 0.5
Difference
048 3.05 093 58.95 138 11.19 173 75.00 0
049 32.31 094 36.84 139 5.93 174 76.21 −0.5
060 24.18 095 1.59 140 35.45 175 0.00 −1
086 087 088 089 090 091 092 093 094 095 096 097 098 099 110 111 112 113 114 115 116 117 118 119 130
061 35.00 096 26.84 141 90.48 176 64.13 Topic Identifier
Ad−Hoc Robust Monolingual Italian track − Comparison to Median Mean Average Precision by Topic (Topics 131 to 165)
062 45.27 097 65.28 142 82.81 177 42.64 1
reinaITttest
063 59.53 098 69.31 143 17.90 178 27.91 0.5
Difference
064 0.00 099 38.06 144 0.00 179 64.30
0
065 11.85 110 54.70 145 100.00 190 36.16
−0.5
066 66.51 111 15.37 146 0.00 191 0.00
−1
067 22.75 112 39.09 147 1.90 192 24.93 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 160 161 162 163 164 165
Topic Identifier
068 53.21 113 10.57 148 54.00 193 58.60 Ad−Hoc Robust Monolingual Italian track − Comparison to Median Mean Average Precision by Topic (Topics 166 to 200)
1
069 4.00 114 12.71 149 47.88 194 0.21 reinaITttest
0.5
080 39.08 115 49.20 160 0.00 195 45.58
Difference
081 24.84 116 39.81 161 0.00 196 100.00 0
082 92.75 117 14.98 162 39.73 197 61.03 −0.5
083 79.17 118 3.79 163 16.47 198 39.55 −1
166 167 168 169 170 171 172 173 174 175 176 177 178 179 190 191 192 193 194 195 196 197 198 199 200
084 8.75 119 29.26 164 16.48 199 43.59 Topic Identifier
085 60.53 130 44.51 165 100.00 200 37.26
221
reina reinaITttest AH-ROBUST-MONO-IT-CLEF2006
Docs Cutoff Levels Precision at DCL (%)
Ad−Hoc Robust Monolingual Italian track − Retrieved documents vs Precision
100%
5 docs 35.60
reinaITttest
10 docs 31.90 90%
15 docs 29.73
80%
20 docs 27.70
30 docs 23.67 70%
100 docs 12.07
60%
200 docs 7.31
R−Precision
500 docs 3.35 50%
1000 docs 1.76 40%
R-Precision (precision after R document retrieved,
where R = Relevant retrieved) 30%
35.63
20%
10%
0%
5 10 15 20 30 100 200 500 1000
Retrieved Documents (logarithmic scale)
Exact R-Precision Ad−Hoc Robust Monolingual Italian track − Box plot of the Topics of the Experiment
Maximum 1.0000
Minimum 0.0000
First Quartile 0.0956
Second Quartile 0.3750
Third Quartile 0.5000
Interquartile range 0.4044
Mean 0.3563
Standard Deviation 0.2712
Lower Outlier Threshold 0.0000
Upper Outlier Threshold 1.0000 0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Exact R−Precision
Mean With No Outliers 0.3563
Std With No Outliers 0.2712
Ad−Hoc Robust Monolingual Italian track − Distribution of the Topics of the Experiment
25
Number of Topics of the Experiment
20
15
10
5
0
0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Exact R−Precision
Precision averages (%) for individual Ad−Hoc Robust Monolingual Italian track − Comparison to Median Mean Exact R−Precision by Topic (Topics 041 to 085)
queries 1
reinaITttest
0.5
041 71.43 086 29.03 131 31.71 166 0.00
Difference
0
042 48.19 087 60.00 132 66.67 167 100.00
−0.5
043 0.00 088 40.00 133 43.75 168 37.50
044 0.00 089 76.92 134 40.00 169 0.00 −1
041 042 043 044 045 046 047 048 049 060 061 062 063 064 065 066 067 068 069 080 081 082 083 084 085
Topic Identifier
045 33.33 090 0.00 135 53.25 170 0.00 Ad−Hoc Robust Monolingual Italian track − Comparison to Median Mean Exact R−Precision by Topic (Topics 086 to 130)
1
046 6.25 091 0.00 136 58.33 171 42.86 reinaITttest
047 11.54 092 22.22 137 25.00 172 0.00 0.5
Difference
048 8.00 093 66.67 138 15.38 173 50.00 0
049 31.03 094 45.00 139 0.00 174 62.50 −0.5
060 40.85 095 4.29 140 38.00 175 0.00 −1
086 087 088 089 090 091 092 093 094 095 096 097 098 099 110 111 112 113 114 115 116 117 118 119 130
061 50.00 096 37.50 141 85.71 176 62.50 Topic Identifier
Ad−Hoc Robust Monolingual Italian track − Comparison to Median Mean Exact R−Precision by Topic (Topics 131 to 165)
062 46.15 097 50.00 142 85.71 177 33.33 1
reinaITttest
063 50.00 098 58.33 143 19.23 178 25.00 0.5
Difference
064 0.00 099 38.46 144 0.00 179 57.50
0
065 13.33 110 56.52 145 100.00 190 41.67
−0.5
066 64.86 111 27.27 146 0.00 191 0.00
−1
067 35.29 112 41.67 147 0.00 192 29.03 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 160 161 162 163 164 165
Topic Identifier
068 50.00 113 11.11 148 50.00 193 50.00 Ad−Hoc Robust Monolingual Italian track − Comparison to Median Mean Exact R−Precision by Topic (Topics 166 to 200)
1
069 0.00 114 7.69 149 33.33 194 0.00 reinaITttest
0.5
080 42.86 115 51.85 160 0.00 195 50.00
Difference
081 33.33 116 30.00 161 0.00 196 100.00 0
082 81.82 117 17.65 162 50.00 197 63.64 −0.5
083 66.67 118 0.00 163 21.74 198 28.57 −1
166 167 168 169 170 171 172 173 174 175 176 177 178 179 190 191 192 193 194 195 196 197 198 199 200
084 0.00 119 36.67 164 25.81 199 50.00 Topic Identifier
085 56.16 130 45.00 165 100.00 200 40.00
222
reina reinaITtdtest AH-ROBUST-MONO-IT-CLEF2006
Overall statistics for 100 queries : Priority 1
Total number of documents over all queries Query Construction AUTOMATIC
Retrieved 100,000 Source Language Italian
Relevant 1,930 Topic Fields title, description
Relevant retrieved 1,826 Pooled false
Geometric Mean Average Precision 0.1055 local expansion with term windows, title+desc
Binary Preference (BPREF) 0.4131 fields
Interploated Recall (%) Precision Averages (%)
Ad−Hoc Robust Monolingual Italian track − Interpolated Recall vs Average Precision
100%
0 69.53
reinaITtdtest
10 62.80 90%
20 56.99
80%
30 47.85
40 42.59 70%
50 39.00
Average Precision
60%
60 33.51
70 29.55 50%
80 25.17 40%
90 21.01
30%
100 16.73
Average precision (non-interpolated) for all 20%
relevant documents (averaged over queries)
38.45 10%
0%
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
Interpolated Recall
Mean Average Precision Ad−Hoc Robust Monolingual Italian track − Box plot of the Topics of the Experiment
Maximum 1.0000
Minimum 0.0000
First Quartile 0.1739
Second Quartile 0.3616
Third Quartile 0.5708
Interquartile range 0.3969
Mean 0.3845
Standard Deviation 0.2718
Lower Outlier Threshold 0.0000
Upper Outlier Threshold 1.0000 0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Mean Average Precision
Mean With No Outliers 0.3845
Std With No Outliers 0.2718
Ad−Hoc Robust Monolingual Italian track − Distribution of the Topics of the Experiment
15
Number of Topics of the Experiment
10
5
0
0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Mean Average Precision
Precision averages (%) for individual Ad−Hoc Robust Monolingual Italian track − Comparison to Median Mean Average Precision by Topic (Topics 041 to 085)
queries 1
reinaITtdtest
0.5
041 70.52 086 19.30 131 29.83 166 0.24
Difference
0
042 49.02 087 51.40 132 45.13 167 100.00
−0.5
043 0.00 088 45.57 133 35.29 168 32.47
044 41.67 089 74.39 134 67.68 169 0.00 −1
041 042 043 044 045 046 047 048 049 060 061 062 063 064 065 066 067 068 069 080 081 082 083 084 085
Topic Identifier
045 24.45 090 3.81 135 55.65 170 0.00 Ad−Hoc Robust Monolingual Italian track − Comparison to Median Mean Average Precision by Topic (Topics 086 to 130)
1
046 13.91 091 36.23 136 53.81 171 39.83 reinaITtdtest
047 6.03 092 29.05 137 35.71 172 0.00 0.5
Difference
048 11.66 093 75.02 138 28.57 173 70.00 0
049 27.27 094 42.78 139 5.57 174 79.91 −0.5
060 18.67 095 8.02 140 32.23 175 0.00 −1
086 087 088 089 090 091 092 093 094 095 096 097 098 099 110 111 112 113 114 115 116 117 118 119 130
061 34.45 096 21.02 141 75.60 176 62.40 Topic Identifier
Ad−Hoc Robust Monolingual Italian track − Comparison to Median Mean Average Precision by Topic (Topics 131 to 165)
062 43.00 097 63.26 142 97.22 177 49.61 1
reinaITtdtest
063 67.66 098 81.89 143 26.53 178 7.91 0.5
Difference
064 0.00 099 23.50 144 0.00 179 71.67
0
065 23.33 110 75.15 145 100.00 190 34.25
−0.5
066 68.40 111 16.21 146 0.00 191 0.00
−1
067 36.07 112 46.08 147 11.03 192 46.59 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 160 161 162 163 164 165
Topic Identifier
068 20.24 113 16.85 148 30.56 193 51.45 Ad−Hoc Robust Monolingual Italian track − Comparison to Median Mean Average Precision by Topic (Topics 166 to 200)
1
069 4.31 114 38.08 149 47.78 194 36.87 reinaITtdtest
0.5
080 37.66 115 39.17 160 0.00 195 52.96
Difference
081 37.60 116 26.16 161 0.00 196 100.00 0
082 58.70 117 40.38 162 60.43 197 71.35 −0.5
083 65.56 118 3.47 163 54.69 198 29.40 −1
166 167 168 169 170 171 172 173 174 175 176 177 178 179 190 191 192 193 194 195 196 197 198 199 200
084 10.52 119 67.76 164 17.94 199 52.12 Topic Identifier
085 58.52 130 36.09 165 100.00 200 34.44
223
reina reinaITtdtest AH-ROBUST-MONO-IT-CLEF2006
Docs Cutoff Levels Precision at DCL (%)
Ad−Hoc Robust Monolingual Italian track − Retrieved documents vs Precision
100%
5 docs 41.80
reinaITtdtest
10 docs 35.80 90%
15 docs 32.73
80%
20 docs 29.80
30 docs 25.17 70%
100 docs 12.96
60%
200 docs 7.74
R−Precision
500 docs 3.48 50%
1000 docs 1.83 40%
R-Precision (precision after R document retrieved,
where R = Relevant retrieved) 30%
37.15
20%
10%
0%
5 10 15 20 30 100 200 500 1000
Retrieved Documents (logarithmic scale)
Exact R-Precision Ad−Hoc Robust Monolingual Italian track − Box plot of the Topics of the Experiment
Maximum 1.0000
Minimum 0.0000
First Quartile 0.2283
Second Quartile 0.3846
Third Quartile 0.5000
Interquartile range 0.2717
Mean 0.3715
Standard Deviation 0.2559
Lower Outlier Threshold 0.0000
Upper Outlier Threshold 0.8077 0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Exact R−Precision
Mean With No Outliers 0.3392
Std With No Outliers 0.2186
Ad−Hoc Robust Monolingual Italian track − Distribution of the Topics of the Experiment
20
Number of Topics of the Experiment
15
10
5
0
0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Exact R−Precision
Precision averages (%) for individual Ad−Hoc Robust Monolingual Italian track − Comparison to Median Mean Exact R−Precision by Topic (Topics 041 to 085)
queries 1
reinaITtdtest
0.5
041 71.43 086 25.81 131 46.34 166 0.00
Difference
0
042 53.01 087 40.00 132 33.33 167 100.00
−0.5
043 0.00 088 33.33 133 43.75 168 40.62
044 0.00 089 80.77 134 40.00 169 0.00 −1
041 042 043 044 045 046 047 048 049 060 061 062 063 064 065 066 067 068 069 080 081 082 083 084 085
Topic Identifier
045 26.19 090 0.00 135 50.65 170 0.00 Ad−Hoc Robust Monolingual Italian track − Comparison to Median Mean Exact R−Precision by Topic (Topics 086 to 130)
1
046 18.75 091 25.00 136 41.67 171 42.86 reinaITtdtest
047 11.54 092 25.00 137 25.00 172 0.00 0.5
Difference
048 12.00 093 77.78 138 23.08 173 50.00 0
049 31.03 094 50.00 139 0.00 174 62.50 −0.5
060 25.35 095 15.71 140 36.00 175 0.00 −1
086 087 088 089 090 091 092 093 094 095 096 097 098 099 110 111 112 113 114 115 116 117 118 119 130
061 43.75 096 37.50 141 57.14 176 62.50 Topic Identifier
Ad−Hoc Robust Monolingual Italian track − Comparison to Median Mean Exact R−Precision by Topic (Topics 131 to 165)
062 38.46 097 50.00 142 92.86 177 50.00 1
reinaITtdtest
063 50.00 098 58.33 143 26.92 178 0.00 0.5
Difference
064 0.00 099 28.21 144 0.00 179 60.00
0
065 20.00 110 69.57 145 100.00 190 41.67
−0.5
066 70.27 111 27.27 146 0.00 191 0.00
−1
067 35.29 112 41.67 147 20.00 192 48.39 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 160 161 162 163 164 165
Topic Identifier
068 33.33 113 27.78 148 50.00 193 50.00 Ad−Hoc Robust Monolingual Italian track − Comparison to Median Mean Exact R−Precision by Topic (Topics 166 to 200)
1
069 0.00 114 38.46 149 33.33 194 23.08 reinaITtdtest
0.5
080 41.67 115 40.74 160 0.00 195 41.67
Difference
081 33.33 116 40.00 161 0.00 196 100.00 0
082 54.55 117 41.18 162 50.00 197 65.45 −0.5
083 66.67 118 0.00 163 69.57 198 28.57 −1
166 167 168 169 170 171 172 173 174 175 176 177 178 179 190 191 192 193 194 195 196 197 198 199 200
084 25.00 119 65.00 164 22.58 199 52.78 Topic Identifier
085 57.53 130 40.00 165 100.00 200 34.29
224
colesir CoLesIRnlTst AH-ROBUST-MONO-NL-CLEF2006
Overall statistics for 100 queries : Priority 1
Total number of documents over all queries Query Construction AUTOMATIC
Retrieved 100,000 Source Language Dutch; Flemish
Relevant 2,717 Topic Fields title, description
Relevant retrieved 2,374 Pooled false
Geometric Mean Average Precision 0.1640 robust monolingual NL - TEST; automatic; TD
Binary Preference (BPREF) 0.4017
Interploated Recall (%) Precision Averages (%)
Ad−Hoc Robust Monolingual Dutch track − Interpolated Recall vs Average Precision
100%
0 75.33
CoLesIRnlTst
10 64.76 90%
20 58.71
80%
30 51.64
40 46.38 70%
50 42.53
Average Precision
60%
60 35.79
70 32.01 50%
80 28.64 40%
90 22.59
30%
100 16.29
Average precision (non-interpolated) for all 20%
relevant documents (averaged over queries)
41.60 10%
0%
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
Interpolated Recall
Mean Average Precision Ad−Hoc Robust Monolingual Dutch track − Box plot of the Topics of the Experiment
Maximum 1.0000
Minimum 0.0000
First Quartile 0.1273
Second Quartile 0.3545
Third Quartile 0.6605
Interquartile range 0.5332
Mean 0.4160
Standard Deviation 0.3183
Lower Outlier Threshold 0.0000
Upper Outlier Threshold 1.0000 0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Mean Average Precision
Mean With No Outliers 0.4160
Std With No Outliers 0.3183
Ad−Hoc Robust Monolingual Dutch track − Distribution of the Topics of the Experiment
15
Number of Topics of the Experiment
10
5
0
0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Mean Average Precision
Precision averages (%) for individual Ad−Hoc Robust Monolingual Dutch track − Comparison to Median Mean Average Precision by Topic (Topics 041 to 085)
queries 1
CoLesIRnlTst
0.5
041 21.76 086 48.92 131 36.59 166 0.00
Difference
0
042 15.16 087 64.83 132 56.95 167 75.00
−0.5
043 7.63 088 27.98 133 8.67 168 6.55
044 27.47 089 84.95 134 100.00 169 0.04 −1
041 042 043 044 045 046 047 048 049 060 061 062 063 064 065 066 067 068 069 080 081 082 083 084 085
Topic Identifier
045 39.43 090 48.78 135 66.02 170 55.10 Ad−Hoc Robust Monolingual Dutch track − Comparison to Median Mean Average Precision by Topic (Topics 086 to 130)
1
046 16.49 091 16.87 136 96.67 171 36.83 CoLesIRnlTst
047 26.71 092 34.03 137 0.02 172 12.98 0.5
Difference
048 21.28 093 72.64 138 0.28 173 100.00 0
049 6.79 094 78.56 139 20.86 174 96.20 −0.5
060 11.38 095 53.54 140 80.53 175 0.42 −1
086 087 088 089 090 091 092 093 094 095 096 097 098 099 110 111 112 113 114 115 116 117 118 119 130
061 61.82 096 42.28 141 49.32 176 96.00 Topic Identifier
Ad−Hoc Robust Monolingual Dutch track − Comparison to Median Mean Average Precision by Topic (Topics 131 to 165)
062 53.91 097 56.12 142 98.62 177 22.06 1
CoLesIRnlTst
063 85.76 098 84.86 143 79.66 178 51.33 0.5
Difference
064 6.41 099 29.97 144 100.00 179 26.35
0
065 4.86 110 100.00 145 11.88 190 33.43
−0.5
066 60.97 111 8.31 146 30.95 191 0.00
−1
067 9.68 112 43.35 147 16.38 192 30.11 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 160 161 162 163 164 165
Topic Identifier
068 41.02 113 0.57 148 57.83 193 53.62 Ad−Hoc Robust Monolingual Dutch track − Comparison to Median Mean Average Precision by Topic (Topics 166 to 200)
1
069 1.70 114 34.30 149 100.00 194 0.00 CoLesIRnlTst
0.5
080 68.55 115 15.42 160 0.00 195 0.52
Difference
081 63.50 116 83.68 161 100.00 196 61.10 0
082 29.42 117 12.49 162 86.58 197 66.38 −0.5
083 6.14 118 21.26 163 20.79 198 21.11 −1
166 167 168 169 170 171 172 173 174 175 176 177 178 179 190 191 192 193 194 195 196 197 198 199 200
084 16.54 119 64.65 164 11.16 199 69.99 Topic Identifier
085 66.09 130 37.04 165 70.00 200 8.92
225
colesir CoLesIRnlTst AH-ROBUST-MONO-NL-CLEF2006
Docs Cutoff Levels Precision at DCL (%)
Ad−Hoc Robust Monolingual Dutch track − Retrieved documents vs Precision
100%
5 docs 53.60
CoLesIRnlTst
10 docs 43.50 90%
15 docs 38.80
80%
20 docs 34.90
30 docs 30.13 70%
100 docs 15.41
60%
200 docs 9.39
R−Precision
500 docs 4.42 50%
1000 docs 2.37 40%
R-Precision (precision after R document retrieved,
where R = Relevant retrieved) 30%
39.85
20%
10%
0%
5 10 15 20 30 100 200 500 1000
Retrieved Documents (logarithmic scale)
Exact R-Precision Ad−Hoc Robust Monolingual Dutch track − Box plot of the Topics of the Experiment
Maximum 1.0000
Minimum 0.0000
First Quartile 0.1429
Second Quartile 0.3613
Third Quartile 0.5939
Interquartile range 0.4511
Mean 0.3985
Standard Deviation 0.2981
Lower Outlier Threshold 0.0000
Upper Outlier Threshold 1.0000 0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Exact R−Precision
Mean With No Outliers 0.3985
Std With No Outliers 0.2981
Ad−Hoc Robust Monolingual Dutch track − Distribution of the Topics of the Experiment
20
Number of Topics of the Experiment
15
10
5
0
0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Exact R−Precision
Precision averages (%) for individual Ad−Hoc Robust Monolingual Dutch track − Comparison to Median Mean Exact R−Precision by Topic (Topics 041 to 085)
queries 1
CoLesIRnlTst
0.5
041 17.39 086 46.00 131 35.00 166 0.00
Difference
0
042 12.50 087 46.15 132 46.15 167 50.00
−0.5
043 25.00 088 0.00 133 10.71 168 12.12
044 33.33 089 76.74 134 100.00 169 0.00 −1
041 042 043 044 045 046 047 048 049 060 061 062 063 064 065 066 067 068 069 080 081 082 083 084 085
Topic Identifier
045 37.50 090 53.85 135 64.52 170 50.00 Ad−Hoc Robust Monolingual Dutch track − Comparison to Median Mean Exact R−Precision by Topic (Topics 086 to 130)
1
046 29.17 091 11.11 136 80.00 171 27.78 CoLesIRnlTst
047 32.26 092 39.47 137 0.00 172 14.29 0.5
Difference
048 25.45 093 75.00 138 0.00 173 100.00 0
049 14.29 094 78.95 139 23.91 174 91.30 −0.5
060 7.69 095 52.49 140 71.53 175 0.00 −1
086 087 088 089 090 091 092 093 094 095 096 097 098 099 110 111 112 113 114 115 116 117 118 119 130
061 56.52 096 42.11 141 50.00 176 85.00 Topic Identifier
Ad−Hoc Robust Monolingual Dutch track − Comparison to Median Mean Exact R−Precision by Topic (Topics 131 to 165)
062 50.00 097 50.00 142 95.45 177 33.33 1
CoLesIRnlTst
063 80.00 098 66.67 143 79.31 178 42.86 0.5
Difference
064 12.50 099 25.00 144 100.00 179 29.03
0
065 8.70 110 100.00 145 21.43 190 48.39
−0.5
066 60.71 111 14.29 146 0.00 191 0.00
−1
067 15.00 112 40.00 147 16.67 192 40.00 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 160 161 162 163 164 165
Topic Identifier
068 30.43 113 3.03 148 59.52 193 57.89 Ad−Hoc Robust Monolingual Dutch track − Comparison to Median Mean Exact R−Precision by Topic (Topics 166 to 200)
1
069 0.00 114 37.25 149 100.00 194 0.00 CoLesIRnlTst
0.5
080 69.89 115 23.08 160 0.00 195 0.00
Difference
081 59.26 116 82.76 161 100.00 196 55.56 0
082 30.00 117 22.22 162 72.73 197 57.14 −0.5
083 0.00 118 18.18 163 28.57 198 0.00 −1
166 167 168 169 170 171 172 173 174 175 176 177 178 179 190 191 192 193 194 195 196 197 198 199 200
084 30.00 119 58.06 164 18.00 199 68.75 Topic Identifier
085 61.11 130 33.33 165 50.00 200 3.57
226
daedalus nlFSnlR4S AH-ROBUST-MONO-NL-CLEF2006
Overall statistics for 100 queries : Priority 1
Total number of documents over all queries Query Construction AUTOMATIC
Retrieved 100,000 Source Language Dutch; Flemish
Relevant 2,717 Topic Fields title, description, narrative
Relevant retrieved 2,429 Pooled false
Geometric Mean Average Precision 0.2073 TDN run
Binary Preference (BPREF) 0.4259
Interploated Recall (%) Precision Averages (%)
Ad−Hoc Robust Monolingual Dutch track − Interpolated Recall vs Average Precision
100%
0 75.04
nlFSnlR4S
10 69.21 90%
20 63.33
80%
30 57.04
40 51.52 70%
50 48.21
Average Precision
60%
60 40.78
70 36.45 50%
80 30.71 40%
90 23.43
30%
100 15.10
Average precision (non-interpolated) for all 20%
relevant documents (averaged over queries)
45.02 10%
0%
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
Interpolated Recall
Mean Average Precision Ad−Hoc Robust Monolingual Dutch track − Box plot of the Topics of the Experiment
Maximum 1.0000
Minimum 0.0000
First Quartile 0.1742
Second Quartile 0.4429
Third Quartile 0.6904
Interquartile range 0.5162
Mean 0.4502
Standard Deviation 0.3047
Lower Outlier Threshold 0.0000
Upper Outlier Threshold 1.0000 0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Mean Average Precision
Mean With No Outliers 0.4502
Std With No Outliers 0.3047
Ad−Hoc Robust Monolingual Dutch track − Distribution of the Topics of the Experiment
10
Number of Topics of the Experiment
8
6
4
2
0
0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Mean Average Precision
Precision averages (%) for individual Ad−Hoc Robust Monolingual Dutch track − Comparison to Median Mean Average Precision by Topic (Topics 041 to 085)
queries 1
nlFSnlR4S
0.5
041 18.26 086 53.84 131 21.53 166 0.00
Difference
0
042 16.95 087 72.62 132 61.35 167 36.67
−0.5
043 45.95 088 20.96 133 20.97 168 45.71
044 28.70 089 91.05 134 100.00 169 2.45 −1
041 042 043 044 045 046 047 048 049 060 061 062 063 064 065 066 067 068 069 080 081 082 083 084 085
Topic Identifier
045 39.79 090 23.96 135 74.47 170 80.46 Ad−Hoc Robust Monolingual Dutch track − Comparison to Median Mean Average Precision by Topic (Topics 086 to 130)
1
046 13.96 091 5.23 136 96.67 171 11.50 nlFSnlR4S
047 32.54 092 43.41 137 3.83 172 20.65 0.5
Difference
048 17.37 093 77.07 138 41.25 173 58.33 0
049 2.85 094 76.83 139 7.85 174 94.79 −0.5
060 8.27 095 67.12 140 75.06 175 0.23 −1
086 087 088 089 090 091 092 093 094 095 096 097 098 099 110 111 112 113 114 115 116 117 118 119 130
061 37.78 096 45.17 141 60.71 176 91.20 Topic Identifier
Ad−Hoc Robust Monolingual Dutch track − Comparison to Median Mean Average Precision by Topic (Topics 131 to 165)
062 59.15 097 69.10 142 72.32 177 50.00 1
nlFSnlR4S
063 66.53 098 88.19 143 73.89 178 56.29 0.5
Difference
064 31.64 099 25.95 144 100.00 179 48.39
0
065 14.53 110 100.00 145 36.16 190 55.86
−0.5
066 68.96 111 0.12 146 29.17 191 0.00
−1
067 17.48 112 58.82 147 25.23 192 36.61 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 160 161 162 163 164 165
Topic Identifier
068 54.23 113 0.64 148 65.26 193 64.77 Ad−Hoc Robust Monolingual Dutch track − Comparison to Median Mean Average Precision by Topic (Topics 166 to 200)
1
069 8.49 114 42.57 149 88.19 194 0.00 nlFSnlR4S
0.5
080 84.19 115 41.01 160 0.00 195 7.69
Difference
081 66.72 116 68.98 161 100.00 196 84.01 0
082 22.78 117 10.92 162 84.90 197 90.98 −0.5
083 8.34 118 15.09 163 13.76 198 36.67 −1
166 167 168 169 170 171 172 173 174 175 176 177 178 179 190 191 192 193 194 195 196 197 198 199 200
084 51.23 119 63.75 164 6.84 199 84.55 Topic Identifier
085 57.95 130 72.01 165 54.35 200 21.29
227
daedalus nlFSnlR4S AH-ROBUST-MONO-NL-CLEF2006
Docs Cutoff Levels Precision at DCL (%)
Ad−Hoc Robust Monolingual Dutch track − Retrieved documents vs Precision
100%
5 docs 55.00
nlFSnlR4S
10 docs 47.30 90%
15 docs 41.33
80%
20 docs 37.55
30 docs 32.10 70%
100 docs 16.60
60%
200 docs 10.03
R−Precision
500 docs 4.60 50%
1000 docs 2.43 40%
R-Precision (precision after R document retrieved,
where R = Relevant retrieved) 30%
42.89
20%
10%
0%
5 10 15 20 30 100 200 500 1000
Retrieved Documents (logarithmic scale)
Exact R-Precision Ad−Hoc Robust Monolingual Dutch track − Box plot of the Topics of the Experiment
Maximum 1.0000
Minimum 0.0000
First Quartile 0.1978
Second Quartile 0.4615
Third Quartile 0.6667
Interquartile range 0.4688
Mean 0.4289
Standard Deviation 0.2845
Lower Outlier Threshold 0.0000
Upper Outlier Threshold 1.0000 0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Exact R−Precision
Mean With No Outliers 0.4289
Std With No Outliers 0.2845
Ad−Hoc Robust Monolingual Dutch track − Distribution of the Topics of the Experiment
20
Number of Topics of the Experiment
15
10
5
0
0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Exact R−Precision
Precision averages (%) for individual Ad−Hoc Robust Monolingual Dutch track − Comparison to Median Mean Exact R−Precision by Topic (Topics 041 to 085)
queries 1
nlFSnlR4S
0.5
041 26.09 086 56.00 131 25.00 166 0.00
Difference
0
042 16.67 087 46.15 132 61.54 167 0.00
−0.5
043 25.00 088 50.00 133 17.86 168 51.52
044 26.67 089 86.05 134 100.00 169 0.00 −1
041 042 043 044 045 046 047 048 049 060 061 062 063 064 065 066 067 068 069 080 081 082 083 084 085
Topic Identifier
045 35.00 090 38.46 135 66.94 170 71.43 Ad−Hoc Robust Monolingual Dutch track − Comparison to Median Mean Exact R−Precision by Topic (Topics 086 to 130)
1
046 25.00 091 11.11 136 80.00 171 16.67 nlFSnlR4S
047 35.48 092 55.26 137 0.00 172 28.57 0.5
Difference
048 16.36 093 80.00 138 33.33 173 50.00 0
049 0.00 094 78.95 139 19.57 174 91.30 −0.5
060 11.54 095 61.46 140 67.36 175 0.00 −1
086 087 088 089 090 091 092 093 094 095 096 097 098 099 110 111 112 113 114 115 116 117 118 119 130
061 39.13 096 42.11 141 50.00 176 80.00 Topic Identifier
Ad−Hoc Robust Monolingual Dutch track − Comparison to Median Mean Exact R−Precision by Topic (Topics 131 to 165)
062 50.00 097 75.00 142 72.73 177 50.00 1
nlFSnlR4S
063 53.33 098 66.67 143 68.97 178 42.86 0.5
Difference
064 37.50 099 33.33 144 100.00 179 46.77
0
065 30.43 110 100.00 145 28.57 190 61.29
−0.5
066 71.43 111 0.00 146 0.00 191 0.00
−1
067 25.00 112 60.00 147 33.33 192 53.33 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 160 161 162 163 164 165
Topic Identifier
068 47.83 113 3.03 148 61.90 193 57.89 Ad−Hoc Robust Monolingual Dutch track − Comparison to Median Mean Exact R−Precision by Topic (Topics 166 to 200)
1
069 0.00 114 43.14 149 66.67 194 0.00 nlFSnlR4S
0.5
080 81.72 115 46.15 160 0.00 195 0.00
Difference
081 62.96 116 72.41 161 100.00 196 66.67 0
082 20.00 117 22.22 162 72.73 197 80.67 −0.5
083 0.00 118 22.73 163 14.29 198 0.00 −1
166 167 168 169 170 171 172 173 174 175 176 177 178 179 190 191 192 193 194 195 196 197 198 199 200
084 50.00 119 58.06 164 18.00 199 78.12 Topic Identifier
085 66.67 130 55.56 165 50.00 200 35.71
228
daedalus nlynlRFS3456 AH-ROBUST-MONO-NL-CLEF2006
Overall statistics for 100 queries : Priority 2
Total number of documents over all queries Query Construction AUTOMATIC
Retrieved 100,000 Source Language Dutch; Flemish
Relevant 2,717 Topic Fields title, description, narrative
Relevant retrieved 2,425 Pooled false
Geometric Mean Average Precision 0.2068 Average run
Binary Preference (BPREF) 0.4235
Interploated Recall (%) Precision Averages (%)
Ad−Hoc Robust Monolingual Dutch track − Interpolated Recall vs Average Precision
100%
0 75.15
nlynlRFS3456
10 69.41 90%
20 62.65
80%
30 56.93
40 51.52 70%
50 47.93
Average Precision
60%
60 40.75
70 36.35 50%
80 30.95 40%
90 23.69
30%
100 15.44
Average precision (non-interpolated) for all 20%
relevant documents (averaged over queries)
44.99 10%
0%
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
Interpolated Recall
Mean Average Precision Ad−Hoc Robust Monolingual Dutch track − Box plot of the Topics of the Experiment
Maximum 1.0000
Minimum 0.0000
First Quartile 0.1749
Second Quartile 0.4476
Third Quartile 0.6956
Interquartile range 0.5207
Mean 0.4499
Standard Deviation 0.3057
Lower Outlier Threshold 0.0000
Upper Outlier Threshold 1.0000 0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Mean Average Precision
Mean With No Outliers 0.4499
Std With No Outliers 0.3057
Ad−Hoc Robust Monolingual Dutch track − Distribution of the Topics of the Experiment
10
Number of Topics of the Experiment
8
6
4
2
0
0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Mean Average Precision
Precision averages (%) for individual Ad−Hoc Robust Monolingual Dutch track − Comparison to Median Mean Average Precision by Topic (Topics 041 to 085)
queries 1
nlynlRFS3456
0.5
041 18.15 086 54.18 131 21.42 166 0.00
Difference
0
042 17.01 087 72.64 132 61.05 167 36.67
−0.5
043 46.10 088 20.83 133 20.82 168 45.73
044 28.35 089 91.14 134 100.00 169 2.37 −1
041 042 043 044 045 046 047 048 049 060 061 062 063 064 065 066 067 068 069 080 081 082 083 084 085
Topic Identifier
045 39.88 090 21.43 135 73.96 170 80.30 Ad−Hoc Robust Monolingual Dutch track − Comparison to Median Mean Average Precision by Topic (Topics 086 to 130)
1
046 13.87 091 6.97 136 96.67 171 11.68 nlynlRFS3456
047 32.34 092 45.21 137 2.71 172 21.05 0.5
Difference
048 17.97 093 78.42 138 40.20 173 58.33 0
049 2.90 094 77.15 139 7.85 174 94.52 −0.5
060 8.43 095 67.67 140 75.01 175 0.27 −1
086 087 088 089 090 091 092 093 094 095 096 097 098 099 110 111 112 113 114 115 116 117 118 119 130
061 37.13 096 44.32 141 60.71 176 91.22 Topic Identifier
Ad−Hoc Robust Monolingual Dutch track − Comparison to Median Mean Average Precision by Topic (Topics 131 to 165)
062 59.27 097 69.38 142 72.37 177 50.00 1
nlynlRFS3456
063 66.03 098 88.19 143 73.99 178 52.67 0.5
Difference
064 25.28 099 25.66 144 100.00 179 47.72
0
065 14.57 110 100.00 145 36.54 190 54.47
−0.5
066 69.20 111 0.12 146 33.33 191 0.00
−1
067 15.35 112 55.87 147 22.50 192 37.35 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 160 161 162 163 164 165
Topic Identifier
068 54.39 113 0.64 148 64.87 193 64.98 Ad−Hoc Robust Monolingual Dutch track − Comparison to Median Mean Average Precision by Topic (Topics 166 to 200)
1
069 8.64 114 41.78 149 88.19 194 0.00 nlynlRFS3456
0.5
080 83.98 115 42.26 160 0.00 195 7.69
Difference
081 66.70 116 69.73 161 100.00 196 86.23 0
082 21.49 117 10.84 162 85.21 197 90.87 −0.5
083 8.12 118 15.70 163 14.26 198 41.67 −1
166 167 168 169 170 171 172 173 174 175 176 177 178 179 190 191 192 193 194 195 196 197 198 199 200
084 51.64 119 64.55 164 6.99 199 84.20 Topic Identifier
085 57.81 130 71.81 165 59.09 200 20.18
229
daedalus nlynlRFS3456 AH-ROBUST-MONO-NL-CLEF2006
Docs Cutoff Levels Precision at DCL (%)
Ad−Hoc Robust Monolingual Dutch track − Retrieved documents vs Precision
100%
5 docs 54.60
nlynlRFS3456
10 docs 46.90 90%
15 docs 41.27
80%
20 docs 37.55
30 docs 32.17 70%
100 docs 16.53
60%
200 docs 10.03
R−Precision
500 docs 4.59 50%
1000 docs 2.43 40%
R-Precision (precision after R document retrieved,
where R = Relevant retrieved) 30%
42.85
20%
10%
0%
5 10 15 20 30 100 200 500 1000
Retrieved Documents (logarithmic scale)
Exact R-Precision Ad−Hoc Robust Monolingual Dutch track − Box plot of the Topics of the Experiment
Maximum 1.0000
Minimum 0.0000
First Quartile 0.2113
Second Quartile 0.4730
Third Quartile 0.6667
Interquartile range 0.4554
Mean 0.4285
Standard Deviation 0.2841
Lower Outlier Threshold 0.0000
Upper Outlier Threshold 1.0000 0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Exact R−Precision
Mean With No Outliers 0.4285
Std With No Outliers 0.2841
Ad−Hoc Robust Monolingual Dutch track − Distribution of the Topics of the Experiment
20
Number of Topics of the Experiment
15
10
5
0
0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Exact R−Precision
Precision averages (%) for individual Ad−Hoc Robust Monolingual Dutch track − Comparison to Median Mean Exact R−Precision by Topic (Topics 041 to 085)
queries 1
nlynlRFS3456
0.5
041 26.09 086 56.00 131 22.50 166 0.00
Difference
0
042 20.83 087 53.85 132 61.54 167 0.00
−0.5
043 25.00 088 50.00 133 17.86 168 51.52
044 26.67 089 86.05 134 100.00 169 0.00 −1
041 042 043 044 045 046 047 048 049 060 061 062 063 064 065 066 067 068 069 080 081 082 083 084 085
Topic Identifier
045 35.00 090 38.46 135 66.94 170 71.43 Ad−Hoc Robust Monolingual Dutch track − Comparison to Median Mean Exact R−Precision by Topic (Topics 086 to 130)
1
046 25.00 091 11.11 136 80.00 171 16.67 nlynlRFS3456
047 35.48 092 52.63 137 0.00 172 28.57 0.5
Difference
048 16.36 093 80.00 138 33.33 173 50.00 0
049 0.00 094 78.95 139 19.57 174 91.30 −0.5
060 11.54 095 62.13 140 67.36 175 0.00 −1
086 087 088 089 090 091 092 093 094 095 096 097 098 099 110 111 112 113 114 115 116 117 118 119 130
061 39.13 096 42.11 141 50.00 176 80.00 Topic Identifier
Ad−Hoc Robust Monolingual Dutch track − Comparison to Median Mean Exact R−Precision by Topic (Topics 131 to 165)
062 50.00 097 75.00 142 72.73 177 50.00 1
nlynlRFS3456
063 53.33 098 66.67 143 68.97 178 42.86 0.5
Difference
064 25.00 099 33.33 144 100.00 179 46.77
0
065 30.43 110 100.00 145 28.57 190 61.29
−0.5
066 71.43 111 0.00 146 0.00 191 0.00
−1
067 25.00 112 60.00 147 33.33 192 53.33 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 160 161 162 163 164 165
Topic Identifier
068 47.83 113 3.03 148 61.90 193 57.89 Ad−Hoc Robust Monolingual Dutch track − Comparison to Median Mean Exact R−Precision by Topic (Topics 166 to 200)
1
069 0.00 114 43.14 149 66.67 194 0.00 nlynlRFS3456
0.5
080 81.72 115 50.00 160 0.00 195 0.00
Difference
081 62.96 116 72.41 161 100.00 196 66.67 0
082 20.00 117 22.22 162 69.70 197 80.67 −0.5
083 0.00 118 22.73 163 21.43 198 0.00 −1
166 167 168 169 170 171 172 173 174 175 176 177 178 179 190 191 192 193 194 195 196 197 198 199 200
084 50.00 119 54.84 164 18.00 199 78.12 Topic Identifier
085 66.67 130 55.56 165 50.00 200 32.14
230
daedalus nlx011nlRFW4FS4 AH-ROBUST-MONO-NL-CLEF2006
Overall statistics for 100 queries : Priority 3
Total number of documents over all queries Query Construction AUTOMATIC
Retrieved 100,000 Source Language Dutch; Flemish
Relevant 2,717 Topic Fields title, description, narrative
Relevant retrieved 2,429 Pooled false
Geometric Mean Average Precision 0.2067 Combined run
Binary Preference (BPREF) 0.4255
Interploated Recall (%) Precision Averages (%)
Ad−Hoc Robust Monolingual Dutch track − Interpolated Recall vs Average Precision
100%
0 74.54
nlx011nlRFW4FS4
10 68.95 90%
20 63.11
80%
30 56.93
40 51.50 70%
50 48.20
Average Precision
60%
60 40.77
70 36.45 50%
80 30.71 40%
90 23.43
30%
100 15.10
Average precision (non-interpolated) for all 20%
relevant documents (averaged over queries)
44.88 10%
0%
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
Interpolated Recall
Mean Average Precision Ad−Hoc Robust Monolingual Dutch track − Box plot of the Topics of the Experiment
Maximum 1.0000
Minimum 0.0000
First Quartile 0.1716
Second Quartile 0.4429
Third Quartile 0.6904
Interquartile range 0.5188
Mean 0.4488
Standard Deviation 0.3045
Lower Outlier Threshold 0.0000
Upper Outlier Threshold 1.0000 0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Mean Average Precision
Mean With No Outliers 0.4488
Std With No Outliers 0.3045
Ad−Hoc Robust Monolingual Dutch track − Distribution of the Topics of the Experiment
10
Number of Topics of the Experiment
8
6
4
2
0
0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Mean Average Precision
Precision averages (%) for individual Ad−Hoc Robust Monolingual Dutch track − Comparison to Median Mean Average Precision by Topic (Topics 041 to 085)
queries 1
nlx011nlRFW4FS4
0.5
041 18.26 086 53.84 131 21.53 166 0.00
Difference
0
042 16.95 087 72.62 132 61.35 167 36.67
−0.5
043 45.95 088 20.96 133 20.97 168 45.71
044 28.70 089 91.05 134 100.00 169 2.45 −1
041 042 043 044 045 046 047 048 049 060 061 062 063 064 065 066 067 068 069 080 081 082 083 084 085
Topic Identifier
045 39.79 090 23.96 135 74.47 170 80.46 Ad−Hoc Robust Monolingual Dutch track − Comparison to Median Mean Average Precision by Topic (Topics 086 to 130)
1
046 13.96 091 5.23 136 96.67 171 11.50 nlx011nlRFW4FS4
047 32.54 092 43.41 137 3.83 172 20.65 0.5
Difference
048 16.46 093 77.07 138 41.25 173 58.33 0
049 2.85 094 76.83 139 7.85 174 94.79 −0.5
060 8.27 095 67.12 140 75.06 175 0.23 −1
086 087 088 089 090 091 092 093 094 095 096 097 098 099 110 111 112 113 114 115 116 117 118 119 130
061 37.78 096 45.17 141 60.71 176 91.20 Topic Identifier
Ad−Hoc Robust Monolingual Dutch track − Comparison to Median Mean Average Precision by Topic (Topics 131 to 165)
062 59.15 097 69.10 142 72.32 177 50.00 1
nlx011nlRFW4FS4
063 66.53 098 88.19 143 73.89 178 56.29 0.5
Difference
064 31.64 099 25.95 144 100.00 179 48.39
0
065 14.53 110 100.00 145 36.16 190 55.86
−0.5
066 68.96 111 0.12 146 29.17 191 0.00
−1
067 16.32 112 58.82 147 25.23 192 36.61 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 160 161 162 163 164 165
Topic Identifier
068 45.79 113 0.64 148 65.26 193 64.77 Ad−Hoc Robust Monolingual Dutch track − Comparison to Median Mean Average Precision by Topic (Topics 166 to 200)
1
069 8.49 114 42.57 149 88.19 194 0.00 nlx011nlRFW4FS4
0.5
080 82.77 115 36.49 160 0.00 195 7.69
Difference
081 66.72 116 68.98 161 100.00 196 84.01 0
082 22.78 117 10.92 162 84.90 197 90.98 −0.5
083 8.34 118 17.36 163 13.76 198 36.67 −1
166 167 168 169 170 171 172 173 174 175 176 177 178 179 190 191 192 193 194 195 196 197 198 199 200
084 51.23 119 63.75 164 6.84 199 84.55 Topic Identifier
085 57.95 130 72.01 165 54.35 200 21.29
231
daedalus nlx011nlRFW4FS4 AH-ROBUST-MONO-NL-CLEF2006
Docs Cutoff Levels Precision at DCL (%)
Ad−Hoc Robust Monolingual Dutch track − Retrieved documents vs Precision
100%
5 docs 54.40
nlx011nlRFW4FS4
10 docs 47.10 90%
15 docs 41.33
80%
20 docs 37.55
30 docs 32.10 70%
100 docs 16.60
60%
200 docs 10.03
R−Precision
500 docs 4.60 50%
1000 docs 2.43 40%
R-Precision (precision after R document retrieved,
where R = Relevant retrieved) 30%
42.85
20%
10%
0%
5 10 15 20 30 100 200 500 1000
Retrieved Documents (logarithmic scale)
Exact R-Precision Ad−Hoc Robust Monolingual Dutch track − Box plot of the Topics of the Experiment
Maximum 1.0000
Minimum 0.0000
First Quartile 0.1978
Second Quartile 0.4482
Third Quartile 0.6667
Interquartile range 0.4688
Mean 0.4285
Standard Deviation 0.2845
Lower Outlier Threshold 0.0000
Upper Outlier Threshold 1.0000 0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Exact R−Precision
Mean With No Outliers 0.4285
Std With No Outliers 0.2845
Ad−Hoc Robust Monolingual Dutch track − Distribution of the Topics of the Experiment
20
Number of Topics of the Experiment
15
10
5
0
0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Exact R−Precision
Precision averages (%) for individual Ad−Hoc Robust Monolingual Dutch track − Comparison to Median Mean Exact R−Precision by Topic (Topics 041 to 085)
queries 1
nlx011nlRFW4FS4
0.5
041 26.09 086 56.00 131 25.00 166 0.00
Difference
0
042 16.67 087 46.15 132 61.54 167 0.00
−0.5
043 25.00 088 50.00 133 17.86 168 51.52
044 26.67 089 86.05 134 100.00 169 0.00 −1
041 042 043 044 045 046 047 048 049 060 061 062 063 064 065 066 067 068 069 080 081 082 083 084 085
Topic Identifier
045 35.00 090 38.46 135 66.94 170 71.43 Ad−Hoc Robust Monolingual Dutch track − Comparison to Median Mean Exact R−Precision by Topic (Topics 086 to 130)
1
046 25.00 091 11.11 136 80.00 171 16.67 nlx011nlRFW4FS4
047 35.48 092 55.26 137 0.00 172 28.57 0.5
Difference
048 16.36 093 80.00 138 33.33 173 50.00 0
049 0.00 094 78.95 139 19.57 174 91.30 −0.5
060 11.54 095 61.46 140 67.36 175 0.00 −1
086 087 088 089 090 091 092 093 094 095 096 097 098 099 110 111 112 113 114 115 116 117 118 119 130
061 39.13 096 42.11 141 50.00 176 80.00 Topic Identifier
Ad−Hoc Robust Monolingual Dutch track − Comparison to Median Mean Exact R−Precision by Topic (Topics 131 to 165)
062 50.00 097 75.00 142 72.73 177 50.00 1
nlx011nlRFW4FS4
063 53.33 098 66.67 143 68.97 178 42.86 0.5
Difference
064 37.50 099 33.33 144 100.00 179 46.77
0
065 30.43 110 100.00 145 28.57 190 61.29
−0.5
066 71.43 111 0.00 146 0.00 191 0.00
−1
067 25.00 112 60.00 147 33.33 192 53.33 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 160 161 162 163 164 165
Topic Identifier
068 43.48 113 3.03 148 61.90 193 57.89 Ad−Hoc Robust Monolingual Dutch track − Comparison to Median Mean Exact R−Precision by Topic (Topics 166 to 200)
1
069 0.00 114 43.14 149 66.67 194 0.00 nlx011nlRFW4FS4
0.5
080 81.72 115 46.15 160 0.00 195 0.00
Difference
081 62.96 116 72.41 161 100.00 196 66.67 0
082 20.00 117 22.22 162 72.73 197 80.67 −0.5
083 0.00 118 22.73 163 14.29 198 0.00 −1
166 167 168 169 170 171 172 173 174 175 176 177 178 179 190 191 192 193 194 195 196 197 198 199 200
084 50.00 119 58.06 164 18.00 199 78.12 Topic Identifier
085 66.67 130 55.56 165 50.00 200 35.71
232
daedalus nlFSnlR2S AH-ROBUST-MONO-NL-CLEF2006
Overall statistics for 100 queries : Priority 4
Total number of documents over all queries Query Construction AUTOMATIC
Retrieved 100,000 Source Language Dutch; Flemish
Relevant 2,717 Topic Fields title, description
Relevant retrieved 2,362 Pooled false
Geometric Mean Average Precision 0.1757 Mandatory TD run
Binary Preference (BPREF) 0.4014
Interploated Recall (%) Precision Averages (%)
Ad−Hoc Robust Monolingual Dutch track − Interpolated Recall vs Average Precision
100%
0 73.68
nlFSnlR2S
10 66.17 90%
20 58.24
80%
30 53.64
40 49.19 70%
50 44.61
Average Precision
60%
60 38.67
70 33.45 50%
80 28.40 40%
90 21.89
30%
100 14.51
Average precision (non-interpolated) for all 20%
relevant documents (averaged over queries)
42.39 10%
0%
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
Interpolated Recall
Mean Average Precision Ad−Hoc Robust Monolingual Dutch track − Box plot of the Topics of the Experiment
Maximum 1.0000
Minimum 0.0000
First Quartile 0.1230
Second Quartile 0.4294
Third Quartile 0.7007
Interquartile range 0.5777
Mean 0.4239
Standard Deviation 0.3140
Lower Outlier Threshold 0.0000
Upper Outlier Threshold 1.0000 0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Mean Average Precision
Mean With No Outliers 0.4239
Std With No Outliers 0.3140
Ad−Hoc Robust Monolingual Dutch track − Distribution of the Topics of the Experiment
20
Number of Topics of the Experiment
15
10
5
0
0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Mean Average Precision
Precision averages (%) for individual Ad−Hoc Robust Monolingual Dutch track − Comparison to Median Mean Average Precision by Topic (Topics 041 to 085)
queries 1
nlFSnlR2S
0.5
041 15.07 086 51.54 131 17.26 166 0.00
Difference
0
042 17.81 087 70.14 132 56.95 167 36.67
−0.5
043 44.64 088 16.25 133 18.05 168 42.64
044 25.55 089 86.98 134 100.00 169 2.68 −1
041 042 043 044 045 046 047 048 049 060 061 062 063 064 065 066 067 068 069 080 081 082 083 084 085
Topic Identifier
045 38.97 090 0.67 135 71.52 170 72.20 Ad−Hoc Robust Monolingual Dutch track − Comparison to Median Mean Average Precision by Topic (Topics 086 to 130)
1
046 16.41 091 9.33 136 94.29 171 3.19 nlFSnlR2S
047 26.69 092 43.23 137 1.98 172 16.82 0.5
Difference
048 20.63 093 79.65 138 12.10 173 83.33 0
049 4.88 094 75.27 139 7.23 174 69.86 −0.5
060 10.05 095 60.74 140 73.83 175 0.52 −1
086 087 088 089 090 091 092 093 094 095 096 097 098 099 110 111 112 113 114 115 116 117 118 119 130
061 35.07 096 38.05 141 65.00 176 90.89 Topic Identifier
Ad−Hoc Robust Monolingual Dutch track − Comparison to Median Mean Average Precision by Topic (Topics 131 to 165)
062 54.68 097 63.29 142 72.32 177 50.00 1
nlFSnlR2S
063 61.33 098 88.19 143 75.32 178 43.53 0.5
Difference
064 12.50 099 19.77 144 100.00 179 47.82
0
065 3.59 110 100.00 145 34.77 190 54.51
−0.5
066 70.79 111 0.09 146 36.67 191 0.00
−1
067 12.03 112 45.79 147 24.09 192 25.69 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 160 161 162 163 164 165
Topic Identifier
068 52.95 113 0.46 148 62.46 193 65.84 Ad−Hoc Robust Monolingual Dutch track − Comparison to Median Mean Average Precision by Topic (Topics 166 to 200)
1
069 6.53 114 24.36 149 83.07 194 0.00 nlFSnlR2S
0.5
080 80.25 115 30.71 160 0.00 195 2.38
Difference
081 54.82 116 64.45 161 100.00 196 81.57 0
082 9.79 117 8.14 162 85.05 197 88.70 −0.5
083 6.41 118 15.68 163 16.92 198 70.00 −1
166 167 168 169 170 171 172 173 174 175 176 177 178 179 190 191 192 193 194 195 196 197 198 199 200
084 51.56 119 65.49 164 3.05 199 80.26 Topic Identifier
085 61.25 130 59.35 165 83.33 200 3.22
233
daedalus nlFSnlR2S AH-ROBUST-MONO-NL-CLEF2006
Docs Cutoff Levels Precision at DCL (%)
Ad−Hoc Robust Monolingual Dutch track − Retrieved documents vs Precision
100%
5 docs 51.60
nlFSnlR2S
10 docs 43.60 90%
15 docs 38.67
80%
20 docs 35.30
30 docs 29.97 70%
100 docs 15.48
60%
200 docs 9.44
R−Precision
500 docs 4.36 50%
1000 docs 2.36 40%
R-Precision (precision after R document retrieved,
where R = Relevant retrieved) 30%
39.51
20%
10%
0%
5 10 15 20 30 100 200 500 1000
Retrieved Documents (logarithmic scale)
Exact R-Precision Ad−Hoc Robust Monolingual Dutch track − Box plot of the Topics of the Experiment
Maximum 1.0000
Minimum 0.0000
First Quartile 0.1339
Second Quartile 0.4258
Third Quartile 0.6306
Interquartile range 0.4967
Mean 0.3951
Standard Deviation 0.2889
Lower Outlier Threshold 0.0000
Upper Outlier Threshold 1.0000 0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Exact R−Precision
Mean With No Outliers 0.3951
Std With No Outliers 0.2889
Ad−Hoc Robust Monolingual Dutch track − Distribution of the Topics of the Experiment
20
Number of Topics of the Experiment
15
10
5
0
0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Exact R−Precision
Precision averages (%) for individual Ad−Hoc Robust Monolingual Dutch track − Comparison to Median Mean Exact R−Precision by Topic (Topics 041 to 085)
queries 1
nlFSnlR2S
0.5
041 21.74 086 50.00 131 22.50 166 0.00
Difference
0
042 20.83 087 53.85 132 46.15 167 0.00
−0.5
043 25.00 088 25.00 133 17.86 168 45.45
044 20.00 089 79.07 134 100.00 169 0.00 −1
041 042 043 044 045 046 047 048 049 060 061 062 063 064 065 066 067 068 069 080 081 082 083 084 085
Topic Identifier
045 35.00 090 0.00 135 65.32 170 57.14 Ad−Hoc Robust Monolingual Dutch track − Comparison to Median Mean Exact R−Precision by Topic (Topics 086 to 130)
1
046 25.00 091 22.22 136 80.00 171 5.56 nlFSnlR2S
047 30.65 092 50.00 137 0.00 172 14.29 0.5
Difference
048 25.45 093 85.00 138 11.11 173 50.00 0
049 7.14 094 78.95 139 15.22 174 60.87 −0.5
060 11.54 095 58.80 140 67.36 175 0.00 −1
086 087 088 089 090 091 092 093 094 095 096 097 098 099 110 111 112 113 114 115 116 117 118 119 130
061 34.78 096 36.84 141 50.00 176 80.00 Topic Identifier
Ad−Hoc Robust Monolingual Dutch track − Comparison to Median Mean Exact R−Precision by Topic (Topics 131 to 165)
062 50.00 097 68.75 142 72.73 177 50.00 1
nlFSnlR2S
063 53.33 098 66.67 143 72.41 178 42.86 0.5
Difference
064 12.50 099 16.67 144 100.00 179 46.77
0
065 5.80 110 100.00 145 28.57 190 61.29
−0.5
066 67.86 111 0.00 146 0.00 191 0.00
−1
067 20.00 112 50.00 147 33.33 192 40.00 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 160 161 162 163 164 165
Topic Identifier
068 47.83 113 1.52 148 59.52 193 63.16 Ad−Hoc Robust Monolingual Dutch track − Comparison to Median Mean Exact R−Precision by Topic (Topics 166 to 200)
1
069 0.00 114 31.37 149 66.67 194 0.00 nlFSnlR2S
0.5
080 78.49 115 42.31 160 0.00 195 0.00
Difference
081 62.96 116 75.86 161 100.00 196 66.67 0
082 10.00 117 11.11 162 72.73 197 78.99 −0.5
083 0.00 118 22.73 163 14.29 198 50.00 −1
166 167 168 169 170 171 172 173 174 175 176 177 178 179 190 191 192 193 194 195 196 197 198 199 200
084 50.00 119 51.61 164 6.00 199 71.88 Topic Identifier
085 72.22 130 44.44 165 50.00 200 3.57
234
hummingbird humNL06Rtde AH-ROBUST-MONO-NL-CLEF2006
Overall statistics for 100 queries : Priority 2
Total number of documents over all queries Query Construction AUTOMATIC
Retrieved 100,000 Source Language Dutch; Flemish
Relevant 2,717 Topic Fields title, description
Relevant retrieved 2,512 Pooled false
Geometric Mean Average Precision 0.2576 same as humNL06Rtd except expansion from blind
Binary Preference (BPREF) 0.4856 feedback also used
Interploated Recall (%) Precision Averages (%)
Ad−Hoc Robust Monolingual Dutch track − Interpolated Recall vs Average Precision
100%
0 77.63
humNL06Rtde
10 73.62 90%
20 68.24
80%
30 62.71
40 57.50 70%
50 53.67
Average Precision
60%
60 48.49
70 43.09 50%
80 37.67 40%
90 29.80
30%
100 20.23
Average precision (non-interpolated) for all 20%
relevant documents (averaged over queries)
51.06 10%
0%
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
Interpolated Recall
Mean Average Precision Ad−Hoc Robust Monolingual Dutch track − Box plot of the Topics of the Experiment
Maximum 1.0000
Minimum 0.0000
First Quartile 0.2376
Second Quartile 0.5022
Third Quartile 0.7768
Interquartile range 0.5392
Mean 0.5106
Standard Deviation 0.3114
Lower Outlier Threshold 0.0000
Upper Outlier Threshold 1.0000 0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Mean Average Precision
Mean With No Outliers 0.5106
Std With No Outliers 0.3114
Ad−Hoc Robust Monolingual Dutch track − Distribution of the Topics of the Experiment
8
Number of Topics of the Experiment
7
6
5
4
3
2
1
0
0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Mean Average Precision
Precision averages (%) for individual Ad−Hoc Robust Monolingual Dutch track − Comparison to Median Mean Average Precision by Topic (Topics 041 to 085)
queries 1
humNL06Rtde
0.5
041 27.07 086 62.58 131 29.62 166 0.00
Difference
0
042 17.70 087 63.50 132 77.97 167 100.00
−0.5
043 70.42 088 25.62 133 46.34 168 60.71
044 23.54 089 94.20 134 100.00 169 0.98 −1
041 042 043 044 045 046 047 048 049 060 061 062 063 064 065 066 067 068 069 080 081 082 083 084 085
Topic Identifier
045 48.27 090 46.46 135 81.34 170 85.33 Ad−Hoc Robust Monolingual Dutch track − Comparison to Median Mean Average Precision by Topic (Topics 086 to 130)
1
046 25.63 091 14.39 136 100.00 171 38.43 humNL06Rtde
047 30.89 092 44.09 137 4.69 172 22.13 0.5
Difference
048 31.61 093 77.40 138 23.98 173 83.33 0
049 15.68 094 77.22 139 48.27 174 94.04 −0.5
060 35.47 095 65.65 140 79.20 175 0.98 −1
086 087 088 089 090 091 092 093 094 095 096 097 098 099 110 111 112 113 114 115 116 117 118 119 130
061 66.37 096 34.14 141 40.62 176 94.81 Topic Identifier
Ad−Hoc Robust Monolingual Dutch track − Comparison to Median Mean Average Precision by Topic (Topics 131 to 165)
062 57.91 097 68.44 142 99.95 177 11.26 1
humNL06Rtde
063 81.70 098 88.19 143 87.57 178 50.04 0.5
Difference
064 36.88 099 20.69 144 100.00 179 73.77
0
065 9.63 110 100.00 145 44.68 190 46.16
−0.5
066 67.17 111 21.54 146 75.00 191 0.00
−1
067 23.17 112 56.54 147 16.97 192 64.80 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 160 161 162 163 164 165
Topic Identifier
068 63.07 113 0.45 148 67.79 193 76.80 Ad−Hoc Robust Monolingual Dutch track − Comparison to Median Mean Average Precision by Topic (Topics 166 to 200)
1
069 7.19 114 55.64 149 84.82 194 0.00 humNL06Rtde
0.5
080 80.42 115 40.54 160 0.00 195 8.33
Difference
081 82.55 116 46.24 161 100.00 196 75.35 0
082 53.33 117 16.98 162 89.09 197 83.90 −0.5
083 13.30 118 26.64 163 25.80 198 41.67 −1
166 167 168 169 170 171 172 173 174 175 176 177 178 179 190 191 192 193 194 195 196 197 198 199 200
084 50.40 119 76.25 164 5.98 199 85.85 Topic Identifier
085 52.45 130 71.17 165 100.00 200 10.86
235
hummingbird humNL06Rtde AH-ROBUST-MONO-NL-CLEF2006
Docs Cutoff Levels Precision at DCL (%)
Ad−Hoc Robust Monolingual Dutch track − Retrieved documents vs Precision
100%
5 docs 58.20
humNL06Rtde
10 docs 51.00 90%
15 docs 46.67
80%
20 docs 42.10
30 docs 35.60 70%
100 docs 18.39
60%
200 docs 10.83
R−Precision
500 docs 4.84 50%
1000 docs 2.51 40%
R-Precision (precision after R document retrieved,
where R = Relevant retrieved) 30%
48.92
20%
10%
0%
5 10 15 20 30 100 200 500 1000
Retrieved Documents (logarithmic scale)
Exact R-Precision Ad−Hoc Robust Monolingual Dutch track − Box plot of the Topics of the Experiment
Maximum 1.0000
Minimum 0.0000
First Quartile 0.2740
Second Quartile 0.5000
Third Quartile 0.6972
Interquartile range 0.4232
Mean 0.4892
Standard Deviation 0.2901
Lower Outlier Threshold 0.0000
Upper Outlier Threshold 1.0000 0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Exact R−Precision
Mean With No Outliers 0.4892
Std With No Outliers 0.2901
Ad−Hoc Robust Monolingual Dutch track − Distribution of the Topics of the Experiment
15
Number of Topics of the Experiment
10
5
0
0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Exact R−Precision
Precision averages (%) for individual Ad−Hoc Robust Monolingual Dutch track − Comparison to Median Mean Exact R−Precision by Topic (Topics 041 to 085)
queries 1
humNL06Rtde
0.5
041 30.43 086 58.00 131 35.00 166 0.00
Difference
0
042 20.83 087 61.54 132 61.54 167 100.00
−0.5
043 75.00 088 25.00 133 46.43 168 57.58
044 26.67 089 90.70 134 100.00 169 0.00 −1
041 042 043 044 045 046 047 048 049 060 061 062 063 064 065 066 067 068 069 080 081 082 083 084 085
Topic Identifier
045 52.50 090 50.00 135 74.19 170 85.71 Ad−Hoc Robust Monolingual Dutch track − Comparison to Median Mean Exact R−Precision by Topic (Topics 086 to 130)
1
046 35.42 091 11.11 136 100.00 171 44.44 humNL06Rtde
047 30.65 092 39.47 137 0.00 172 28.57 0.5
Difference
048 34.55 093 80.00 138 33.33 173 50.00 0
049 21.43 094 78.95 139 43.48 174 91.30 −0.5
060 26.92 095 62.79 140 69.44 175 0.00 −1
086 087 088 089 090 091 092 093 094 095 096 097 098 099 110 111 112 113 114 115 116 117 118 119 130
061 65.22 096 36.84 141 25.00 176 80.00 Topic Identifier
Ad−Hoc Robust Monolingual Dutch track − Comparison to Median Mean Exact R−Precision by Topic (Topics 131 to 165)
062 55.00 097 56.25 142 97.73 177 16.67 1
humNL06Rtde
063 73.33 098 66.67 143 79.31 178 42.86 0.5
Difference
064 37.50 099 16.67 144 100.00 179 67.74
0
065 27.54 110 100.00 145 35.71 190 38.71
−0.5
066 60.71 111 35.71 146 50.00 191 0.00
−1
067 30.00 112 60.00 147 16.67 192 66.67 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 160 161 162 163 164 165
Topic Identifier
068 56.52 113 1.52 148 66.67 193 73.68 Ad−Hoc Robust Monolingual Dutch track − Comparison to Median Mean Exact R−Precision by Topic (Topics 166 to 200)
1
069 0.00 114 50.98 149 66.67 194 0.00 humNL06Rtde
0.5
080 80.65 115 46.15 160 0.00 195 0.00
Difference
081 70.37 116 48.28 161 100.00 196 66.67 0
082 70.00 117 11.11 162 78.79 197 74.79 −0.5
083 0.00 118 27.27 163 42.86 198 50.00 −1
166 167 168 169 170 171 172 173 174 175 176 177 178 179 190 191 192 193 194 195 196 197 198 199 200
084 40.00 119 64.52 164 10.00 199 75.00 Topic Identifier
085 66.67 130 66.67 165 100.00 200 14.29
236
hummingbird humNL06Rtd AH-ROBUST-MONO-NL-CLEF2006
Overall statistics for 100 queries : Priority 1
Total number of documents over all queries Query Construction AUTOMATIC
Retrieved 100,000 Source Language Dutch; Flemish
Relevant 2,717 Topic Fields title, description
Relevant retrieved 2,479 Pooled false
Geometric Mean Average Precision 0.2281 plain SearchServer stemming run
Binary Preference (BPREF) 0.4389
Interploated Recall (%) Precision Averages (%)
Ad−Hoc Robust Monolingual Dutch track − Interpolated Recall vs Average Precision
100%
0 78.69
humNL06Rtd
10 72.26 90%
20 63.49
80%
30 56.32
40 51.45 70%
50 47.58
Average Precision
60%
60 42.54
70 37.48 50%
80 32.42 40%
90 25.53
30%
100 17.48
Average precision (non-interpolated) for all 20%
relevant documents (averaged over queries)
46.47 10%
0%
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
Interpolated Recall
Mean Average Precision Ad−Hoc Robust Monolingual Dutch track − Box plot of the Topics of the Experiment
Maximum 1.0000
Minimum 0.0000
First Quartile 0.2009
Second Quartile 0.4424
Third Quartile 0.7455
Interquartile range 0.5446
Mean 0.4647
Standard Deviation 0.3105
Lower Outlier Threshold 0.0000
Upper Outlier Threshold 1.0000 0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Mean Average Precision
Mean With No Outliers 0.4647
Std With No Outliers 0.3105
Ad−Hoc Robust Monolingual Dutch track − Distribution of the Topics of the Experiment
10
Number of Topics of the Experiment
8
6
4
2
0
0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Mean Average Precision
Precision averages (%) for individual Ad−Hoc Robust Monolingual Dutch track − Comparison to Median Mean Average Precision by Topic (Topics 041 to 085)
queries 1
humNL06Rtd
0.5
041 22.67 086 54.33 131 26.51 166 0.00
Difference
0
042 16.79 087 77.11 132 60.02 167 83.33
−0.5
043 65.00 088 6.76 133 18.93 168 46.53
044 26.40 089 86.07 134 100.00 169 1.56 −1
041 042 043 044 045 046 047 048 049 060 061 062 063 064 065 066 067 068 069 080 081 082 083 084 085
Topic Identifier
045 41.12 090 19.26 135 66.60 170 79.81 Ad−Hoc Robust Monolingual Dutch track − Comparison to Median Mean Average Precision by Topic (Topics 086 to 130)
1
046 24.91 091 12.88 136 100.00 171 35.39 humNL06Rtd
047 25.23 092 42.37 137 2.55 172 18.75 0.5
Difference
048 26.60 093 75.77 138 21.70 173 100.00 0
049 17.44 094 75.56 139 35.45 174 92.33 −0.5
060 26.33 095 55.11 140 75.64 175 0.79 −1
086 087 088 089 090 091 092 093 094 095 096 097 098 099 110 111 112 113 114 115 116 117 118 119 130
061 57.51 096 35.78 141 31.79 176 93.48 Topic Identifier
Ad−Hoc Robust Monolingual Dutch track − Comparison to Median Mean Average Precision by Topic (Topics 131 to 165)
062 57.90 097 45.53 142 99.90 177 9.08 1
humNL06Rtd
063 83.53 098 88.19 143 83.71 178 45.71 0.5
Difference
064 29.29 099 17.22 144 100.00 179 43.81
0
065 8.99 110 100.00 145 33.38 190 38.95
−0.5
066 66.23 111 20.97 146 70.00 191 0.00
−1
067 13.13 112 56.33 147 35.76 192 52.53 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 160 161 162 163 164 165
Topic Identifier
068 54.02 113 1.54 148 63.78 193 69.53 Ad−Hoc Robust Monolingual Dutch track − Comparison to Median Mean Average Precision by Topic (Topics 166 to 200)
1
069 11.27 114 23.64 149 82.18 194 0.00 humNL06Rtd
0.5
080 81.30 115 27.26 160 0.00 195 1.39
Difference
081 64.45 116 50.57 161 100.00 196 73.54 0
082 44.68 117 6.93 162 84.10 197 91.11 −0.5
083 20.91 118 18.13 163 25.21 198 41.67 −1
166 167 168 169 170 171 172 173 174 175 176 177 178 179 190 191 192 193 194 195 196 197 198 199 200
084 58.98 119 49.10 164 8.46 199 79.67 Topic Identifier
085 63.46 130 58.67 165 100.00 200 9.38
237
hummingbird humNL06Rtd AH-ROBUST-MONO-NL-CLEF2006
Docs Cutoff Levels Precision at DCL (%)
Ad−Hoc Robust Monolingual Dutch track − Retrieved documents vs Precision
100%
5 docs 55.80
humNL06Rtd
10 docs 45.90 90%
15 docs 41.73
80%
20 docs 37.65
30 docs 32.23 70%
100 docs 16.90
60%
200 docs 10.24
R−Precision
500 docs 4.69 50%
1000 docs 2.48 40%
R-Precision (precision after R document retrieved,
where R = Relevant retrieved) 30%
44.15
20%
10%
0%
5 10 15 20 30 100 200 500 1000
Retrieved Documents (logarithmic scale)
Exact R-Precision Ad−Hoc Robust Monolingual Dutch track − Box plot of the Topics of the Experiment
Maximum 1.0000
Minimum 0.0000
First Quartile 0.2111
Second Quartile 0.4221
Third Quartile 0.6667
Interquartile range 0.4556
Mean 0.4415
Standard Deviation 0.2929
Lower Outlier Threshold 0.0000
Upper Outlier Threshold 1.0000 0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Exact R−Precision
Mean With No Outliers 0.4415
Std With No Outliers 0.2929
Ad−Hoc Robust Monolingual Dutch track − Distribution of the Topics of the Experiment
15
Number of Topics of the Experiment
10
5
0
0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Exact R−Precision
Precision averages (%) for individual Ad−Hoc Robust Monolingual Dutch track − Comparison to Median Mean Exact R−Precision by Topic (Topics 041 to 085)
queries 1
humNL06Rtd
0.5
041 26.09 086 48.00 131 35.00 166 0.00
Difference
0
042 16.67 087 61.54 132 38.46 167 50.00
−0.5
043 50.00 088 25.00 133 17.86 168 42.42
044 26.67 089 81.40 134 100.00 169 0.00 −1
041 042 043 044 045 046 047 048 049 060 061 062 063 064 065 066 067 068 069 080 081 082 083 084 085
Topic Identifier
045 50.00 090 19.23 135 66.13 170 78.57 Ad−Hoc Robust Monolingual Dutch track − Comparison to Median Mean Exact R−Precision by Topic (Topics 086 to 130)
1
046 35.42 091 11.11 136 100.00 171 33.33 humNL06Rtd
047 25.81 092 42.11 137 0.00 172 14.29 0.5
Difference
048 30.91 093 75.00 138 22.22 173 100.00 0
049 14.29 094 73.68 139 36.96 174 86.96 −0.5
060 26.92 095 52.49 140 69.44 175 0.00 −1
086 087 088 089 090 091 092 093 094 095 096 097 098 099 110 111 112 113 114 115 116 117 118 119 130
061 60.87 096 36.84 141 25.00 176 85.00 Topic Identifier
Ad−Hoc Robust Monolingual Dutch track − Comparison to Median Mean Exact R−Precision by Topic (Topics 131 to 165)
062 55.00 097 43.75 142 97.73 177 0.00 1
humNL06Rtd
063 73.33 098 66.67 143 79.31 178 42.86 0.5
Difference
064 25.00 099 16.67 144 100.00 179 45.16
0
065 24.64 110 100.00 145 35.71 190 41.94
−0.5
066 67.86 111 28.57 146 50.00 191 0.00
−1
067 20.00 112 60.00 147 33.33 192 60.00 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 160 161 162 163 164 165
Topic Identifier
068 52.17 113 6.06 148 64.29 193 63.16 Ad−Hoc Robust Monolingual Dutch track − Comparison to Median Mean Exact R−Precision by Topic (Topics 166 to 200)
1
069 0.00 114 27.45 149 66.67 194 0.00 humNL06Rtd
0.5
080 82.80 115 42.31 160 0.00 195 0.00
Difference
081 59.26 116 62.07 161 100.00 196 66.67 0
082 40.00 117 11.11 162 72.73 197 82.35 −0.5
083 16.67 118 18.18 163 28.57 198 0.00 −1
166 167 168 169 170 171 172 173 174 175 176 177 178 179 190 191 192 193 194 195 196 197 198 199 200
084 40.00 119 45.16 164 12.00 199 71.88 Topic Identifier
085 72.22 130 44.44 165 100.00 200 7.14
238
colesir CoLesIRmultTst AH-ROBUST-MULTI-CLEF2006
Overall statistics for 100 queries : Priority 1
Total number of documents over all queries Query Construction AUTOMATIC
Retrieved 100,000 Source Language English
Relevant 17,019 Topic Fields title, description
Relevant retrieved 8,877 Pooled false
Geometric Mean Average Precision 0.1124 robut multilingual ENX - TEST; automatic; TD
Binary Preference (BPREF) 0.3082
Interploated Recall (%) Precision Averages (%)
Ad−Hoc Robust Multilingual track − Interpolated Recall vs Average Precision
100%
0 66.50
CoLesIRmultTst
10 44.44 90%
20 37.27
80%
30 30.78
40 26.89 70%
50 22.05
Average Precision
60%
60 17.04
70 13.26 50%
80 8.72 40%
90 4.73
30%
100 0.24
Average precision (non-interpolated) for all 20%
relevant documents (averaged over queries)
22.63 10%
0%
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
Interpolated Recall
Mean Average Precision Ad−Hoc Robust Multilingual track − Box plot of the Topics of the Experiment
Maximum 0.8997
Minimum 0.0000
First Quartile 0.0623
Second Quartile 0.1443
Third Quartile 0.3577
Interquartile range 0.2954
Mean 0.2263
Standard Deviation 0.2125
Lower Outlier Threshold 0.0000
Upper Outlier Threshold 0.7928 0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Mean Average Precision
Mean With No Outliers 0.2132
Std With No Outliers 0.1934
Ad−Hoc Robust Multilingual track − Distribution of the Topics of the Experiment
20
Number of Topics of the Experiment
15
10
5
0
0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Mean Average Precision
Precision averages (%) for individual Ad−Hoc Robust Multilingual track − Comparison to Median Mean Average Precision by Topic (Topics 041 to 085)
queries 1
CoLesIRmultTst
0.5
041 9.77 086 11.04 131 10.65 166 0.00
Difference
0
042 27.71 087 28.80 132 43.03 167 45.96
−0.5
043 13.66 088 37.56 133 1.92 168 8.88
044 7.85 089 24.94 134 89.97 169 0.02 −1
041 042 043 044 045 046 047 048 049 060 061 062 063 064 065 066 067 068 069 080 081 082 083 084 085
Topic Identifier
045 27.60 090 12.03 135 26.03 170 21.47 Ad−Hoc Robust Multilingual track − Comparison to Median Mean Average Precision by Topic (Topics 086 to 130)
1
046 6.02 091 5.30 136 17.42 171 12.14 CoLesIRmultTst
047 7.55 092 15.72 137 4.04 172 1.11 0.5
Difference
048 3.13 093 51.94 138 15.35 173 47.48 0
049 6.99 094 57.42 139 0.74 174 83.56 −0.5
060 14.94 095 5.37 140 18.92 175 22.15 −1
086 087 088 089 090 091 092 093 094 095 096 097 098 099 110 111 112 113 114 115 116 117 118 119 130
061 12.32 096 9.76 141 46.80 176 74.32 Topic Identifier
Ad−Hoc Robust Multilingual track − Comparison to Median Mean Average Precision by Topic (Topics 131 to 165)
062 30.25 097 24.53 142 79.28 177 3.31 1
CoLesIRmultTst
063 44.66 098 54.59 143 18.41 178 5.19 0.5
Difference
064 18.32 099 7.00 144 36.67 179 12.38
0
065 3.72 110 59.46 145 9.93 190 5.77
−0.5
066 41.74 111 3.01 146 13.92 191 20.00
−1
067 2.33 112 37.73 147 7.18 192 6.04 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 160 161 162 163 164 165
Topic Identifier
068 10.25 113 0.04 148 7.00 193 20.95 Ad−Hoc Robust Multilingual track − Comparison to Median Mean Average Precision by Topic (Topics 166 to 200)
1
069 5.58 114 19.07 149 51.29 194 3.65 CoLesIRmultTst
0.5
080 4.26 115 11.32 160 22.14 195 1.77
Difference
081 33.28 116 42.49 161 3.50 196 61.75 0
082 60.81 117 6.42 162 31.11 197 39.82 −0.5
083 12.82 118 6.82 163 40.20 198 3.70 −1
166 167 168 169 170 171 172 173 174 175 176 177 178 179 190 191 192 193 194 195 196 197 198 199 200
084 8.33 119 61.41 164 0.05 199 62.78 Topic Identifier
085 23.97 130 25.74 165 34.87 200 10.97
239
colesir CoLesIRmultTst AH-ROBUST-MULTI-CLEF2006
Docs Cutoff Levels Precision at DCL (%)
Ad−Hoc Robust Multilingual track − Retrieved documents vs Precision
100%
5 docs 43.60
CoLesIRmultTst
10 docs 43.60 90%
15 docs 41.67
80%
20 docs 40.40
30 docs 37.33 70%
100 docs 28.04
60%
200 docs 22.10
R−Precision
500 docs 13.66 50%
1000 docs 8.88 40%
R-Precision (precision after R document retrieved,
where R = Relevant retrieved) 30%
29.02
20%
10%
0%
5 10 15 20 30 100 200 500 1000
Retrieved Documents (logarithmic scale)
Exact R-Precision Ad−Hoc Robust Multilingual track − Box plot of the Topics of the Experiment
Maximum 0.8519
Minimum 0.0000
First Quartile 0.1361
Second Quartile 0.2337
Third Quartile 0.4392
Interquartile range 0.3031
Mean 0.2902
Standard Deviation 0.1975
Lower Outlier Threshold 0.0000
Upper Outlier Threshold 0.8519 0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Exact R−Precision
Mean With No Outliers 0.2902
Std With No Outliers 0.1975
Ad−Hoc Robust Multilingual track − Distribution of the Topics of the Experiment
15
Number of Topics of the Experiment
10
5
0
0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Exact R−Precision
Precision averages (%) for individual Ad−Hoc Robust Multilingual track − Comparison to Median Mean Exact R−Precision by Topic (Topics 041 to 085)
queries 1
CoLesIRmultTst
0.5
041 13.10 086 25.14 131 23.22 166 0.00
Difference
0
042 45.69 087 36.48 132 48.65 167 44.00
−0.5
043 19.44 088 43.97 133 7.91 168 18.42
044 13.85 089 39.30 134 85.19 169 0.00 −1
041 042 043 044 045 046 047 048 049 060 061 062 063 064 065 066 067 068 069 080 081 082 083 084 085
Topic Identifier
045 39.65 090 23.60 135 41.72 170 33.33 Ad−Hoc Robust Multilingual track − Comparison to Median Mean Exact R−Precision by Topic (Topics 086 to 130)
1
046 8.93 091 12.03 136 31.43 171 22.03 CoLesIRmultTst
047 18.69 092 28.25 137 8.00 172 3.57 0.5
Difference
048 11.81 093 59.52 138 19.33 173 48.00 0
049 16.98 094 58.68 139 4.81 174 78.16 −0.5
060 24.14 095 21.77 140 35.47 175 22.73 −1
086 087 088 089 090 091 092 093 094 095 096 097 098 099 110 111 112 113 114 115 116 117 118 119 130
061 21.21 096 19.25 141 47.06 176 75.10 Topic Identifier
Ad−Hoc Robust Multilingual track − Comparison to Median Mean Exact R−Precision by Topic (Topics 131 to 165)
062 33.33 097 29.41 142 72.67 177 9.68 1
CoLesIRmultTst
063 53.62 098 51.72 143 34.36 178 14.17 0.5
Difference
064 25.00 099 15.10 144 46.67 179 21.77
0
065 10.36 110 64.95 145 14.52 190 13.74
−0.5
066 48.74 111 11.84 146 16.67 191 0.00
−1
067 8.76 112 46.38 147 15.74 192 13.48 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 160 161 162 163 164 165
Topic Identifier
068 16.67 113 1.06 148 9.33 193 27.34 Ad−Hoc Robust Multilingual track − Comparison to Median Mean Exact R−Precision by Topic (Topics 166 to 200)
1
069 13.04 114 28.16 149 53.85 194 12.50 CoLesIRmultTst
0.5
080 16.85 115 21.43 160 20.00 195 3.28
Difference
081 41.20 116 43.88 161 11.11 196 63.64 0
082 58.96 117 16.13 162 38.50 197 51.42 −0.5
083 23.53 118 13.04 163 49.80 198 9.09 −1
166 167 168 169 170 171 172 173 174 175 176 177 178 179 190 191 192 193 194 195 196 197 198 199 200
084 18.60 119 60.51 164 1.47 199 63.52 Topic Identifier
085 37.13 130 37.59 165 42.11 200 24.63
240
daedalus ml4XRSFSen4S AH-ROBUST-MULTI-CLEF2006
Overall statistics for 100 queries : Priority 3
Total number of documents over all queries Query Construction AUTOMATIC
Retrieved 100,000 Source Language English
Relevant 17,019 Topic Fields title, description, narrative
Relevant retrieved 9,423 Pooled false
Geometric Mean Average Precision 0.1516 TDN run
Binary Preference (BPREF) 0.3331
Interploated Recall (%) Precision Averages (%)
Ad−Hoc Robust Multilingual track − Interpolated Recall vs Average Precision
100%
0 77.92
ml4XRSFSen4S
10 52.44 90%
20 43.46
80%
30 36.24
40 30.06 70%
50 25.01
Average Precision
60%
60 19.61
70 13.18 50%
80 7.34 40%
90 3.05
30%
100 0.24
Average precision (non-interpolated) for all 20%
relevant documents (averaged over queries)
25.65 10%
0%
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
Interpolated Recall
Mean Average Precision Ad−Hoc Robust Multilingual track − Box plot of the Topics of the Experiment
Maximum 0.8191
Minimum 0.0013
First Quartile 0.0802
Second Quartile 0.1834
Third Quartile 0.4191
Interquartile range 0.3390
Mean 0.2565
Standard Deviation 0.2087
Lower Outlier Threshold 0.0013
Upper Outlier Threshold 0.8191 0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Mean Average Precision
Mean With No Outliers 0.2565
Std With No Outliers 0.2087
Ad−Hoc Robust Multilingual track − Distribution of the Topics of the Experiment
15
Number of Topics of the Experiment
10
5
0
0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Mean Average Precision
Precision averages (%) for individual Ad−Hoc Robust Multilingual track − Comparison to Median Mean Average Precision by Topic (Topics 041 to 085)
queries 1
ml4XRSFSen4S
0.5
041 16.01 086 16.64 131 16.89 166 0.29
Difference
0
042 7.68 087 38.65 132 55.33 167 42.00
−0.5
043 40.02 088 30.85 133 2.04 168 17.14
044 7.30 089 40.98 134 62.96 169 0.57 −1
041 042 043 044 045 046 047 048 049 060 061 062 063 064 065 066 067 068 069 080 081 082 083 084 085
Topic Identifier
045 36.27 090 11.96 135 5.24 170 51.39 Ad−Hoc Robust Multilingual track − Comparison to Median Mean Average Precision by Topic (Topics 086 to 130)
1
046 10.13 091 17.11 136 34.15 171 8.30 ml4XRSFSen4S
047 12.14 092 22.33 137 12.87 172 5.40 0.5
Difference
048 5.12 093 48.79 138 33.97 173 68.27 0
049 6.74 094 13.28 139 1.55 174 78.06 −0.5
060 14.96 095 7.83 140 19.57 175 43.49 −1
086 087 088 089 090 091 092 093 094 095 096 097 098 099 110 111 112 113 114 115 116 117 118 119 130
061 8.22 096 11.59 141 71.23 176 81.91 Topic Identifier
Ad−Hoc Robust Multilingual track − Comparison to Median Mean Average Precision by Topic (Topics 131 to 165)
062 29.24 097 10.48 142 56.69 177 15.87 1
ml4XRSFSen4S
063 47.20 098 44.17 143 27.00 178 3.09 0.5
Difference
064 41.83 099 8.10 144 19.54 179 26.56
0
065 3.68 110 51.70 145 48.46 190 16.33
−0.5
066 48.11 111 9.24 146 34.15 191 2.04
−1
067 7.28 112 20.16 147 6.01 192 16.20 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 160 161 162 163 164 165
Topic Identifier
068 11.13 113 0.14 148 4.65 193 26.04 Ad−Hoc Robust Multilingual track − Comparison to Median Mean Average Precision by Topic (Topics 166 to 200)
1
069 13.92 114 26.94 149 68.11 194 4.59 ml4XRSFSen4S
0.5
080 22.02 115 14.55 160 6.57 195 7.94
Difference
081 48.05 116 35.79 161 0.13 196 56.20 0
082 56.61 117 3.41 162 29.00 197 44.75 −0.5
083 16.15 118 2.65 163 49.49 198 22.38 −1
166 167 168 169 170 171 172 173 174 175 176 177 178 179 190 191 192 193 194 195 196 197 198 199 200
084 31.86 119 52.22 164 0.65 199 72.57 Topic Identifier
085 21.25 130 42.77 165 10.84 200 31.23
241
daedalus ml4XRSFSen4S AH-ROBUST-MULTI-CLEF2006
Docs Cutoff Levels Precision at DCL (%)
Ad−Hoc Robust Multilingual track − Retrieved documents vs Precision
100%
5 docs 57.60
ml4XRSFSen4S
10 docs 53.10 90%
15 docs 50.67
80%
20 docs 48.40
30 docs 43.50 70%
100 docs 30.08
60%
200 docs 22.49
R−Precision
500 docs 14.28 50%
1000 docs 9.42 40%
R-Precision (precision after R document retrieved,
where R = Relevant retrieved) 30%
30.82
20%
10%
0%
5 10 15 20 30 100 200 500 1000
Retrieved Documents (logarithmic scale)
Exact R-Precision Ad−Hoc Robust Multilingual track − Box plot of the Topics of the Experiment
Maximum 0.7959
Minimum 0.0000
First Quartile 0.1558
Second Quartile 0.2792
Third Quartile 0.4634
Interquartile range 0.3076
Mean 0.3082
Standard Deviation 0.1973
Lower Outlier Threshold 0.0000
Upper Outlier Threshold 0.7959 0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Exact R−Precision
Mean With No Outliers 0.3082
Std With No Outliers 0.1973
Ad−Hoc Robust Multilingual track − Distribution of the Topics of the Experiment
15
Number of Topics of the Experiment
10
5
0
0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Exact R−Precision
Precision averages (%) for individual Ad−Hoc Robust Multilingual track − Comparison to Median Mean Exact R−Precision by Topic (Topics 041 to 085)
queries 1
ml4XRSFSen4S
0.5
041 21.43 086 24.32 131 30.87 166 0.00
Difference
0
042 26.31 087 41.51 132 50.00 167 44.00
−0.5
043 52.78 088 36.21 133 7.91 168 27.63
044 4.62 089 46.30 134 62.96 169 0.00 −1
041 042 043 044 045 046 047 048 049 060 061 062 063 064 065 066 067 068 069 080 081 082 083 084 085
Topic Identifier
045 49.67 090 21.35 135 12.97 170 47.62 Ad−Hoc Robust Multilingual track − Comparison to Median Mean Exact R−Precision by Topic (Topics 086 to 130)
1
046 17.86 091 30.38 136 40.00 171 13.56 ml4XRSFSen4S
047 17.80 092 29.00 137 20.00 172 10.71 0.5
Difference
048 14.76 093 59.52 138 36.97 173 68.00 0
049 14.15 094 22.31 139 10.90 174 72.41 −0.5
060 28.21 095 23.91 140 39.53 175 45.45 −1
086 087 088 089 090 091 092 093 094 095 096 097 098 099 110 111 112 113 114 115 116 117 118 119 130
061 13.13 096 19.88 141 70.59 176 79.59 Topic Identifier
Ad−Hoc Robust Multilingual track − Comparison to Median Mean Exact R−Precision by Topic (Topics 131 to 165)
062 33.33 097 14.71 142 59.88 177 22.58 1
ml4XRSFSen4S
063 46.38 098 55.17 143 37.43 178 11.81 0.5
Difference
064 50.00 099 17.19 144 20.00 179 38.44
0
065 7.17 110 55.67 145 51.61 190 25.95
−0.5
066 46.57 111 17.11 146 33.33 191 0.00
−1
067 18.56 112 23.19 147 6.48 192 20.57 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 160 161 162 163 164 165
Topic Identifier
068 18.12 113 0.70 148 8.00 193 29.50 Ad−Hoc Robust Multilingual track − Comparison to Median Mean Exact R−Precision by Topic (Topics 166 to 200)
1
069 21.74 114 30.46 149 64.10 194 8.33 ml4XRSFSen4S
0.5
080 41.65 115 24.60 160 0.00 195 16.39
Difference
081 57.41 116 40.82 161 0.00 196 60.61 0
082 57.46 117 5.16 162 32.30 197 55.57 −0.5
083 20.59 118 6.52 163 54.66 198 21.21 −1
166 167 168 169 170 171 172 173 174 175 176 177 178 179 190 191 192 193 194 195 196 197 198 199 200
084 37.21 119 53.04 164 6.16 199 68.03 Topic Identifier
085 32.04 130 45.11 165 10.53 200 33.99
242
daedalus ml5XRSFSen4S AH-ROBUST-MULTI-CLEF2006
Overall statistics for 100 queries : Priority 2
Total number of documents over all queries Query Construction AUTOMATIC
Retrieved 100,000 Source Language English
Relevant 17,019 Topic Fields title, description, narrative
Relevant retrieved 9,450 Pooled false
Geometric Mean Average Precision 0.1517 TDN run
Binary Preference (BPREF) 0.3335
Interploated Recall (%) Precision Averages (%)
Ad−Hoc Robust Multilingual track − Interpolated Recall vs Average Precision
100%
0 77.92
ml5XRSFSen4S
10 52.44 90%
20 43.46
80%
30 36.23
40 30.06 70%
50 25.01
Average Precision
60%
60 19.64
70 13.51 50%
80 7.28 40%
90 3.26
30%
100 0.24
Average precision (non-interpolated) for all 20%
relevant documents (averaged over queries)
25.67 10%
0%
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
Interpolated Recall
Mean Average Precision Ad−Hoc Robust Multilingual track − Box plot of the Topics of the Experiment
Maximum 0.8200
Minimum 0.0013
First Quartile 0.0801
Second Quartile 0.1847
Third Quartile 0.4191
Interquartile range 0.3390
Mean 0.2567
Standard Deviation 0.2088
Lower Outlier Threshold 0.0013
Upper Outlier Threshold 0.8200 0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Mean Average Precision
Mean With No Outliers 0.2567
Std With No Outliers 0.2088
Ad−Hoc Robust Multilingual track − Distribution of the Topics of the Experiment
15
Number of Topics of the Experiment
10
5
0
0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Mean Average Precision
Precision averages (%) for individual Ad−Hoc Robust Multilingual track − Comparison to Median Mean Average Precision by Topic (Topics 041 to 085)
queries 1
ml5XRSFSen4S
0.5
041 15.99 086 16.66 131 17.02 166 0.29
Difference
0
042 7.15 087 38.85 132 55.33 167 42.00
−0.5
043 40.03 088 30.85 133 2.04 168 17.21
044 7.40 089 41.10 134 62.95 169 0.57 −1
041 042 043 044 045 046 047 048 049 060 061 062 063 064 065 066 067 068 069 080 081 082 083 084 085
Topic Identifier
045 36.81 090 11.96 135 5.20 170 51.39 Ad−Hoc Robust Multilingual track − Comparison to Median Mean Average Precision by Topic (Topics 086 to 130)
1
046 10.13 091 17.26 136 34.16 171 8.36 ml5XRSFSen4S
047 12.41 092 22.40 137 12.96 172 5.40 0.5
Difference
048 5.12 093 48.79 138 33.96 173 68.27 0
049 6.74 094 13.27 139 1.55 174 78.05 −0.5
060 14.96 095 7.78 140 19.76 175 43.48 −1
086 087 088 089 090 091 092 093 094 095 096 097 098 099 110 111 112 113 114 115 116 117 118 119 130
061 8.22 096 11.66 141 71.32 176 82.00 Topic Identifier
Ad−Hoc Robust Multilingual track − Comparison to Median Mean Average Precision by Topic (Topics 131 to 165)
062 29.23 097 10.48 142 56.69 177 15.87 1
ml5XRSFSen4S
063 47.37 098 44.15 143 27.17 178 3.09 0.5
Difference
064 41.83 099 8.10 144 19.69 179 26.64
0
065 3.71 110 51.71 145 48.47 190 16.32
−0.5
066 48.22 111 9.24 146 34.15 191 2.04
−1
067 7.28 112 20.04 147 5.99 192 16.20 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 160 161 162 163 164 165
Topic Identifier
068 11.07 113 0.14 148 4.65 193 26.03 Ad−Hoc Robust Multilingual track − Comparison to Median Mean Average Precision by Topic (Topics 166 to 200)
1
069 13.90 114 26.93 149 68.11 194 4.59 ml5XRSFSen4S
0.5
080 22.24 115 14.54 160 6.57 195 7.93
Difference
081 48.05 116 35.79 161 0.13 196 56.20 0
082 56.74 117 3.49 162 29.10 197 44.37 −0.5
083 16.15 118 2.65 163 49.62 198 22.38 −1
166 167 168 169 170 171 172 173 174 175 176 177 178 179 190 191 192 193 194 195 196 197 198 199 200
084 31.86 119 52.27 164 0.66 199 72.56 Topic Identifier
085 21.26 130 42.77 165 10.83 200 31.23
243
daedalus ml5XRSFSen4S AH-ROBUST-MULTI-CLEF2006
Docs Cutoff Levels Precision at DCL (%)
Ad−Hoc Robust Multilingual track − Retrieved documents vs Precision
100%
5 docs 57.60
ml5XRSFSen4S
10 docs 53.10 90%
15 docs 50.67
80%
20 docs 48.40
30 docs 43.50 70%
100 docs 30.08
60%
200 docs 22.49
R−Precision
500 docs 14.25 50%
1000 docs 9.45 40%
R-Precision (precision after R document retrieved,
where R = Relevant retrieved) 30%
30.80
20%
10%
0%
5 10 15 20 30 100 200 500 1000
Retrieved Documents (logarithmic scale)
Exact R-Precision Ad−Hoc Robust Multilingual track − Box plot of the Topics of the Experiment
Maximum 0.7959
Minimum 0.0000
First Quartile 0.1558
Second Quartile 0.2792
Third Quartile 0.4634
Interquartile range 0.3076
Mean 0.3080
Standard Deviation 0.1973
Lower Outlier Threshold 0.0000
Upper Outlier Threshold 0.7959 0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Exact R−Precision
Mean With No Outliers 0.3080
Std With No Outliers 0.1973
Ad−Hoc Robust Multilingual track − Distribution of the Topics of the Experiment
15
Number of Topics of the Experiment
10
5
0
0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Exact R−Precision
Precision averages (%) for individual Ad−Hoc Robust Multilingual track − Comparison to Median Mean Exact R−Precision by Topic (Topics 041 to 085)
queries 1
ml5XRSFSen4S
0.5
041 21.43 086 24.32 131 30.87 166 0.00
Difference
0
042 25.18 087 41.51 132 50.00 167 44.00
−0.5
043 52.78 088 36.21 133 7.91 168 27.63
044 4.62 089 46.30 134 62.96 169 0.00 −1
041 042 043 044 045 046 047 048 049 060 061 062 063 064 065 066 067 068 069 080 081 082 083 084 085
Topic Identifier
045 49.67 090 21.35 135 12.97 170 47.62 Ad−Hoc Robust Multilingual track − Comparison to Median Mean Exact R−Precision by Topic (Topics 086 to 130)
1
046 17.86 091 30.38 136 40.00 171 13.56 ml5XRSFSen4S
047 17.80 092 29.00 137 20.00 172 10.71 0.5
Difference
048 14.76 093 59.52 138 36.97 173 68.00 0
049 14.15 094 22.31 139 10.90 174 72.41 −0.5
060 28.21 095 23.72 140 39.53 175 45.45 −1
086 087 088 089 090 091 092 093 094 095 096 097 098 099 110 111 112 113 114 115 116 117 118 119 130
061 13.13 096 19.88 141 70.59 176 79.59 Topic Identifier
Ad−Hoc Robust Multilingual track − Comparison to Median Mean Exact R−Precision by Topic (Topics 131 to 165)
062 33.33 097 14.71 142 59.88 177 22.58 1
ml5XRSFSen4S
063 46.38 098 55.17 143 37.43 178 11.81 0.5
Difference
064 50.00 099 17.19 144 20.00 179 38.44
0
065 7.17 110 55.67 145 51.61 190 25.95
−0.5
066 46.57 111 17.11 146 33.33 191 0.00
−1
067 18.56 112 23.19 147 6.48 192 20.57 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 160 161 162 163 164 165
Topic Identifier
068 18.12 113 0.70 148 8.00 193 29.50 Ad−Hoc Robust Multilingual track − Comparison to Median Mean Exact R−Precision by Topic (Topics 166 to 200)
1
069 21.74 114 30.46 149 64.10 194 8.33 ml5XRSFSen4S
0.5
080 40.70 115 24.60 160 0.00 195 16.39
Difference
081 57.41 116 40.82 161 0.00 196 60.61 0
082 57.46 117 5.16 162 32.30 197 55.44 −0.5
083 20.59 118 6.52 163 54.66 198 21.21 −1
166 167 168 169 170 171 172 173 174 175 176 177 178 179 190 191 192 193 194 195 196 197 198 199 200
084 37.21 119 53.04 164 6.16 199 68.03 Topic Identifier
085 32.04 130 45.11 165 10.53 200 33.99
244
daedalus ml6XRSFSen4S AH-ROBUST-MULTI-CLEF2006
Overall statistics for 100 queries : Priority 1
Total number of documents over all queries Query Construction AUTOMATIC
Retrieved 100,000 Source Language English
Relevant 17,019 Topic Fields title, description, narrative
Relevant retrieved 9,445 Pooled false
Geometric Mean Average Precision 0.1512 TDN run
Binary Preference (BPREF) 0.3335
Interploated Recall (%) Precision Averages (%)
Ad−Hoc Robust Multilingual track − Interpolated Recall vs Average Precision
100%
0 77.91
ml6XRSFSen4S
10 52.42 90%
20 43.45
80%
30 36.21
40 30.19 70%
50 24.99
Average Precision
60%
60 19.85
70 13.53 50%
80 7.29 40%
90 3.31
30%
100 0.24
Average precision (non-interpolated) for all 20%
relevant documents (averaged over queries)
25.67 10%
0%
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
Interpolated Recall
Mean Average Precision Ad−Hoc Robust Multilingual track − Box plot of the Topics of the Experiment
Maximum 0.8208
Minimum 0.0009
First Quartile 0.0799
Second Quartile 0.1856
Third Quartile 0.4191
Interquartile range 0.3393
Mean 0.2567
Standard Deviation 0.2090
Lower Outlier Threshold 0.0009
Upper Outlier Threshold 0.8208 0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Mean Average Precision
Mean With No Outliers 0.2567
Std With No Outliers 0.2090
Ad−Hoc Robust Multilingual track − Distribution of the Topics of the Experiment
15
Number of Topics of the Experiment
10
5
0
0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Mean Average Precision
Precision averages (%) for individual Ad−Hoc Robust Multilingual track − Comparison to Median Mean Average Precision by Topic (Topics 041 to 085)
queries 1
ml6XRSFSen4S
0.5
041 15.54 086 16.58 131 16.89 166 0.29
Difference
0
042 6.26 087 38.84 132 55.33 167 41.99
−0.5
043 40.07 088 30.95 133 2.04 168 17.43
044 7.39 089 41.23 134 62.95 169 0.57 −1
041 042 043 044 045 046 047 048 049 060 061 062 063 064 065 066 067 068 069 080 081 082 083 084 085
Topic Identifier
045 36.91 090 11.87 135 5.02 170 51.39 Ad−Hoc Robust Multilingual track − Comparison to Median Mean Average Precision by Topic (Topics 086 to 130)
1
046 10.14 091 16.97 136 34.14 171 8.46 ml6XRSFSen4S
047 13.02 092 22.72 137 12.96 172 5.40 0.5
Difference
048 5.16 093 48.78 138 33.96 173 68.27 0
049 6.88 094 13.19 139 1.60 174 78.03 −0.5
060 14.96 095 7.51 140 19.80 175 43.48 −1
086 087 088 089 090 091 092 093 094 095 096 097 098 099 110 111 112 113 114 115 116 117 118 119 130
061 8.23 096 11.82 141 71.23 176 82.08 Topic Identifier
Ad−Hoc Robust Multilingual track − Comparison to Median Mean Average Precision by Topic (Topics 131 to 165)
062 28.96 097 10.47 142 56.69 177 15.87 1
ml6XRSFSen4S
063 47.37 098 44.12 143 27.25 178 3.09 0.5
Difference
064 41.83 099 8.05 144 19.69 179 26.71
0
065 3.72 110 51.72 145 48.37 190 16.22
−0.5
066 48.85 111 9.24 146 34.15 191 2.04
−1
067 7.28 112 20.02 147 5.97 192 16.21 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 160 161 162 163 164 165
Topic Identifier
068 11.08 113 0.14 148 4.77 193 25.96 Ad−Hoc Robust Multilingual track − Comparison to Median Mean Average Precision by Topic (Topics 166 to 200)
1
069 13.86 114 27.02 149 68.11 194 4.59 ml6XRSFSen4S
0.5
080 21.86 115 14.63 160 6.57 195 7.92
Difference
081 48.06 116 35.79 161 0.09 196 56.19 0
082 56.64 117 3.66 162 29.21 197 43.86 −0.5
083 16.26 118 2.65 163 49.62 198 22.37 −1
166 167 168 169 170 171 172 173 174 175 176 177 178 179 190 191 192 193 194 195 196 197 198 199 200
084 31.86 119 52.78 164 0.66 199 72.55 Topic Identifier
085 21.57 130 42.80 165 10.83 200 31.19
245
daedalus ml6XRSFSen4S AH-ROBUST-MULTI-CLEF2006
Docs Cutoff Levels Precision at DCL (%)
Ad−Hoc Robust Multilingual track − Retrieved documents vs Precision
100%
5 docs 57.60
ml6XRSFSen4S
10 docs 53.10 90%
15 docs 50.67
80%
20 docs 48.40
30 docs 43.50 70%
100 docs 30.08
60%
200 docs 22.49
R−Precision
500 docs 14.17 50%
1000 docs 9.44 40%
R-Precision (precision after R document retrieved,
where R = Relevant retrieved) 30%
30.76
20%
10%
0%
5 10 15 20 30 100 200 500 1000
Retrieved Documents (logarithmic scale)
Exact R-Precision Ad−Hoc Robust Multilingual track − Box plot of the Topics of the Experiment
Maximum 0.7959
Minimum 0.0000
First Quartile 0.1558
Second Quartile 0.2792
Third Quartile 0.4634
Interquartile range 0.3076
Mean 0.3076
Standard Deviation 0.1970
Lower Outlier Threshold 0.0000
Upper Outlier Threshold 0.7959 0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Exact R−Precision
Mean With No Outliers 0.3076
Std With No Outliers 0.1970
Ad−Hoc Robust Multilingual track − Distribution of the Topics of the Experiment
15
Number of Topics of the Experiment
10
5
0
0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Exact R−Precision
Precision averages (%) for individual Ad−Hoc Robust Multilingual track − Comparison to Median Mean Exact R−Precision by Topic (Topics 041 to 085)
queries 1
ml6XRSFSen4S
0.5
041 21.43 086 24.32 131 30.87 166 0.00
Difference
0
042 22.91 087 41.51 132 50.00 167 44.00
−0.5
043 52.78 088 36.21 133 7.91 168 27.63
044 4.62 089 46.30 134 62.96 169 0.00 −1
041 042 043 044 045 046 047 048 049 060 061 062 063 064 065 066 067 068 069 080 081 082 083 084 085
Topic Identifier
045 47.49 090 21.35 135 12.97 170 47.62 Ad−Hoc Robust Multilingual track − Comparison to Median Mean Exact R−Precision by Topic (Topics 086 to 130)
1
046 17.86 091 30.38 136 40.00 171 13.56 ml6XRSFSen4S
047 19.58 092 29.00 137 20.00 172 10.71 0.5
Difference
048 14.76 093 59.52 138 36.97 173 68.00 0
049 14.15 094 22.31 139 10.90 174 72.41 −0.5
060 28.21 095 23.07 140 39.36 175 45.45 −1
086 087 088 089 090 091 092 093 094 095 096 097 098 099 110 111 112 113 114 115 116 117 118 119 130
061 13.13 096 19.88 141 70.59 176 79.59 Topic Identifier
Ad−Hoc Robust Multilingual track − Comparison to Median Mean Exact R−Precision by Topic (Topics 131 to 165)
062 33.33 097 14.71 142 59.88 177 22.58 1
ml6XRSFSen4S
063 46.38 098 55.17 143 37.43 178 11.81 0.5
Difference
064 50.00 099 17.19 144 20.00 179 38.44
0
065 7.17 110 55.67 145 51.61 190 25.95
−0.5
066 46.57 111 17.11 146 33.33 191 0.00
−1
067 18.56 112 23.19 147 6.48 192 20.57 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 160 161 162 163 164 165
Topic Identifier
068 18.12 113 0.70 148 8.00 193 29.50 Ad−Hoc Robust Multilingual track − Comparison to Median Mean Exact R−Precision by Topic (Topics 166 to 200)
1
069 21.74 114 30.46 149 64.10 194 8.33 ml6XRSFSen4S
0.5
080 41.18 115 24.60 160 0.00 195 16.39
Difference
081 57.41 116 40.82 161 0.00 196 60.61 0
082 57.46 117 5.16 162 32.30 197 54.66 −0.5
083 20.59 118 6.52 163 54.66 198 21.21 −1
166 167 168 169 170 171 172 173 174 175 176 177 178 179 190 191 192 193 194 195 196 197 198 199 200
084 37.21 119 53.04 164 6.16 199 68.03 Topic Identifier
085 32.04 130 45.11 165 10.53 200 33.99
246
daedalus mlRSFSen2S AH-ROBUST-MULTI-CLEF2006
Overall statistics for 100 queries : Priority 4
Total number of documents over all queries Query Construction AUTOMATIC
Retrieved 100,000 Source Language English
Relevant 17,019 Topic Fields title, description
Relevant retrieved 8,174 Pooled false
Geometric Mean Average Precision 0.1104 Mandatory TD run
Binary Preference (BPREF) 0.3079
Interploated Recall (%) Precision Averages (%)
Ad−Hoc Robust Multilingual track − Interpolated Recall vs Average Precision
100%
0 75.33
mlRSFSen2S
10 50.07 90%
20 39.08
80%
30 32.38
40 27.24 70%
50 21.13
Average Precision
60%
60 15.17
70 10.10 50%
80 4.67 40%
90 2.22
30%
100 0.31
Average precision (non-interpolated) for all 20%
relevant documents (averaged over queries)
22.67 10%
0%
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
Interpolated Recall
Mean Average Precision Ad−Hoc Robust Multilingual track − Box plot of the Topics of the Experiment
Maximum 0.7986
Minimum 0.0000
First Quartile 0.0660
Second Quartile 0.1677
Third Quartile 0.3618
Interquartile range 0.2958
Mean 0.2267
Standard Deviation 0.1964
Lower Outlier Threshold 0.0000
Upper Outlier Threshold 0.7986 0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Mean Average Precision
Mean With No Outliers 0.2267
Std With No Outliers 0.1964
Ad−Hoc Robust Multilingual track − Distribution of the Topics of the Experiment
20
Number of Topics of the Experiment
15
10
5
0
0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Mean Average Precision
Precision averages (%) for individual Ad−Hoc Robust Multilingual track − Comparison to Median Mean Average Precision by Topic (Topics 041 to 085)
queries 1
mlRSFSen2S
0.5
041 12.28 086 7.19 131 14.69 166 0.17
Difference
0
042 0.00 087 39.77 132 56.43 167 39.48
−0.5
043 41.69 088 30.90 133 1.69 168 10.63
044 7.01 089 40.11 134 59.24 169 0.61 −1
041 042 043 044 045 046 047 048 049 060 061 062 063 064 065 066 067 068 069 080 081 082 083 084 085
Topic Identifier
045 28.01 090 13.70 135 7.10 170 32.54 Ad−Hoc Robust Multilingual track − Comparison to Median Mean Average Precision by Topic (Topics 086 to 130)
1
046 10.64 091 20.80 136 32.88 171 7.28 mlRSFSen2S
047 9.62 092 21.44 137 10.27 172 4.79 0.5
Difference
048 2.48 093 48.99 138 12.50 173 18.80 0
049 6.48 094 16.62 139 1.35 174 61.87 −0.5
060 16.57 095 6.73 140 20.36 175 54.72 −1
086 087 088 089 090 091 092 093 094 095 096 097 098 099 110 111 112 113 114 115 116 117 118 119 130
061 4.71 096 11.36 141 70.09 176 79.86 Topic Identifier
Ad−Hoc Robust Multilingual track − Comparison to Median Mean Average Precision by Topic (Topics 131 to 165)
062 27.90 097 6.28 142 56.87 177 16.92 1
mlRSFSen2S
063 53.47 098 40.98 143 28.10 178 3.05 0.5
Difference
064 39.58 099 6.10 144 18.38 179 26.35
0
065 2.61 110 53.38 145 46.85 190 10.72
−0.5
066 43.35 111 6.99 146 24.52 191 4.17
−1
067 5.32 112 27.09 147 1.73 192 7.03 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 160 161 162 163 164 165
Topic Identifier
068 3.64 113 0.02 148 2.83 193 24.31 Ad−Hoc Robust Multilingual track − Comparison to Median Mean Average Precision by Topic (Topics 166 to 200)
1
069 10.34 114 12.26 149 49.92 194 6.37 mlRSFSen2S
0.5
080 20.59 115 14.52 160 25.89 195 3.94
Difference
081 47.94 116 32.63 161 0.00 196 53.90 0
082 44.80 117 2.42 162 22.55 197 26.32 −0.5
083 15.77 118 2.25 163 48.98 198 14.41 −1
166 167 168 169 170 171 172 173 174 175 176 177 178 179 190 191 192 193 194 195 196 197 198 199 200
084 21.70 119 55.12 164 0.84 199 69.98 Topic Identifier
085 20.83 130 30.14 165 8.52 200 20.83
247
daedalus mlRSFSen2S AH-ROBUST-MULTI-CLEF2006
Docs Cutoff Levels Precision at DCL (%)
Ad−Hoc Robust Multilingual track − Retrieved documents vs Precision
100%
5 docs 52.00
mlRSFSen2S
10 docs 49.70 90%
15 docs 46.47
80%
20 docs 44.30
30 docs 39.93 70%
100 docs 28.37
60%
200 docs 21.39
R−Precision
500 docs 13.04 50%
1000 docs 8.17 40%
R-Precision (precision after R document retrieved,
where R = Relevant retrieved) 30%
28.75
20%
10%
0%
5 10 15 20 30 100 200 500 1000
Retrieved Documents (logarithmic scale)
Exact R-Precision Ad−Hoc Robust Multilingual track − Box plot of the Topics of the Experiment
Maximum 0.7796
Minimum 0.0000
First Quartile 0.1450
Second Quartile 0.2369
Third Quartile 0.4248
Interquartile range 0.2798
Mean 0.2875
Standard Deviation 0.1901
Lower Outlier Threshold 0.0000
Upper Outlier Threshold 0.7796 0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Exact R−Precision
Mean With No Outliers 0.2875
Std With No Outliers 0.1901
Ad−Hoc Robust Multilingual track − Distribution of the Topics of the Experiment
15
Number of Topics of the Experiment
10
5
0
0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Exact R−Precision
Precision averages (%) for individual Ad−Hoc Robust Multilingual track − Comparison to Median Mean Exact R−Precision by Topic (Topics 041 to 085)
queries 1
mlRSFSen2S
0.5
041 19.05 086 17.49 131 28.23 166 0.00
Difference
0
042 0.14 087 44.65 132 56.76 167 44.00
−0.5
043 47.22 088 37.07 133 6.05 168 21.05
044 7.69 089 44.36 134 59.26 169 5.26 −1
041 042 043 044 045 046 047 048 049 060 061 062 063 064 065 066 067 068 069 080 081 082 083 084 085
Topic Identifier
045 40.96 090 20.22 135 22.16 170 33.33 Ad−Hoc Robust Multilingual track − Comparison to Median Mean Exact R−Precision by Topic (Topics 086 to 130)
1
046 18.75 091 29.11 136 40.00 171 11.86 mlRSFSen2S
047 16.91 092 31.23 137 20.00 172 14.29 0.5
Difference
048 9.59 093 59.52 138 16.81 173 16.00 0
049 16.98 094 24.79 139 10.58 174 67.82 −0.5
060 29.78 095 21.12 140 37.84 175 59.09 −1
086 087 088 089 090 091 092 093 094 095 096 097 098 099 110 111 112 113 114 115 116 117 118 119 130
061 8.08 096 19.25 141 70.59 176 77.96 Topic Identifier
Ad−Hoc Robust Multilingual track − Comparison to Median Mean Exact R−Precision by Topic (Topics 131 to 165)
062 30.86 097 14.71 142 59.88 177 22.58 1
mlRSFSen2S
063 55.80 098 51.72 143 39.11 178 7.87 0.5
Difference
064 45.00 099 14.06 144 16.67 179 35.37
0
065 6.37 110 57.73 145 50.00 190 19.85
−0.5
066 48.01 111 17.11 146 33.33 191 0.00
−1
067 15.98 112 31.88 147 6.48 192 12.06 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 160 161 162 163 164 165
Topic Identifier
068 8.70 113 0.35 148 9.33 193 33.09 Ad−Hoc Robust Multilingual track − Comparison to Median Mean Exact R−Precision by Topic (Topics 166 to 200)
1
069 21.74 114 19.54 149 53.85 194 12.50 mlRSFSen2S
0.5
080 37.20 115 25.40 160 20.00 195 9.84
Difference
081 55.56 116 37.76 161 0.00 196 60.61 0
082 44.03 117 8.39 162 28.32 197 36.27 −0.5
083 17.65 118 4.35 163 52.63 198 21.21 −1
166 167 168 169 170 171 172 173 174 175 176 177 178 179 190 191 192 193 194 195 196 197 198 199 200
084 32.56 119 57.48 164 6.16 199 68.03 Topic Identifier
085 31.14 130 38.35 165 15.79 200 31.53
248
jaen ujamlblr AH-ROBUST-MULTI-CLEF2006
Overall statistics for 100 queries : Priority 3
Total number of documents over all queries Query Construction AUTOMATIC
Retrieved 100,000 Source Language English
Relevant 17,019 Topic Fields title, description
Relevant retrieved 9,442 Pooled false
Geometric Mean Average Precision 0.1496 bayesian logistic regression used
Binary Preference (BPREF) 0.3558
Interploated Recall (%) Precision Averages (%)
Ad−Hoc Robust Multilingual track − Interpolated Recall vs Average Precision
100%
0 78.05
ujamlblr
10 53.15 90%
20 45.59
80%
30 38.04
40 30.85 70%
50 25.07
Average Precision
60%
60 19.80
70 14.10 50%
80 9.81 40%
90 5.23
30%
100 1.31
Average precision (non-interpolated) for all 20%
relevant documents (averaged over queries)
27.01 10%
0%
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
Interpolated Recall
Mean Average Precision Ad−Hoc Robust Multilingual track − Box plot of the Topics of the Experiment
Maximum 1.0000
Minimum 0.0003
First Quartile 0.1019
Second Quartile 0.1831
Third Quartile 0.3981
Interquartile range 0.2962
Mean 0.2701
Standard Deviation 0.2332
Lower Outlier Threshold 0.0003
Upper Outlier Threshold 0.7789 0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Mean Average Precision
Mean With No Outliers 0.2502
Std With No Outliers 0.2065
Ad−Hoc Robust Multilingual track − Distribution of the Topics of the Experiment
20
Number of Topics of the Experiment
15
10
5
0
0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Mean Average Precision
Precision averages (%) for individual Ad−Hoc Robust Multilingual track − Comparison to Median Mean Average Precision by Topic (Topics 041 to 085)
queries 1
ujamlblr
0.5
041 14.65 086 12.37 131 14.56 166 0.18
Difference
0
042 31.77 087 29.42 132 57.19 167 36.17
−0.5
043 56.54 088 17.16 133 0.59 168 10.89
044 10.70 089 37.57 134 77.89 169 1.35 −1
041 042 043 044 045 046 047 048 049 060 061 062 063 064 065 066 067 068 069 080 081 082 083 084 085
Topic Identifier
045 35.28 090 14.75 135 20.98 170 17.60 Ad−Hoc Robust Multilingual track − Comparison to Median Mean Average Precision by Topic (Topics 086 to 130)
1
046 2.18 091 8.59 136 42.46 171 6.86 ujamlblr
047 7.24 092 17.44 137 4.69 172 7.82 0.5
Difference
048 4.77 093 10.97 138 3.11 173 71.94 0
049 4.73 094 17.43 139 9.98 174 47.68 −0.5
060 19.09 095 8.66 140 13.76 175 33.44 −1
086 087 088 089 090 091 092 093 094 095 096 097 098 099 110 111 112 113 114 115 116 117 118 119 130
061 14.64 096 10.60 141 74.54 176 87.45 Topic Identifier
Ad−Hoc Robust Multilingual track − Comparison to Median Mean Average Precision by Topic (Topics 131 to 165)
062 24.52 097 17.83 142 86.47 177 17.86 1
ujamlblr
063 50.95 098 43.82 143 24.65 178 0.33 0.5
Difference
064 33.00 099 3.95 144 52.43 179 14.33
0
065 3.01 110 53.04 145 39.20 190 10.98
−0.5
066 74.01 111 10.40 146 17.10 191 100.00
−1
067 16.25 112 33.47 147 43.35 192 29.05 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 160 161 162 163 164 165
Topic Identifier
068 12.43 113 0.06 148 4.18 193 25.19 Ad−Hoc Robust Multilingual track − Comparison to Median Mean Average Precision by Topic (Topics 166 to 200)
1
069 3.27 114 10.59 149 72.55 194 23.85 ujamlblr
0.5
080 20.82 115 13.27 160 26.88 195 16.39
Difference
081 40.83 116 38.16 161 0.03 196 54.29 0
082 40.41 117 8.61 162 43.56 197 53.26 −0.5
083 25.39 118 1.15 163 70.21 198 6.23 −1
166 167 168 169 170 171 172 173 174 175 176 177 178 179 190 191 192 193 194 195 196 197 198 199 200
084 30.46 119 51.76 164 1.21 199 72.53 Topic Identifier
085 18.76 130 34.39 165 23.63 200 28.99
249
jaen ujamlblr AH-ROBUST-MULTI-CLEF2006
Docs Cutoff Levels Precision at DCL (%)
Ad−Hoc Robust Multilingual track − Retrieved documents vs Precision
100%
5 docs 56.00
ujamlblr
10 docs 53.10 90%
15 docs 49.47
80%
20 docs 47.35
30 docs 43.27 70%
100 docs 30.49
60%
200 docs 23.25
R−Precision
500 docs 14.61 50%
1000 docs 9.44 40%
R-Precision (precision after R document retrieved,
where R = Relevant retrieved) 30%
32.34
20%
10%
0%
5 10 15 20 30 100 200 500 1000
Retrieved Documents (logarithmic scale)
Exact R-Precision Ad−Hoc Robust Multilingual track − Box plot of the Topics of the Experiment
Maximum 1.0000
Minimum 0.0000
First Quartile 0.1683
Second Quartile 0.2910
Third Quartile 0.4441
Interquartile range 0.2758
Mean 0.3234
Standard Deviation 0.2089
Lower Outlier Threshold 0.0000
Upper Outlier Threshold 0.8081 0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Exact R−Precision
Mean With No Outliers 0.3166
Std With No Outliers 0.1984
Ad−Hoc Robust Multilingual track − Distribution of the Topics of the Experiment
15
Number of Topics of the Experiment
10
5
0
0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Exact R−Precision
Precision averages (%) for individual Ad−Hoc Robust Multilingual track − Comparison to Median Mean Exact R−Precision by Topic (Topics 041 to 085)
queries 1
ujamlblr
0.5
041 20.24 086 22.68 131 26.12 166 0.00
Difference
0
042 43.14 087 36.48 132 56.76 167 40.00
−0.5
043 55.56 088 31.03 133 4.65 168 21.05
044 15.38 089 47.47 134 70.37 169 5.26 −1
041 042 043 044 045 046 047 048 049 060 061 062 063 064 065 066 067 068 069 080 081 082 083 084 085
Topic Identifier
045 45.53 090 21.35 135 32.53 170 33.33 Ad−Hoc Robust Multilingual track − Comparison to Median Mean Exact R−Precision by Topic (Topics 086 to 130)
1
046 4.46 091 11.39 136 42.86 171 8.47 ujamlblr
047 18.69 092 27.14 137 12.00 172 17.86 0.5
Difference
048 13.28 093 16.67 138 10.08 173 64.00 0
049 10.38 094 22.31 139 16.99 174 47.13 −0.5
060 30.09 095 24.74 140 27.03 175 31.82 −1
086 087 088 089 090 091 092 093 094 095 096 097 098 099 110 111 112 113 114 115 116 117 118 119 130
061 19.19 096 20.50 141 79.41 176 78.78 Topic Identifier
Ad−Hoc Robust Multilingual track − Comparison to Median Mean Exact R−Precision by Topic (Topics 131 to 165)
062 34.57 097 30.88 142 80.81 177 22.58 1
ujamlblr
063 50.72 098 55.17 143 34.36 178 3.94 0.5
Difference
064 40.00 099 10.94 144 53.33 179 19.05
0
065 10.76 110 63.92 145 40.32 190 21.37
−0.5
066 67.87 111 14.47 146 33.33 191 100.00
−1
067 28.35 112 39.13 147 47.22 192 37.59 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 160 161 162 163 164 165
Topic Identifier
068 23.91 113 1.41 148 9.33 193 28.06 Ad−Hoc Robust Multilingual track − Comparison to Median Mean Exact R−Precision by Topic (Topics 166 to 200)
1
069 6.52 114 15.52 149 66.67 194 29.17 ujamlblr
0.5
080 37.36 115 24.60 160 20.00 195 21.31
Difference
081 48.61 116 38.78 161 0.00 196 57.58 0
082 43.28 117 15.48 162 50.44 197 59.97 −0.5
083 23.53 118 6.52 163 65.18 198 9.09 −1
166 167 168 169 170 171 172 173 174 175 176 177 178 179 190 191 192 193 194 195 196 197 198 199 200
084 34.88 119 54.21 164 7.04 199 70.49 Topic Identifier
085 29.04 130 42.11 165 31.58 200 37.44
250
jaen ujamlrsv2 AH-ROBUST-MULTI-CLEF2006
Overall statistics for 100 queries : Priority 1
Total number of documents over all queries Query Construction AUTOMATIC
Retrieved 100,000 Source Language English
Relevant 17,019 Topic Fields title, description
Relevant retrieved 9,443 Pooled false
Geometric Mean Average Precision 0.1569 automatic, TD, no training. Only mixed 2-step RSV
Binary Preference (BPREF) 0.3619 merging
Interploated Recall (%) Precision Averages (%)
Ad−Hoc Robust Multilingual track − Interpolated Recall vs Average Precision
100%
0 79.95
ujamlrsv2
10 55.42 90%
20 46.84
80%
30 38.12
40 32.13 70%
50 25.80
Average Precision
60%
60 20.01
70 14.59 50%
80 10.32 40%
90 5.66
30%
100 1.28
Average precision (non-interpolated) for all 20%
relevant documents (averaged over queries)
27.85 10%
0%
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
Interpolated Recall
Mean Average Precision Ad−Hoc Robust Multilingual track − Box plot of the Topics of the Experiment
Maximum 1.0000
Minimum 0.0003
First Quartile 0.1090
Second Quartile 0.2040
Third Quartile 0.4008
Interquartile range 0.2918
Mean 0.2785
Standard Deviation 0.2323
Lower Outlier Threshold 0.0003
Upper Outlier Threshold 0.8033 0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Mean Average Precision
Mean With No Outliers 0.2584
Std With No Outliers 0.2049
Ad−Hoc Robust Multilingual track − Distribution of the Topics of the Experiment
20
Number of Topics of the Experiment
15
10
5
0
0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Mean Average Precision
Precision averages (%) for individual Ad−Hoc Robust Multilingual track − Comparison to Median Mean Average Precision by Topic (Topics 041 to 085)
queries 1
ujamlrsv2
0.5
041 16.40 086 11.65 131 14.66 166 0.17
Difference
0
042 30.01 087 30.63 132 61.22 167 36.96
−0.5
043 56.67 088 18.56 133 0.50 168 13.06
044 12.37 089 41.78 134 75.25 169 1.26 −1
041 042 043 044 045 046 047 048 049 060 061 062 063 064 065 066 067 068 069 080 081 082 083 084 085
Topic Identifier
045 34.83 090 19.34 135 21.08 170 34.11 Ad−Hoc Robust Multilingual track − Comparison to Median Mean Average Precision by Topic (Topics 086 to 130)
1
046 2.34 091 8.66 136 45.35 171 7.04 ujamlrsv2
047 7.89 092 17.97 137 3.86 172 11.64 0.5
Difference
048 5.59 093 12.34 138 1.50 173 69.24 0
049 3.97 094 15.50 139 9.79 174 50.24 −0.5
060 18.70 095 8.95 140 18.88 175 28.56 −1
086 087 088 089 090 091 092 093 094 095 096 097 098 099 110 111 112 113 114 115 116 117 118 119 130
061 17.98 096 11.09 141 80.33 176 86.57 Topic Identifier
Ad−Hoc Robust Multilingual track − Comparison to Median Mean Average Precision by Topic (Topics 131 to 165)
062 27.89 097 18.19 142 91.82 177 20.64 1
ujamlrsv2
063 51.18 098 46.33 143 24.37 178 0.60 0.5
Difference
064 34.95 099 3.32 144 48.11 179 14.63
0
065 3.46 110 56.19 145 38.76 190 10.72
−0.5
066 75.34 111 11.08 146 43.00 191 100.00
−1
067 16.46 112 35.18 147 40.17 192 29.50 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 160 161 162 163 164 165
Topic Identifier
068 12.55 113 0.07 148 4.72 193 22.46 Ad−Hoc Robust Multilingual track − Comparison to Median Mean Average Precision by Topic (Topics 166 to 200)
1
069 3.19 114 13.78 149 67.15 194 21.50 ujamlrsv2
0.5
080 20.15 115 12.53 160 24.46 195 19.27
Difference
081 40.00 116 37.72 161 0.03 196 57.96 0
082 39.71 117 9.58 162 48.46 197 53.42 −0.5
083 27.50 118 2.27 163 67.13 198 10.41 −1
166 167 168 169 170 171 172 173 174 175 176 177 178 179 190 191 192 193 194 195 196 197 198 199 200
084 29.78 119 46.40 164 1.15 199 70.31 Topic Identifier
085 18.94 130 32.68 165 26.46 200 26.46
251
jaen ujamlrsv2 AH-ROBUST-MULTI-CLEF2006
Docs Cutoff Levels Precision at DCL (%)
Ad−Hoc Robust Multilingual track − Retrieved documents vs Precision
100%
5 docs 59.20
ujamlrsv2
10 docs 54.20 90%
15 docs 51.40
80%
20 docs 48.45
30 docs 44.37 70%
100 docs 30.88
60%
200 docs 23.68
R−Precision
500 docs 14.71 50%
1000 docs 9.44 40%
R-Precision (precision after R document retrieved,
where R = Relevant retrieved) 30%
32.70
20%
10%
0%
5 10 15 20 30 100 200 500 1000
Retrieved Documents (logarithmic scale)
Exact R-Precision Ad−Hoc Robust Multilingual track − Box plot of the Topics of the Experiment
Maximum 1.0000
Minimum 0.0000
First Quartile 0.1784
Second Quartile 0.3049
Third Quartile 0.4501
Interquartile range 0.2717
Mean 0.3270
Standard Deviation 0.2083
Lower Outlier Threshold 0.0000
Upper Outlier Threshold 0.8488 0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Exact R−Precision
Mean With No Outliers 0.3202
Std With No Outliers 0.1978
Ad−Hoc Robust Multilingual track − Distribution of the Topics of the Experiment
15
Number of Topics of the Experiment
10
5
0
0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Exact R−Precision
Precision averages (%) for individual Ad−Hoc Robust Multilingual track − Comparison to Median Mean Exact R−Precision by Topic (Topics 041 to 085)
queries 1
ujamlrsv2
0.5
041 20.24 086 22.95 131 26.12 166 0.00
Difference
0
042 41.02 087 35.22 132 55.41 167 32.00
−0.5
043 52.78 088 32.76 133 4.19 168 22.37
044 16.92 089 50.58 134 70.37 169 0.00 −1
041 042 043 044 045 046 047 048 049 060 061 062 063 064 065 066 067 068 069 080 081 082 083 084 085
Topic Identifier
045 43.36 090 22.47 135 32.93 170 42.86 Ad−Hoc Robust Multilingual track − Comparison to Median Mean Exact R−Precision by Topic (Topics 086 to 130)
1
046 4.46 091 11.39 136 42.86 171 10.17 ujamlrsv2
047 18.40 092 29.00 137 12.00 172 17.86 0.5
Difference
048 15.50 093 20.24 138 7.56 173 68.00 0
049 9.43 094 23.14 139 14.74 174 50.57 −0.5
060 30.09 095 24.37 140 33.61 175 31.82 −1
086 087 088 089 090 091 092 093 094 095 096 097 098 099 110 111 112 113 114 115 116 117 118 119 130
061 26.26 096 20.50 141 79.41 176 77.96 Topic Identifier
Ad−Hoc Robust Multilingual track − Comparison to Median Mean Exact R−Precision by Topic (Topics 131 to 165)
062 33.33 097 30.88 142 84.88 177 20.97 1
ujamlrsv2
063 50.00 098 53.45 143 36.87 178 3.15 0.5
Difference
064 40.00 099 10.42 144 46.67 179 21.09
0
065 11.55 110 62.89 145 39.52 190 22.14
−0.5
066 70.40 111 15.79 146 50.00 191 100.00
−1
067 27.32 112 36.23 147 50.00 192 39.01 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 160 161 162 163 164 165
Topic Identifier
068 20.29 113 2.11 148 10.67 193 27.34 Ad−Hoc Robust Multilingual track − Comparison to Median Mean Exact R−Precision by Topic (Topics 166 to 200)
1
069 6.52 114 17.82 149 61.54 194 25.00 ujamlrsv2
0.5
080 36.72 115 22.22 160 20.00 195 22.95
Difference
081 48.15 116 40.82 161 0.00 196 60.61 0
082 41.04 117 15.48 162 53.54 197 60.62 −0.5
083 32.35 118 8.70 163 62.35 198 12.12 −1
166 167 168 169 170 171 172 173 174 175 176 177 178 179 190 191 192 193 194 195 196 197 198 199 200
084 34.88 119 52.10 164 7.04 199 65.98 Topic Identifier
085 28.74 130 41.35 165 36.84 200 33.50
252
jaen ujamllr AH-ROBUST-MULTI-CLEF2006
Overall statistics for 100 queries : Priority 2
Total number of documents over all queries Query Construction AUTOMATIC
Retrieved 100,000 Source Language English
Relevant 17,019 Topic Fields title, description
Relevant retrieved 9,326 Pooled false
Geometric Mean Average Precision 0.1486 merging with logistic regression
Binary Preference (BPREF) 0.3567
Interploated Recall (%) Precision Averages (%)
Ad−Hoc Robust Multilingual track − Interpolated Recall vs Average Precision
100%
0 78.84
ujamllr
10 53.36 90%
20 45.52
80%
30 37.97
40 30.70 70%
50 24.71
Average Precision
60%
60 19.37
70 14.86 50%
80 9.62 40%
90 4.38
30%
100 1.05
Average precision (non-interpolated) for all 20%
relevant documents (averaged over queries)
27.11 10%
0%
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
Interpolated Recall
Mean Average Precision Ad−Hoc Robust Multilingual track − Box plot of the Topics of the Experiment
Maximum 1.0000
Minimum 0.0003
First Quartile 0.0969
Second Quartile 0.1918
Third Quartile 0.3930
Interquartile range 0.2961
Mean 0.2711
Standard Deviation 0.2338
Lower Outlier Threshold 0.0003
Upper Outlier Threshold 0.7935 0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Mean Average Precision
Mean With No Outliers 0.2515
Std With No Outliers 0.2079
Ad−Hoc Robust Multilingual track − Distribution of the Topics of the Experiment
20
Number of Topics of the Experiment
15
10
5
0
0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Mean Average Precision
Precision averages (%) for individual Ad−Hoc Robust Multilingual track − Comparison to Median Mean Average Precision by Topic (Topics 041 to 085)
queries 1
ujamllr
0.5
041 15.48 086 13.83 131 14.69 166 0.20
Difference
0
042 35.91 087 29.84 132 56.03 167 36.85
−0.5
043 57.46 088 19.50 133 0.63 168 9.99
044 10.56 089 36.07 134 79.35 169 1.46 −1
041 042 043 044 045 046 047 048 049 060 061 062 063 064 065 066 067 068 069 080 081 082 083 084 085
Topic Identifier
045 37.01 090 16.04 135 20.51 170 18.17 Ad−Hoc Robust Multilingual track − Comparison to Median Mean Average Precision by Topic (Topics 086 to 130)
1
046 1.76 091 8.25 136 43.47 171 7.06 ujamllr
047 7.67 092 15.70 137 4.79 172 11.95 0.5
Difference
048 3.64 093 10.57 138 1.86 173 73.48 0
049 4.30 094 15.98 139 8.55 174 46.97 −0.5
060 18.59 095 8.28 140 11.02 175 32.61 −1
086 087 088 089 090 091 092 093 094 095 096 097 098 099 110 111 112 113 114 115 116 117 118 119 130
061 15.11 096 9.86 141 78.19 176 87.73 Topic Identifier
Ad−Hoc Robust Multilingual track − Comparison to Median Mean Average Precision by Topic (Topics 131 to 165)
062 24.27 097 17.25 142 84.47 177 20.51 1
ujamllr
063 50.16 098 45.43 143 22.32 178 0.38 0.5
Difference
064 30.85 099 4.76 144 49.23 179 13.43
0
065 3.19 110 55.68 145 37.88 190 11.10
−0.5
066 71.97 111 9.52 146 32.65 191 100.00
−1
067 16.83 112 34.34 147 43.32 192 27.49 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 160 161 162 163 164 165
Topic Identifier
068 13.54 113 0.03 148 3.55 193 26.15 Ad−Hoc Robust Multilingual track − Comparison to Median Mean Average Precision by Topic (Topics 166 to 200)
1
069 2.63 114 10.90 149 72.62 194 23.57 ujamllr
0.5
080 21.82 115 12.37 160 15.62 195 17.30
Difference
081 40.72 116 33.46 161 0.03 196 52.68 0
082 43.06 117 9.01 162 42.93 197 53.89 −0.5
083 28.30 118 1.17 163 69.69 198 8.76 −1
166 167 168 169 170 171 172 173 174 175 176 177 178 179 190 191 192 193 194 195 196 197 198 199 200
084 28.08 119 54.71 164 1.18 199 72.01 Topic Identifier
085 18.86 130 33.26 165 26.07 200 29.40
253
jaen ujamllr AH-ROBUST-MULTI-CLEF2006
Docs Cutoff Levels Precision at DCL (%)
Ad−Hoc Robust Multilingual track − Retrieved documents vs Precision
100%
5 docs 59.00
ujamllr
10 docs 54.20 90%
15 docs 50.47
80%
20 docs 47.30
30 docs 43.87 70%
100 docs 30.51
60%
200 docs 23.26
R−Precision
500 docs 14.52 50%
1000 docs 9.33 40%
R-Precision (precision after R document retrieved,
where R = Relevant retrieved) 30%
32.42
20%
10%
0%
5 10 15 20 30 100 200 500 1000
Retrieved Documents (logarithmic scale)
Exact R-Precision Ad−Hoc Robust Multilingual track − Box plot of the Topics of the Experiment
Maximum 1.0000
Minimum 0.0000
First Quartile 0.1667
Second Quartile 0.2910
Third Quartile 0.4596
Interquartile range 0.2930
Mean 0.3242
Standard Deviation 0.2112
Lower Outlier Threshold 0.0000
Upper Outlier Threshold 0.8023 0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Exact R−Precision
Mean With No Outliers 0.3174
Std With No Outliers 0.2009
Ad−Hoc Robust Multilingual track − Distribution of the Topics of the Experiment
15
Number of Topics of the Experiment
10
5
0
0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Exact R−Precision
Precision averages (%) for individual Ad−Hoc Robust Multilingual track − Comparison to Median Mean Exact R−Precision by Topic (Topics 041 to 085)
queries 1
ujamllr
0.5
041 20.24 086 24.04 131 25.86 166 0.00
Difference
0
042 46.39 087 36.48 132 58.11 167 32.00
−0.5
043 52.78 088 31.90 133 5.58 168 21.71
044 15.38 089 46.69 134 70.37 169 0.00 −1
041 042 043 044 045 046 047 048 049 060 061 062 063 064 065 066 067 068 069 080 081 082 083 084 085
Topic Identifier
045 45.53 090 22.47 135 32.53 170 33.33 Ad−Hoc Robust Multilingual track − Comparison to Median Mean Exact R−Precision by Topic (Topics 086 to 130)
1
046 4.46 091 11.39 136 42.86 171 8.47 ujamllr
047 18.99 092 27.51 137 12.00 172 21.43 0.5
Difference
048 12.92 093 16.67 138 8.40 173 68.00 0
049 9.43 094 23.14 139 15.38 174 48.28 −0.5
060 29.47 095 24.00 140 25.17 175 36.36 −1
086 087 088 089 090 091 092 093 094 095 096 097 098 099 110 111 112 113 114 115 116 117 118 119 130
061 19.19 096 20.50 141 79.41 176 78.37 Topic Identifier
Ad−Hoc Robust Multilingual track − Comparison to Median Mean Exact R−Precision by Topic (Topics 131 to 165)
062 34.57 097 30.88 142 80.23 177 20.97 1
ujamllr
063 50.00 098 55.17 143 34.92 178 3.94 0.5
Difference
064 40.00 099 11.98 144 50.00 179 18.71
0
065 10.36 110 63.92 145 38.71 190 21.37
−0.5
066 66.79 111 15.79 146 33.33 191 100.00
−1
067 28.35 112 39.13 147 48.15 192 34.75 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 160 161 162 163 164 165
Topic Identifier
068 22.46 113 1.41 148 9.33 193 28.06 Ad−Hoc Robust Multilingual track − Comparison to Median Mean Exact R−Precision by Topic (Topics 166 to 200)
1
069 6.52 114 16.67 149 71.79 194 29.17 ujamllr
0.5
080 38.00 115 23.02 160 20.00 195 22.95
Difference
081 50.00 116 38.78 161 0.00 196 60.61 0
082 44.78 117 14.84 162 49.12 197 60.23 −0.5
083 26.47 118 8.70 163 66.80 198 9.09 −1
166 167 168 169 170 171 172 173 174 175 176 177 178 179 190 191 192 193 194 195 196 197 198 199 200
084 32.56 119 58.18 164 7.33 199 70.90 Topic Identifier
085 29.04 130 42.11 165 31.58 200 36.45
254
reina reinaES2mtdtest AH-ROBUST-MULTI-CLEF2006
Overall statistics for 100 queries : Priority 1
Total number of documents over all queries Query Construction AUTOMATIC
Retrieved 100,000 Source Language Spanish; Castilian
Relevant 17,019 Topic Fields title, description
Relevant retrieved 7,862 Pooled false
Geometric Mean Average Precision 0.1325 MT + local expansion with term windows, title+desc
Binary Preference (BPREF) 0.2797 fields
Interploated Recall (%) Precision Averages (%)
Ad−Hoc Robust Multilingual track − Interpolated Recall vs Average Precision
100%
0 80.24
reinaES2mtdtest
10 51.52 90%
20 40.44
80%
30 31.59
40 22.33 70%
50 14.29
Average Precision
60%
60 8.29
70 1.88 50%
80 0.62 40%
90 0.55
30%
100 0.50
Average precision (non-interpolated) for all 20%
relevant documents (averaged over queries)
19.96 10%
0%
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
Interpolated Recall
Mean Average Precision Ad−Hoc Robust Multilingual track − Box plot of the Topics of the Experiment
Maximum 0.5565
Minimum 0.0045
First Quartile 0.0867
Second Quartile 0.1758
Third Quartile 0.2945
Interquartile range 0.2078
Mean 0.1996
Standard Deviation 0.1450
Lower Outlier Threshold 0.0045
Upper Outlier Threshold 0.5565 0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Mean Average Precision
Mean With No Outliers 0.1996
Std With No Outliers 0.1450
Ad−Hoc Robust Multilingual track − Distribution of the Topics of the Experiment
20
Number of Topics of the Experiment
15
10
5
0
0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Mean Average Precision
Precision averages (%) for individual Ad−Hoc Robust Multilingual track − Comparison to Median Mean Average Precision by Topic (Topics 041 to 085)
queries 1
reinaES2mtdtest
0.5
041 15.49 086 12.94 131 21.46 166 0.45
Difference
0
042 18.25 087 45.40 132 4.15 167 30.28
−0.5
043 13.96 088 18.99 133 1.89 168 11.68
044 9.46 089 10.58 134 37.59 169 0.68 −1
041 042 043 044 045 046 047 048 049 060 061 062 063 064 065 066 067 068 069 080 081 082 083 084 085
Topic Identifier
045 26.47 090 6.03 135 32.33 170 15.45 Ad−Hoc Robust Multilingual track − Comparison to Median Mean Average Precision by Topic (Topics 086 to 130)
1
046 3.55 091 4.92 136 20.72 171 9.64 reinaES2mtdtest
047 10.39 092 16.41 137 13.53 172 0.68 0.5
Difference
048 0.91 093 15.00 138 6.04 173 41.73 0
049 5.79 094 17.72 139 3.01 174 27.10 −0.5
060 12.01 095 8.42 140 13.19 175 19.97 −1
086 087 088 089 090 091 092 093 094 095 096 097 098 099 110 111 112 113 114 115 116 117 118 119 130
061 2.29 096 4.66 141 45.94 176 53.62 Topic Identifier
Ad−Hoc Robust Multilingual track − Comparison to Median Mean Average Precision by Topic (Topics 131 to 165)
062 21.41 097 23.53 142 34.03 177 17.45 1
reinaES2mtdtest
063 24.25 098 34.85 143 28.62 178 14.41 0.5
Difference
064 23.68 099 1.58 144 48.98 179 26.97
0
065 5.85 110 38.04 145 46.02 190 0.69
−0.5
066 55.65 111 11.32 146 8.91 191 50.00
−1
067 6.17 112 22.62 147 22.88 192 31.33 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 160 161 162 163 164 165
Topic Identifier
068 1.77 113 2.17 148 2.47 193 14.46 Ad−Hoc Robust Multilingual track − Comparison to Median Mean Average Precision by Topic (Topics 166 to 200)
1
069 26.68 114 10.03 149 22.11 194 16.12 reinaES2mtdtest
0.5
080 18.62 115 17.37 160 23.40 195 15.33
Difference
081 27.26 116 9.41 161 18.35 196 35.03 0
082 30.47 117 10.34 162 39.46 197 48.93 −0.5
083 18.98 118 4.96 163 36.51 198 5.88 −1
166 167 168 169 170 171 172 173 174 175 176 177 178 179 190 191 192 193 194 195 196 197 198 199 200
084 26.01 119 46.57 164 4.48 199 46.89 Topic Identifier
085 40.58 130 35.78 165 31.33 200 18.09
255
reina reinaES2mtdtest AH-ROBUST-MULTI-CLEF2006
Docs Cutoff Levels Precision at DCL (%)
Ad−Hoc Robust Multilingual track − Retrieved documents vs Precision
100%
5 docs 51.40
reinaES2mtdtest
10 docs 47.00 90%
15 docs 45.47
80%
20 docs 43.65
30 docs 40.20 70%
100 docs 28.36
60%
200 docs 21.31
R−Precision
500 docs 12.74 50%
1000 docs 7.86 40%
R-Precision (precision after R document retrieved,
where R = Relevant retrieved) 30%
27.60
20%
10%
0%
5 10 15 20 30 100 200 500 1000
Retrieved Documents (logarithmic scale)
Exact R-Precision Ad−Hoc Robust Multilingual track − Box plot of the Topics of the Experiment
Maximum 0.6082
Minimum 0.0000
First Quartile 0.1500
Second Quartile 0.2833
Third Quartile 0.3652
Interquartile range 0.2153
Mean 0.2760
Standard Deviation 0.1533
Lower Outlier Threshold 0.0000
Upper Outlier Threshold 0.6082 0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Exact R−Precision
Mean With No Outliers 0.2760
Std With No Outliers 0.1533
Ad−Hoc Robust Multilingual track − Distribution of the Topics of the Experiment
15
Number of Topics of the Experiment
10
5
0
0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Exact R−Precision
Precision averages (%) for individual Ad−Hoc Robust Multilingual track − Comparison to Median Mean Exact R−Precision by Topic (Topics 041 to 085)
queries 1
reinaES2mtdtest
0.5
041 22.62 086 24.59 131 34.04 166 0.00
Difference
0
042 35.22 087 50.94 132 9.46 167 40.00
−0.5
043 30.56 088 36.21 133 6.51 168 23.68
044 10.77 089 22.96 134 37.04 169 0.00 −1
041 042 043 044 045 046 047 048 049 060 061 062 063 064 065 066 067 068 069 080 081 082 083 084 085
Topic Identifier
045 38.34 090 13.48 135 40.32 170 19.05 Ad−Hoc Robust Multilingual track − Comparison to Median Mean Exact R−Precision by Topic (Topics 086 to 130)
1
046 6.25 091 8.23 136 25.71 171 15.25 reinaES2mtdtest
047 21.66 092 23.79 137 16.00 172 0.00 0.5
Difference
048 4.80 093 26.19 138 17.65 173 44.00 0
049 12.26 094 29.75 139 6.73 174 31.03 −0.5
060 24.14 095 24.93 140 28.89 175 31.82 −1
086 087 088 089 090 091 092 093 094 095 096 097 098 099 110 111 112 113 114 115 116 117 118 119 130
061 6.06 096 13.04 141 52.94 176 60.82 Topic Identifier
Ad−Hoc Robust Multilingual track − Comparison to Median Mean Exact R−Precision by Topic (Topics 131 to 165)
062 29.63 097 35.29 142 35.47 177 24.19 1
reinaES2mtdtest
063 33.33 098 43.10 143 44.97 178 27.56 0.5
Difference
064 35.00 099 8.85 144 56.67 179 35.37
0
065 14.74 110 52.58 145 50.00 190 4.58
−0.5
066 59.21 111 21.05 146 16.67 191 0.00
−1
067 13.40 112 31.88 147 34.26 192 35.46 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 160 161 162 163 164 165
Topic Identifier
068 6.52 113 8.80 148 10.67 193 23.74 Ad−Hoc Robust Multilingual track − Comparison to Median Mean Exact R−Precision by Topic (Topics 166 to 200)
1
069 32.61 114 17.24 149 30.77 194 29.17 reinaES2mtdtest
0.5
080 37.04 115 28.57 160 20.00 195 32.79
Difference
081 43.06 116 24.49 161 22.22 196 42.42 0
082 35.82 117 16.77 162 51.77 197 55.96 −0.5
083 32.35 118 13.04 163 44.94 198 9.09 −1
166 167 168 169 170 171 172 173 174 175 176 177 178 179 190 191 192 193 194 195 196 197 198 199 200
084 34.88 119 51.64 164 14.37 199 59.02 Topic Identifier
085 47.90 130 46.62 165 36.84 200 28.08
256
reina reinaES2mttest AH-ROBUST-MULTI-CLEF2006
Overall statistics for 100 queries : Priority 2
Total number of documents over all queries Query Construction AUTOMATIC
Retrieved 100,000 Source Language Spanish; Castilian
Relevant 17,019 Topic Fields title
Relevant retrieved 7,300 Pooled false
Geometric Mean Average Precision 0.1071 MT + local expansion with term windows, only title
Binary Preference (BPREF) 0.2708 field
Interploated Recall (%) Precision Averages (%)
Ad−Hoc Robust Multilingual track − Interpolated Recall vs Average Precision
100%
0 73.13
reinaES2mttest
10 46.55 90%
20 37.18
80%
30 26.79
40 18.60 70%
50 12.08
Average Precision
60%
60 5.51
70 1.14 50%
80 0.24 40%
90 0.17
30%
100 0.12
Average precision (non-interpolated) for all 20%
relevant documents (averaged over queries)
17.31 10%
0%
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
Interpolated Recall
Mean Average Precision Ad−Hoc Robust Multilingual track − Box plot of the Topics of the Experiment
Maximum 0.5660
Minimum 0.0018
First Quartile 0.0628
Second Quartile 0.1450
Third Quartile 0.2773
Interquartile range 0.2145
Mean 0.1731
Standard Deviation 0.1361
Lower Outlier Threshold 0.0018
Upper Outlier Threshold 0.5660 0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Mean Average Precision
Mean With No Outliers 0.1731
Std With No Outliers 0.1361
Ad−Hoc Robust Multilingual track − Distribution of the Topics of the Experiment
25
Number of Topics of the Experiment
20
15
10
5
0
0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Mean Average Precision
Precision averages (%) for individual Ad−Hoc Robust Multilingual track − Comparison to Median Mean Average Precision by Topic (Topics 041 to 085)
queries 1
reinaES2mttest
0.5
041 14.75 086 12.39 131 16.98 166 0.40
Difference
0
042 7.95 087 44.64 132 1.92 167 34.24
−0.5
043 4.89 088 18.25 133 2.16 168 15.29
044 7.74 089 10.97 134 17.32 169 0.96 −1
041 042 043 044 045 046 047 048 049 060 061 062 063 064 065 066 067 068 069 080 081 082 083 084 085
Topic Identifier
045 30.50 090 5.07 135 35.25 170 9.02 Ad−Hoc Robust Multilingual track − Comparison to Median Mean Average Precision by Topic (Topics 086 to 130)
1
046 2.37 091 7.42 136 22.98 171 6.65 reinaES2mttest
047 10.26 092 13.71 137 15.86 172 1.06 0.5
Difference
048 0.68 093 11.97 138 1.01 173 41.75 0
049 4.85 094 8.99 139 3.11 174 12.44 −0.5
060 10.02 095 2.21 140 14.56 175 13.69 −1
086 087 088 089 090 091 092 093 094 095 096 097 098 099 110 111 112 113 114 115 116 117 118 119 130
061 2.47 096 5.69 141 40.24 176 53.96 Topic Identifier
Ad−Hoc Robust Multilingual track − Comparison to Median Mean Average Precision by Topic (Topics 131 to 165)
062 14.45 097 21.31 142 34.62 177 17.20 1
reinaES2mttest
063 32.82 098 34.99 143 24.81 178 15.42 0.5
Difference
064 18.12 099 6.03 144 37.02 179 28.07
0
065 6.53 110 30.83 145 49.94 190 0.35
−0.5
066 56.60 111 9.03 146 28.18 191 12.50
−1
067 2.45 112 13.38 147 13.65 192 10.27 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 160 161 162 163 164 165
Topic Identifier
068 6.56 113 0.55 148 2.92 193 7.96 Ad−Hoc Robust Multilingual track − Comparison to Median Mean Average Precision by Topic (Topics 166 to 200)
1
069 16.19 114 4.70 149 21.35 194 0.18 reinaES2mttest
0.5
080 23.77 115 14.58 160 21.79 195 17.69
Difference
081 19.58 116 10.82 161 28.15 196 30.72 0
082 34.97 117 3.48 162 23.58 197 36.95 −0.5
083 18.87 118 5.90 163 27.39 198 2.82 −1
166 167 168 169 170 171 172 173 174 175 176 177 178 179 190 191 192 193 194 195 196 197 198 199 200
084 29.44 119 26.02 164 7.64 199 42.57 Topic Identifier
085 33.51 130 40.61 165 32.35 200 20.47
257
reina reinaES2mttest AH-ROBUST-MULTI-CLEF2006
Docs Cutoff Levels Precision at DCL (%)
Ad−Hoc Robust Multilingual track − Retrieved documents vs Precision
100%
5 docs 46.80
reinaES2mttest
10 docs 44.40 90%
15 docs 41.00
80%
20 docs 40.35
30 docs 36.67 70%
100 docs 25.67
60%
200 docs 19.39
R−Precision
500 docs 11.76 50%
1000 docs 7.30 40%
R-Precision (precision after R document retrieved,
where R = Relevant retrieved) 30%
25.14
20%
10%
0%
5 10 15 20 30 100 200 500 1000
Retrieved Documents (logarithmic scale)
Exact R-Precision Ad−Hoc Robust Multilingual track − Box plot of the Topics of the Experiment
Maximum 0.6029
Minimum 0.0000
First Quartile 0.1310
Second Quartile 0.2396
Third Quartile 0.3614
Interquartile range 0.2305
Mean 0.2514
Standard Deviation 0.1497
Lower Outlier Threshold 0.0000
Upper Outlier Threshold 0.6029 0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Exact R−Precision
Mean With No Outliers 0.2514
Std With No Outliers 0.1497
Ad−Hoc Robust Multilingual track − Distribution of the Topics of the Experiment
20
Number of Topics of the Experiment
15
10
5
0
0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Exact R−Precision
Precision averages (%) for individual Ad−Hoc Robust Multilingual track − Comparison to Median Mean Exact R−Precision by Topic (Topics 041 to 085)
queries 1
reinaES2mttest
0.5
041 23.81 086 24.86 131 31.40 166 0.00
Difference
0
042 24.05 087 55.35 132 1.35 167 36.00
−0.5
043 11.11 088 33.62 133 9.30 168 23.03
044 10.77 089 22.18 134 29.63 169 0.00 −1
041 042 043 044 045 046 047 048 049 060 061 062 063 064 065 066 067 068 069 080 081 082 083 084 085
Topic Identifier
045 42.92 090 15.73 135 41.52 170 19.05 Ad−Hoc Robust Multilingual track − Comparison to Median Mean Exact R−Precision by Topic (Topics 086 to 130)
1
046 6.25 091 12.66 136 28.57 171 11.86 reinaES2mttest
047 20.47 092 24.91 137 24.00 172 3.57 0.5
Difference
048 5.90 093 21.43 138 4.20 173 44.00 0
049 11.32 094 19.83 139 6.73 174 20.69 −0.5
060 21.00 095 13.40 140 26.69 175 18.18 −1
086 087 088 089 090 091 092 093 094 095 096 097 098 099 110 111 112 113 114 115 116 117 118 119 130
061 9.09 096 14.91 141 50.00 176 60.00 Topic Identifier
Ad−Hoc Robust Multilingual track − Comparison to Median Mean Exact R−Precision by Topic (Topics 131 to 165)
062 29.63 097 35.29 142 35.47 177 22.58 1
reinaES2mttest
063 44.93 098 43.10 143 40.50 178 29.92 0.5
Difference
064 15.00 099 15.10 144 46.67 179 36.73
0
065 13.15 110 45.36 145 50.81 190 2.29
−0.5
066 60.29 111 18.42 146 33.33 191 0.00
−1
067 5.15 112 24.64 147 24.07 192 17.73 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 160 161 162 163 164 165
Topic Identifier
068 13.04 113 5.63 148 10.67 193 14.39 Ad−Hoc Robust Multilingual track − Comparison to Median Mean Exact R−Precision by Topic (Topics 166 to 200)
1
069 23.91 114 10.34 149 28.21 194 0.00 reinaES2mttest
0.5
080 42.29 115 26.19 160 20.00 195 32.79
Difference
081 33.80 116 24.49 161 22.22 196 36.36 0
082 41.04 117 12.26 162 36.28 197 50.52 −0.5
083 32.35 118 15.22 163 35.22 198 12.12 −1
166 167 168 169 170 171 172 173 174 175 176 177 178 179 190 191 192 193 194 195 196 197 198 199 200
084 37.21 119 37.62 164 18.48 199 56.15 Topic Identifier
085 44.01 130 48.87 165 36.84 200 30.05
258
colesir CoLesIRendeTst AH-ROBUST-BILI-X2DE-CLEF2006
Overall statistics for 100 queries : Priority 1
Total number of documents over all queries Query Construction AUTOMATIC
Retrieved 100,000 Source Language English
Relevant 3,641 Topic Fields title, description
Relevant retrieved 2,599 Pooled false
Geometric Mean Average Precision 0.0431 robust bilingual EN2DE - TEST; automatic; TD
Binary Preference (BPREF) 0.3157
Interploated Recall (%) Precision Averages (%)
Ad−Hoc Robust Bilingual German track − Interpolated Recall vs Average Precision
100%
0 51.63
CoLesIRendeTst
10 40.98 90%
20 36.31
80%
30 32.49
40 29.44 70%
50 27.38
Average Precision
60%
60 22.85
70 19.63 50%
80 16.50 40%
90 10.66
30%
100 6.45
Average precision (non-interpolated) for all 20%
relevant documents (averaged over queries)
25.24 10%
0%
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
Interpolated Recall
Mean Average Precision Ad−Hoc Robust Bilingual German track − Box plot of the Topics of the Experiment
Maximum 0.9764
Minimum 0.0000
First Quartile 0.0138
Second Quartile 0.1300
Third Quartile 0.4951
Interquartile range 0.4813
Mean 0.2524
Standard Deviation 0.2761
Lower Outlier Threshold 0.0000
Upper Outlier Threshold 0.9764 0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Mean Average Precision
Mean With No Outliers 0.2524
Std With No Outliers 0.2761
Ad−Hoc Robust Bilingual German track − Distribution of the Topics of the Experiment
40
Number of Topics of the Experiment
35
30
25
20
15
10
5
0
0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Mean Average Precision
Precision averages (%) for individual Ad−Hoc Robust Bilingual German track − Comparison to Median Mean Average Precision by Topic (Topics 041 to 085)
queries 1
CoLesIRendeTst
0.5
041 6.43 086 17.63 131 3.50 166 0.24
Difference
0
042 57.68 087 14.87 132 62.35 167 88.75
−0.5
043 0.08 088 41.72 133 0.04 168 5.61
044 0.00 089 33.88 134 86.38 169 0.07 −1
041 042 043 044 045 046 047 048 049 060 061 062 063 064 065 066 067 068 069 080 081 082 083 084 085
Topic Identifier
045 54.02 090 10.47 135 4.84 170 0.00 Ad−Hoc Robust Bilingual German track − Comparison to Median Mean Average Precision by Topic (Topics 086 to 130)
1
046 3.48 091 5.31 136 0.00 171 19.05 CoLesIRendeTst
047 0.55 092 13.11 137 11.11 172 0.90 0.5
Difference
048 0.37 093 69.93 138 34.32 173 67.03 0
049 4.60 094 49.65 139 0.14 174 92.67 −0.5
060 28.29 095 8.38 140 4.64 175 50.58 −1
086 087 088 089 090 091 092 093 094 095 096 097 098 099 110 111 112 113 114 115 116 117 118 119 130
061 9.45 096 9.48 141 53.86 176 77.03 Topic Identifier
Ad−Hoc Robust Bilingual German track − Comparison to Median Mean Average Precision by Topic (Topics 131 to 165)
062 51.67 097 17.23 142 71.14 177 3.15 1
CoLesIRendeTst
063 40.83 098 47.24 143 22.64 178 0.00 0.5
Difference
064 50.00 099 1.02 144 0.00 179 9.41
0
065 1.34 110 51.76 145 19.46 190 16.76
−0.5
066 62.87 111 1.43 146 0.00 191 0.00
−1
067 1.80 112 66.27 147 3.36 192 9.58 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 160 161 162 163 164 165
Topic Identifier
068 15.16 113 0.44 148 3.40 193 43.97 Ad−Hoc Robust Bilingual German track − Comparison to Median Mean Average Precision by Topic (Topics 166 to 200)
1
069 11.67 114 16.53 149 97.64 194 55.00 CoLesIRendeTst
0.5
080 15.19 115 6.25 160 33.33 195 0.10
Difference
081 35.10 116 49.38 161 0.00 196 95.83 0
082 25.62 117 8.64 162 33.50 197 58.31 −0.5
083 17.72 118 0.22 163 12.90 198 0.09 −1
166 167 168 169 170 171 172 173 174 175 176 177 178 179 190 191 192 193 194 195 196 197 198 199 200
084 10.00 119 61.26 164 0.03 199 70.56 Topic Identifier
085 40.48 130 0.01 165 53.57 200 0.93
259
colesir CoLesIRendeTst AH-ROBUST-BILI-X2DE-CLEF2006
Docs Cutoff Levels Precision at DCL (%)
Ad−Hoc Robust Bilingual German track − Retrieved documents vs Precision
100%
5 docs 32.60
CoLesIRendeTst
10 docs 28.70 90%
15 docs 25.80
80%
20 docs 24.90
30 docs 22.80 70%
100 docs 13.54
60%
200 docs 9.02
R−Precision
500 docs 4.58 50%
1000 docs 2.60 40%
R-Precision (precision after R document retrieved,
where R = Relevant retrieved) 30%
26.40
20%
10%
0%
5 10 15 20 30 100 200 500 1000
Retrieved Documents (logarithmic scale)
Exact R-Precision Ad−Hoc Robust Bilingual German track − Box plot of the Topics of the Experiment
Maximum 0.9167
Minimum 0.0000
First Quartile 0.0000
Second Quartile 0.1820
Third Quartile 0.4853
Interquartile range 0.4853
Mean 0.2640
Standard Deviation 0.2701
Lower Outlier Threshold 0.0000
Upper Outlier Threshold 0.9167 0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Exact R−Precision
Mean With No Outliers 0.2640
Std With No Outliers 0.2701
Ad−Hoc Robust Bilingual German track − Distribution of the Topics of the Experiment
40
Number of Topics of the Experiment
35
30
25
20
15
10
5
0
0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Exact R−Precision
Precision averages (%) for individual Ad−Hoc Robust Bilingual German track − Comparison to Median Mean Exact R−Precision by Topic (Topics 041 to 085)
queries 1
CoLesIRendeTst
0.5
041 7.14 086 24.32 131 12.12 166 0.00
Difference
0
042 61.79 087 17.65 132 71.43 167 75.00
−0.5
043 0.00 088 55.17 133 0.91 168 5.88
044 0.00 089 42.07 134 85.71 169 0.00 −1
041 042 043 044 045 046 047 048 049 060 061 062 063 064 065 066 067 068 069 080 081 082 083 084 085
Topic Identifier
045 53.77 090 18.75 135 8.05 170 0.00 Ad−Hoc Robust Bilingual German track − Comparison to Median Mean Exact R−Precision by Topic (Topics 086 to 130)
1
046 0.00 091 6.25 136 0.00 171 37.50 CoLesIRendeTst
047 2.08 092 14.71 137 0.00 172 0.00 0.5
Difference
048 0.00 093 77.78 138 29.03 173 70.00 0
049 16.67 094 50.00 139 0.00 174 83.33 −0.5
060 40.35 095 13.64 140 13.58 175 54.55 −1
086 087 088 089 090 091 092 093 094 095 096 097 098 099 110 111 112 113 114 115 116 117 118 119 130
061 11.63 096 11.11 141 62.50 176 72.97 Topic Identifier
Ad−Hoc Robust Bilingual German track − Comparison to Median Mean Exact R−Precision by Topic (Topics 131 to 165)
062 44.44 097 26.67 142 76.92 177 0.00 1
CoLesIRendeTst
063 42.11 098 44.00 143 22.22 178 0.00 0.5
Difference
064 0.00 099 4.49 144 0.00 179 8.16
0
065 6.15 110 58.33 145 20.00 190 28.07
−0.5
066 66.67 111 0.00 146 0.00 191 0.00
−1
067 2.00 112 57.14 147 6.90 192 19.23 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 160 161 162 163 164 165
Topic Identifier
068 20.45 113 0.00 148 8.33 193 47.06 Ad−Hoc Robust Bilingual German track − Comparison to Median Mean Exact R−Precision by Topic (Topics 166 to 200)
1
069 14.29 114 21.43 149 91.67 194 50.00 CoLesIRendeTst
0.5
080 22.50 115 12.50 160 0.00 195 0.00
Difference
081 40.00 116 54.84 161 0.00 196 83.33 0
082 25.93 117 11.11 162 33.33 197 59.84 −0.5
083 33.33 118 0.00 163 19.15 198 0.00 −1
166 167 168 169 170 171 172 173 174 175 176 177 178 179 190 191 192 193 194 195 196 197 198 199 200
084 20.00 119 55.21 164 0.00 199 77.08 Topic Identifier
085 45.95 130 0.00 165 50.00 200 1.52
260
daedalus deFSdeRSen2S AH-ROBUST-BILI-X2DE-CLEF2006
Overall statistics for 100 queries : Priority 4
Total number of documents over all queries Query Construction AUTOMATIC
Retrieved 100,000 Source Language English
Relevant 3,641 Topic Fields title, description
Relevant retrieved 2,509 Pooled false
Geometric Mean Average Precision 0.0518 Mandatory TD run
Binary Preference (BPREF) 0.3621
Interploated Recall (%) Precision Averages (%)
Ad−Hoc Robust Bilingual German track − Interpolated Recall vs Average Precision
100%
0 57.35
deFSdeRSen2S
10 48.42 90%
20 42.04
80%
30 37.78
40 35.25 70%
50 32.00
Average Precision
60%
60 26.64
70 22.26 50%
80 16.48 40%
90 11.93
30%
100 6.67
Average precision (non-interpolated) for all 20%
relevant documents (averaged over queries)
29.16 10%
0%
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
Interpolated Recall
Mean Average Precision Ad−Hoc Robust Bilingual German track − Box plot of the Topics of the Experiment
Maximum 1.0000
Minimum 0.0000
First Quartile 0.0205
Second Quartile 0.1632
Third Quartile 0.5251
Interquartile range 0.5046
Mean 0.2916
Standard Deviation 0.2956
Lower Outlier Threshold 0.0000
Upper Outlier Threshold 1.0000 0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Mean Average Precision
Mean With No Outliers 0.2916
Std With No Outliers 0.2956
Ad−Hoc Robust Bilingual German track − Distribution of the Topics of the Experiment
40
Number of Topics of the Experiment
35
30
25
20
15
10
5
0
0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Mean Average Precision
Precision averages (%) for individual Ad−Hoc Robust Bilingual German track − Comparison to Median Mean Average Precision by Topic (Topics 041 to 085)
queries 1
deFSdeRSen2S
0.5
041 2.01 086 16.50 131 22.46 166 0.46
Difference
0
042 72.86 087 25.69 132 73.99 167 70.00
−0.5
043 7.14 088 45.22 133 0.51 168 7.71
044 0.00 089 36.30 134 69.89 169 0.00 −1
041 042 043 044 045 046 047 048 049 060 061 062 063 064 065 066 067 068 069 080 081 082 083 084 085
Topic Identifier
045 60.38 090 12.68 135 2.09 170 0.00 Ad−Hoc Robust Bilingual German track − Comparison to Median Mean Average Precision by Topic (Topics 086 to 130)
1
046 18.95 091 0.13 136 52.01 171 3.93 deFSdeRSen2S
047 0.92 092 16.13 137 50.00 172 0.00 0.5
Difference
048 0.94 093 65.49 138 14.50 173 88.56 0
049 3.07 094 10.58 139 0.21 174 67.87 −0.5
060 21.16 095 1.79 140 0.19 175 82.70 −1
086 087 088 089 090 091 092 093 094 095 096 097 098 099 110 111 112 113 114 115 116 117 118 119 130
061 12.73 096 11.73 141 75.00 176 87.86 Topic Identifier
Ad−Hoc Robust Bilingual German track − Comparison to Median Mean Average Precision by Topic (Topics 131 to 165)
062 42.90 097 5.67 142 36.45 177 14.61 1
deFSdeRSen2S
063 51.94 098 40.61 143 15.50 178 0.10 0.5
Difference
064 100.00 099 0.08 144 0.00 179 36.16
0
065 0.00 110 36.77 145 58.46 190 51.30
−0.5
066 77.99 111 2.48 146 0.00 191 0.00
−1
067 7.53 112 56.43 147 11.78 192 0.49 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 160 161 162 163 164 165
Topic Identifier
068 17.24 113 0.59 148 0.26 193 50.13 Ad−Hoc Robust Bilingual German track − Comparison to Median Mean Average Precision by Topic (Topics 166 to 200)
1
069 12.93 114 8.90 149 78.45 194 83.33 deFSdeRSen2S
0.5
080 48.13 115 4.59 160 100.00 195 8.98
Difference
081 57.15 116 53.37 161 0.00 196 53.00 0
082 73.88 117 8.03 162 48.38 197 57.88 −0.5
083 17.61 118 5.92 163 20.77 198 0.01 −1
166 167 168 169 170 171 172 173 174 175 176 177 178 179 190 191 192 193 194 195 196 197 198 199 200
084 10.00 119 44.62 164 0.31 199 72.34 Topic Identifier
085 60.51 130 17.27 165 37.88 200 3.88
261
daedalus deFSdeRSen2S AH-ROBUST-BILI-X2DE-CLEF2006
Docs Cutoff Levels Precision at DCL (%)
Ad−Hoc Robust Bilingual German track − Retrieved documents vs Precision
100%
5 docs 38.20
deFSdeRSen2S
10 docs 32.20 90%
15 docs 29.47
80%
20 docs 28.00
30 docs 24.87 70%
100 docs 14.11
60%
200 docs 9.07
R−Precision
500 docs 4.50 50%
1000 docs 2.51 40%
R-Precision (precision after R document retrieved,
where R = Relevant retrieved) 30%
30.15
20%
10%
0%
5 10 15 20 30 100 200 500 1000
Retrieved Documents (logarithmic scale)
Exact R-Precision Ad−Hoc Robust Bilingual German track − Box plot of the Topics of the Experiment
Maximum 1.0000
Minimum 0.0000
First Quartile 0.0308
Second Quartile 0.2192
Third Quartile 0.5381
Interquartile range 0.5073
Mean 0.3015
Standard Deviation 0.2783
Lower Outlier Threshold 0.0000
Upper Outlier Threshold 1.0000 0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Exact R−Precision
Mean With No Outliers 0.3015
Std With No Outliers 0.2783
Ad−Hoc Robust Bilingual German track − Distribution of the Topics of the Experiment
30
Number of Topics of the Experiment
25
20
15
10
5
0
0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Exact R−Precision
Precision averages (%) for individual Ad−Hoc Robust Bilingual German track − Comparison to Median Mean Exact R−Precision by Topic (Topics 041 to 085)
queries 1
deFSdeRSen2S
0.5
041 3.57 086 24.32 131 31.82 166 0.00
Difference
0
042 66.51 087 23.53 132 75.00 167 50.00
−0.5
043 0.00 088 53.45 133 3.64 168 5.88
044 0.00 089 43.45 134 71.43 169 0.00 −1
041 042 043 044 045 046 047 048 049 060 061 062 063 064 065 066 067 068 069 080 081 082 083 084 085
Topic Identifier
045 61.32 090 18.75 135 3.45 170 0.00 Ad−Hoc Robust Bilingual German track − Comparison to Median Mean Exact R−Precision by Topic (Topics 086 to 130)
1
046 20.00 091 3.12 136 42.86 171 0.00 deFSdeRSen2S
047 2.08 092 20.59 137 0.00 172 0.00 0.5
Difference
048 5.45 093 72.22 138 16.13 173 80.00 0
049 5.56 094 18.75 139 0.00 174 63.89 −0.5
060 31.58 095 4.55 140 2.47 175 72.73 −1
086 087 088 089 090 091 092 093 094 095 096 097 098 099 110 111 112 113 114 115 116 117 118 119 130
061 23.26 096 18.52 141 75.00 176 78.38 Topic Identifier
Ad−Hoc Robust Bilingual German track − Comparison to Median Mean Exact R−Precision by Topic (Topics 131 to 165)
062 33.33 097 13.33 142 40.00 177 14.29 1
deFSdeRSen2S
063 63.16 098 60.00 143 26.98 178 0.00 0.5
Difference
064 100.00 099 0.00 144 0.00 179 38.78
0
065 0.00 110 50.00 145 50.00 190 56.14
−0.5
066 70.59 111 0.00 146 0.00 191 0.00
−1
067 18.00 112 50.00 147 17.24 192 0.00 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 160 161 162 163 164 165
Topic Identifier
068 20.45 113 0.00 148 0.00 193 52.94 Ad−Hoc Robust Bilingual German track − Comparison to Median Mean Exact R−Precision by Topic (Topics 166 to 200)
1
069 28.57 114 10.71 149 75.00 194 50.00 deFSdeRSen2S
0.5
080 54.17 115 12.50 160 100.00 195 27.59
Difference
081 60.00 116 54.84 161 0.00 196 50.00 0
082 70.37 117 17.78 162 54.17 197 56.56 −0.5
083 11.11 118 11.11 163 25.53 198 0.00 −1
166 167 168 169 170 171 172 173 174 175 176 177 178 179 190 191 192 193 194 195 196 197 198 199 200
084 20.00 119 47.92 164 1.39 199 70.83 Topic Identifier
085 56.76 130 26.67 165 50.00 200 3.03
262
daedalus deFSdeRSen4S AH-ROBUST-BILI-X2DE-CLEF2006
Overall statistics for 100 queries : Priority 2
Total number of documents over all queries Query Construction AUTOMATIC
Retrieved 100,000 Source Language English
Relevant 3,641 Topic Fields title, description, narrative
Relevant retrieved 2,626 Pooled false
Geometric Mean Average Precision 0.0601 TDN run
Binary Preference (BPREF) 0.3831
Interploated Recall (%) Precision Averages (%)
Ad−Hoc Robust Bilingual German track − Interpolated Recall vs Average Precision
100%
0 61.37
deFSdeRSen4S
10 51.30 90%
20 45.20
80%
30 40.30
40 37.71 70%
50 34.99
Average Precision
60%
60 30.52
70 26.39 50%
80 19.39 40%
90 13.60
30%
100 7.09
Average precision (non-interpolated) for all 20%
relevant documents (averaged over queries)
32.22 10%
0%
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
Interpolated Recall
Mean Average Precision Ad−Hoc Robust Bilingual German track − Box plot of the Topics of the Experiment
Maximum 1.0000
Minimum 0.0000
First Quartile 0.0214
Second Quartile 0.1928
Third Quartile 0.5991
Interquartile range 0.5777
Mean 0.3222
Standard Deviation 0.3078
Lower Outlier Threshold 0.0000
Upper Outlier Threshold 1.0000 0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Mean Average Precision
Mean With No Outliers 0.3222
Std With No Outliers 0.3078
Ad−Hoc Robust Bilingual German track − Distribution of the Topics of the Experiment
30
Number of Topics of the Experiment
25
20
15
10
5
0
0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Mean Average Precision
Precision averages (%) for individual Ad−Hoc Robust Bilingual German track − Comparison to Median Mean Average Precision by Topic (Topics 041 to 085)
queries 1
deFSdeRSen4S
0.5
041 0.86 086 19.25 131 23.25 166 0.50
Difference
0
042 74.47 087 28.23 132 65.65 167 73.33
−0.5
043 10.21 088 49.05 133 1.87 168 8.75
044 0.00 089 39.17 134 64.69 169 0.00 −1
041 042 043 044 045 046 047 048 049 060 061 062 063 064 065 066 067 068 069 080 081 082 083 084 085
Topic Identifier
045 59.95 090 12.81 135 0.57 170 0.00 Ad−Hoc Robust Bilingual German track − Comparison to Median Mean Average Precision by Topic (Topics 086 to 130)
1
046 21.33 091 0.07 136 62.26 171 8.60 deFSdeRSen4S
047 4.45 092 16.57 137 33.33 172 0.00 0.5
Difference
048 1.07 093 74.42 138 59.87 173 93.19 0
049 4.19 094 6.56 139 0.30 174 86.30 −0.5
060 18.00 095 2.41 140 0.12 175 86.13 −1
086 087 088 089 090 091 092 093 094 095 096 097 098 099 110 111 112 113 114 115 116 117 118 119 130
061 16.70 096 10.84 141 75.00 176 91.94 Topic Identifier
Ad−Hoc Robust Bilingual German track − Comparison to Median Mean Average Precision by Topic (Topics 131 to 165)
062 33.73 097 9.69 142 36.29 177 14.89 1
deFSdeRSen4S
063 53.71 098 43.57 143 14.88 178 0.16 0.5
Difference
064 100.00 099 1.12 144 0.00 179 41.18
0
065 0.07 110 48.33 145 70.29 190 54.36
−0.5
066 76.81 111 4.41 146 0.00 191 0.00
−1
067 9.27 112 64.95 147 17.61 192 8.09 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 160 161 162 163 164 165
Topic Identifier
068 19.32 113 1.56 148 0.26 193 53.70 Ad−Hoc Robust Bilingual German track − Comparison to Median Mean Average Precision by Topic (Topics 166 to 200)
1
069 37.69 114 18.69 149 85.07 194 70.00 deFSdeRSen4S
0.5
080 51.53 115 27.99 160 100.00 195 23.17
Difference
081 58.36 116 66.29 161 0.00 196 79.08 0
082 81.23 117 14.30 162 45.99 197 55.50 −0.5
083 21.99 118 1.81 163 15.17 198 0.03 −1
166 167 168 169 170 171 172 173 174 175 176 177 178 179 190 191 192 193 194 195 196 197 198 199 200
084 0.00 119 46.26 164 0.24 199 75.54 Topic Identifier
085 68.08 130 64.08 165 17.28 200 12.42
263
daedalus deFSdeRSen4S AH-ROBUST-BILI-X2DE-CLEF2006
Docs Cutoff Levels Precision at DCL (%)
Ad−Hoc Robust Bilingual German track − Retrieved documents vs Precision
100%
5 docs 41.80
deFSdeRSen4S
10 docs 37.30 90%
15 docs 33.67
80%
20 docs 30.90
30 docs 27.27 70%
100 docs 15.24
60%
200 docs 9.81
R−Precision
500 docs 4.78 50%
1000 docs 2.63 40%
R-Precision (precision after R document retrieved,
where R = Relevant retrieved) 30%
33.20
20%
10%
0%
5 10 15 20 30 100 200 500 1000
Retrieved Documents (logarithmic scale)
Exact R-Precision Ad−Hoc Robust Bilingual German track − Box plot of the Topics of the Experiment
Maximum 1.0000
Minimum 0.0000
First Quartile 0.0473
Second Quartile 0.2520
Third Quartile 0.5819
Interquartile range 0.5347
Mean 0.3320
Standard Deviation 0.2900
Lower Outlier Threshold 0.0000
Upper Outlier Threshold 1.0000 0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Exact R−Precision
Mean With No Outliers 0.3320
Std With No Outliers 0.2900
Ad−Hoc Robust Bilingual German track − Distribution of the Topics of the Experiment
25
Number of Topics of the Experiment
20
15
10
5
0
0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Exact R−Precision
Precision averages (%) for individual Ad−Hoc Robust Bilingual German track − Comparison to Median Mean Exact R−Precision by Topic (Topics 041 to 085)
queries 1
deFSdeRSen4S
0.5
041 7.14 086 26.13 131 34.85 166 0.00
Difference
0
042 68.87 087 23.53 132 64.29 167 50.00
−0.5
043 0.00 088 58.62 133 7.27 168 11.76
044 0.00 089 46.21 134 57.14 169 0.00 −1
041 042 043 044 045 046 047 048 049 060 061 062 063 064 065 066 067 068 069 080 081 082 083 084 085
Topic Identifier
045 58.49 090 18.75 135 0.00 170 0.00 Ad−Hoc Robust Bilingual German track − Comparison to Median Mean Exact R−Precision by Topic (Topics 086 to 130)
1
046 20.00 091 0.00 136 57.14 171 12.50 deFSdeRSen4S
047 6.25 092 14.71 137 0.00 172 0.00 0.5
Difference
048 3.64 093 75.00 138 67.74 173 80.00 0
049 5.56 094 15.62 139 0.00 174 80.56 −0.5
060 24.56 095 5.45 140 0.00 175 81.82 −1
086 087 088 089 090 091 092 093 094 095 096 097 098 099 110 111 112 113 114 115 116 117 118 119 130
061 27.91 096 18.52 141 75.00 176 87.84 Topic Identifier
Ad−Hoc Robust Bilingual German track − Comparison to Median Mean Exact R−Precision by Topic (Topics 131 to 165)
062 33.33 097 20.00 142 40.00 177 14.29 1
deFSdeRSen4S
063 55.26 098 60.00 143 25.40 178 0.00 0.5
Difference
064 100.00 099 1.12 144 0.00 179 42.86
0
065 0.00 110 54.17 145 70.00 190 57.89
−0.5
066 68.63 111 12.50 146 0.00 191 0.00
−1
067 16.00 112 64.29 147 27.59 192 15.38 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 160 161 162 163 164 165
Topic Identifier
068 22.73 113 4.00 148 0.00 193 55.88 Ad−Hoc Robust Bilingual German track − Comparison to Median Mean Exact R−Precision by Topic (Topics 166 to 200)
1
069 42.86 114 21.43 149 83.33 194 50.00 deFSdeRSen4S
0.5
080 54.17 115 33.33 160 100.00 195 31.03
Difference
081 64.00 116 64.52 161 0.00 196 66.67 0
082 77.78 117 20.00 162 54.17 197 56.56 −0.5
083 33.33 118 0.00 163 23.40 198 0.00 −1
166 167 168 169 170 171 172 173 174 175 176 177 178 179 190 191 192 193 194 195 196 197 198 199 200
084 0.00 119 47.92 164 1.39 199 75.00 Topic Identifier
085 62.16 130 60.00 165 25.00 200 18.18
264
daedalus deFSdeRSen3S AH-ROBUST-BILI-X2DE-CLEF2006
Overall statistics for 100 queries : Priority 1
Total number of documents over all queries Query Construction AUTOMATIC
Retrieved 100,000 Source Language English
Relevant 3,641 Topic Fields title, description, narrative
Relevant retrieved 2,668 Pooled false
Geometric Mean Average Precision 0.0659 TDN run
Binary Preference (BPREF) 0.3854
Interploated Recall (%) Precision Averages (%)
Ad−Hoc Robust Bilingual German track − Interpolated Recall vs Average Precision
100%
0 62.71
deFSdeRSen3S
10 50.79 90%
20 45.76
80%
30 41.14
40 38.05 70%
50 35.02
Average Precision
60%
60 30.06
70 26.60 50%
80 19.80 40%
90 14.48
30%
100 7.45
Average precision (non-interpolated) for all 20%
relevant documents (averaged over queries)
32.44 10%
0%
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
Interpolated Recall
Mean Average Precision Ad−Hoc Robust Bilingual German track − Box plot of the Topics of the Experiment
Maximum 1.0000
Minimum 0.0000
First Quartile 0.0409
Second Quartile 0.2000
Third Quartile 0.5851
Interquartile range 0.5442
Mean 0.3244
Standard Deviation 0.3075
Lower Outlier Threshold 0.0000
Upper Outlier Threshold 1.0000 0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Mean Average Precision
Mean With No Outliers 0.3244
Std With No Outliers 0.3075
Ad−Hoc Robust Bilingual German track − Distribution of the Topics of the Experiment
30
Number of Topics of the Experiment
25
20
15
10
5
0
0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Mean Average Precision
Precision averages (%) for individual Ad−Hoc Robust Bilingual German track − Comparison to Median Mean Average Precision by Topic (Topics 041 to 085)
queries 1
deFSdeRSen3S
0.5
041 0.90 086 20.37 131 25.79 166 0.94
Difference
0
042 74.09 087 28.55 132 61.56 167 74.09
−0.5
043 13.70 088 53.56 133 5.48 168 7.43
044 0.00 089 39.36 134 64.69 169 0.00 −1
041 042 043 044 045 046 047 048 049 060 061 062 063 064 065 066 067 068 069 080 081 082 083 084 085
Topic Identifier
045 57.93 090 12.81 135 1.92 170 0.00 Ad−Hoc Robust Bilingual German track − Comparison to Median Mean Average Precision by Topic (Topics 086 to 130)
1
046 25.76 091 0.17 136 62.26 171 10.35 deFSdeRSen3S
047 4.78 092 17.97 137 33.33 172 0.00 0.5
Difference
048 1.02 093 75.51 138 70.39 173 93.87 0
049 4.27 094 6.01 139 0.25 174 87.74 −0.5
060 18.33 095 9.33 140 0.19 175 83.90 −1
086 087 088 089 090 091 092 093 094 095 096 097 098 099 110 111 112 113 114 115 116 117 118 119 130
061 3.96 096 10.29 141 75.00 176 93.63 Topic Identifier
Ad−Hoc Robust Bilingual German track − Comparison to Median Mean Average Precision by Topic (Topics 131 to 165)
062 38.59 097 10.99 142 36.42 177 1.02 1
deFSdeRSen3S
063 56.50 098 43.79 143 19.63 178 0.22 0.5
Difference
064 100.00 099 0.74 144 0.00 179 42.93
0
065 1.87 110 52.78 145 60.88 190 53.22
−0.5
066 77.29 111 4.23 146 0.00 191 0.00
−1
067 9.36 112 67.21 147 7.28 192 14.52 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 160 161 162 163 164 165
Topic Identifier
068 16.24 113 9.16 148 0.25 193 50.56 Ad−Hoc Robust Bilingual German track − Comparison to Median Mean Average Precision by Topic (Topics 166 to 200)
1
069 18.71 114 26.62 149 91.58 194 59.09 deFSdeRSen3S
0.5
080 50.01 115 21.64 160 100.00 195 14.89
Difference
081 50.85 116 67.12 161 0.00 196 86.81 0
082 67.66 117 14.86 162 48.12 197 55.49 −0.5
083 16.95 118 1.88 163 36.00 198 0.17 −1
166 167 168 169 170 171 172 173 174 175 176 177 178 179 190 191 192 193 194 195 196 197 198 199 200
084 0.00 119 46.07 164 0.26 199 76.18 Topic Identifier
085 73.18 130 61.35 165 35.19 200 15.89
265
daedalus deFSdeRSen3S AH-ROBUST-BILI-X2DE-CLEF2006
Docs Cutoff Levels Precision at DCL (%)
Ad−Hoc Robust Bilingual German track − Retrieved documents vs Precision
100%
5 docs 42.80
deFSdeRSen3S
10 docs 38.00 90%
15 docs 34.40
80%
20 docs 31.85
30 docs 27.80 70%
100 docs 15.61
60%
200 docs 9.99
R−Precision
500 docs 4.89 50%
1000 docs 2.67 40%
R-Precision (precision after R document retrieved,
where R = Relevant retrieved) 30%
33.16
20%
10%
0%
5 10 15 20 30 100 200 500 1000
Retrieved Documents (logarithmic scale)
Exact R-Precision Ad−Hoc Robust Bilingual German track − Box plot of the Topics of the Experiment
Maximum 1.0000
Minimum 0.0000
First Quartile 0.0441
Second Quartile 0.2520
Third Quartile 0.5714
Interquartile range 0.5273
Mean 0.3316
Standard Deviation 0.2881
Lower Outlier Threshold 0.0000
Upper Outlier Threshold 1.0000 0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Exact R−Precision
Mean With No Outliers 0.3316
Std With No Outliers 0.2881
Ad−Hoc Robust Bilingual German track − Distribution of the Topics of the Experiment
30
Number of Topics of the Experiment
25
20
15
10
5
0
0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Exact R−Precision
Precision averages (%) for individual Ad−Hoc Robust Bilingual German track − Comparison to Median Mean Exact R−Precision by Topic (Topics 041 to 085)
queries 1
deFSdeRSen3S
0.5
041 7.14 086 30.63 131 36.36 166 0.00
Difference
0
042 70.28 087 23.53 132 53.57 167 50.00
−0.5
043 0.00 088 60.34 133 10.91 168 5.88
044 0.00 089 47.59 134 57.14 169 0.00 −1
041 042 043 044 045 046 047 048 049 060 061 062 063 064 065 066 067 068 069 080 081 082 083 084 085
Topic Identifier
045 59.43 090 18.75 135 5.75 170 0.00 Ad−Hoc Robust Bilingual German track − Comparison to Median Mean Exact R−Precision by Topic (Topics 086 to 130)
1
046 30.00 091 3.12 136 57.14 171 25.00 deFSdeRSen3S
047 4.17 092 23.53 137 0.00 172 0.00 0.5
Difference
048 3.64 093 75.00 138 70.97 173 80.00 0
049 5.56 094 12.50 139 0.00 174 80.56 −0.5
060 28.07 095 12.73 140 2.47 175 72.73 −1
086 087 088 089 090 091 092 093 094 095 096 097 098 099 110 111 112 113 114 115 116 117 118 119 130
061 4.65 096 11.11 141 75.00 176 87.84 Topic Identifier
Ad−Hoc Robust Bilingual German track − Comparison to Median Mean Exact R−Precision by Topic (Topics 131 to 165)
062 44.44 097 20.00 142 40.00 177 0.00 1
deFSdeRSen3S
063 57.89 098 56.00 143 25.40 178 0.00 0.5
Difference
064 100.00 099 0.00 144 0.00 179 44.90
0
065 9.23 110 54.17 145 60.00 190 56.14
−0.5
066 66.67 111 0.00 146 0.00 191 0.00
−1
067 20.00 112 64.29 147 13.79 192 23.08 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 160 161 162 163 164 165
Topic Identifier
068 20.45 113 16.00 148 0.00 193 47.06 Ad−Hoc Robust Bilingual German track − Comparison to Median Mean Exact R−Precision by Topic (Topics 166 to 200)
1
069 42.86 114 21.43 149 83.33 194 50.00 deFSdeRSen3S
0.5
080 53.33 115 33.33 160 100.00 195 31.03
Difference
081 54.00 116 61.29 161 0.00 196 83.33 0
082 70.37 117 24.44 162 62.50 197 50.82 −0.5
083 11.11 118 0.00 163 42.55 198 0.00 −1
166 167 168 169 170 171 172 173 174 175 176 177 178 179 190 191 192 193 194 195 196 197 198 199 200
084 0.00 119 47.92 164 1.39 199 75.00 Topic Identifier
085 67.57 130 60.00 165 25.00 200 21.21
266
daedalus deydeRSenFS3456 AH-ROBUST-BILI-X2DE-CLEF2006
Overall statistics for 100 queries : Priority 3
Total number of documents over all queries Query Construction AUTOMATIC
Retrieved 100,000 Source Language English
Relevant 3,641 Topic Fields title, description, narrative
Relevant retrieved 2,599 Pooled false
Geometric Mean Average Precision 0.0598 Average run
Binary Preference (BPREF) 0.3821
Interploated Recall (%) Precision Averages (%)
Ad−Hoc Robust Bilingual German track − Interpolated Recall vs Average Precision
100%
0 62.08
deydeRSenFS3456
10 51.17 90%
20 44.71
80%
30 40.12
40 37.51 70%
50 34.96
Average Precision
60%
60 30.79
70 26.22 50%
80 19.22 40%
90 13.52
30%
100 7.07
Average precision (non-interpolated) for all 20%
relevant documents (averaged over queries)
32.13 10%
0%
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
Interpolated Recall
Mean Average Precision Ad−Hoc Robust Bilingual German track − Box plot of the Topics of the Experiment
Maximum 1.0000
Minimum 0.0000
First Quartile 0.0256
Second Quartile 0.2035
Third Quartile 0.5904
Interquartile range 0.5648
Mean 0.3213
Standard Deviation 0.3069
Lower Outlier Threshold 0.0000
Upper Outlier Threshold 1.0000 0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Mean Average Precision
Mean With No Outliers 0.3213
Std With No Outliers 0.3069
Ad−Hoc Robust Bilingual German track − Distribution of the Topics of the Experiment
30
Number of Topics of the Experiment
25
20
15
10
5
0
0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Mean Average Precision
Precision averages (%) for individual Ad−Hoc Robust Bilingual German track − Comparison to Median Mean Average Precision by Topic (Topics 041 to 085)
queries 1
deydeRSenFS3456
0.5
041 0.85 086 19.64 131 23.61 166 0.51
Difference
0
042 74.22 087 28.08 132 67.03 167 73.33
−0.5
043 12.73 088 49.46 133 2.00 168 8.44
044 0.00 089 39.46 134 66.33 169 0.00 −1
041 042 043 044 045 046 047 048 049 060 061 062 063 064 065 066 067 068 069 080 081 082 083 084 085
Topic Identifier
045 60.20 090 11.85 135 0.81 170 0.00 Ad−Hoc Robust Bilingual German track − Comparison to Median Mean Average Precision by Topic (Topics 086 to 130)
1
046 21.46 091 0.08 136 62.26 171 8.42 deydeRSenFS3456
047 4.23 092 16.93 137 33.33 172 0.00 0.5
Difference
048 1.05 093 73.77 138 58.38 173 93.19 0
049 4.18 094 7.42 139 0.31 174 86.01 −0.5
060 18.29 095 2.78 140 0.12 175 86.12 −1
086 087 088 089 090 091 092 093 094 095 096 097 098 099 110 111 112 113 114 115 116 117 118 119 130
061 14.98 096 10.97 141 75.00 176 92.15 Topic Identifier
Ad−Hoc Robust Bilingual German track − Comparison to Median Mean Average Precision by Topic (Topics 131 to 165)
062 35.73 097 9.55 142 36.30 177 14.56 1
deydeRSenFS3456
063 54.91 098 43.32 143 15.28 178 0.17 0.5
Difference
064 100.00 099 0.98 144 0.00 179 41.01
0
065 0.06 110 48.99 145 69.09 190 54.39
−0.5
066 77.18 111 4.60 146 0.00 191 0.00
−1
067 9.42 112 64.87 147 16.28 192 7.77 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 160 161 162 163 164 165
Topic Identifier
068 19.11 113 1.90 148 0.26 193 52.57 Ad−Hoc Robust Bilingual German track − Comparison to Median Mean Average Precision by Topic (Topics 166 to 200)
1
069 28.11 114 18.03 149 85.01 194 70.00 deydeRSenFS3456
0.5
080 50.74 115 27.57 160 100.00 195 22.12
Difference
081 58.12 116 65.98 161 0.00 196 77.78 0
082 80.09 117 14.24 162 47.60 197 56.25 −0.5
083 21.06 118 2.34 163 18.77 198 0.01 −1
166 167 168 169 170 171 172 173 174 175 176 177 178 179 190 191 192 193 194 195 196 197 198 199 200
084 0.00 119 46.39 164 0.25 199 75.42 Topic Identifier
085 68.60 130 59.70 165 21.10 200 9.85
267
daedalus deydeRSenFS3456 AH-ROBUST-BILI-X2DE-CLEF2006
Docs Cutoff Levels Precision at DCL (%)
Ad−Hoc Robust Bilingual German track − Retrieved documents vs Precision
100%
5 docs 41.60
deydeRSenFS3456
10 docs 37.00 90%
15 docs 33.47
80%
20 docs 30.85
30 docs 27.37 70%
100 docs 15.30
60%
200 docs 9.82
R−Precision
500 docs 4.77 50%
1000 docs 2.60 40%
R-Precision (precision after R document retrieved,
where R = Relevant retrieved) 30%
33.15
20%
10%
0%
5 10 15 20 30 100 200 500 1000
Retrieved Documents (logarithmic scale)
Exact R-Precision Ad−Hoc Robust Bilingual German track − Box plot of the Topics of the Experiment
Maximum 1.0000
Minimum 0.0000
First Quartile 0.0408
Second Quartile 0.2466
Third Quartile 0.5798
Interquartile range 0.5390
Mean 0.3315
Standard Deviation 0.2900
Lower Outlier Threshold 0.0000
Upper Outlier Threshold 1.0000 0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Exact R−Precision
Mean With No Outliers 0.3315
Std With No Outliers 0.2900
Ad−Hoc Robust Bilingual German track − Distribution of the Topics of the Experiment
30
Number of Topics of the Experiment
25
20
15
10
5
0
0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Exact R−Precision
Precision averages (%) for individual Ad−Hoc Robust Bilingual German track − Comparison to Median Mean Exact R−Precision by Topic (Topics 041 to 085)
queries 1
deydeRSenFS3456
0.5
041 7.14 086 24.32 131 33.33 166 0.00
Difference
0
042 68.87 087 23.53 132 64.29 167 50.00
−0.5
043 0.00 088 56.90 133 7.27 168 11.76
044 0.00 089 46.21 134 57.14 169 0.00 −1
041 042 043 044 045 046 047 048 049 060 061 062 063 064 065 066 067 068 069 080 081 082 083 084 085
Topic Identifier
045 59.43 090 18.75 135 0.00 170 0.00 Ad−Hoc Robust Bilingual German track − Comparison to Median Mean Exact R−Precision by Topic (Topics 086 to 130)
1
046 20.00 091 0.00 136 57.14 171 12.50 deydeRSenFS3456
047 4.17 092 17.65 137 0.00 172 0.00 0.5
Difference
048 3.64 093 75.00 138 64.52 173 80.00 0
049 5.56 094 15.62 139 0.00 174 80.56 −0.5
060 26.32 095 5.45 140 0.00 175 81.82 −1
086 087 088 089 090 091 092 093 094 095 096 097 098 099 110 111 112 113 114 115 116 117 118 119 130
061 27.91 096 18.52 141 75.00 176 87.84 Topic Identifier
Ad−Hoc Robust Bilingual German track − Comparison to Median Mean Exact R−Precision by Topic (Topics 131 to 165)
062 33.33 097 13.33 142 40.00 177 14.29 1
deydeRSenFS3456
063 57.89 098 64.00 143 23.81 178 0.00 0.5
Difference
064 100.00 099 1.12 144 0.00 179 44.90
0
065 0.00 110 54.17 145 70.00 190 57.89
−0.5
066 68.63 111 12.50 146 0.00 191 0.00
−1
067 18.00 112 64.29 147 24.14 192 19.23 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 160 161 162 163 164 165
Topic Identifier
068 18.18 113 4.00 148 0.00 193 52.94 Ad−Hoc Robust Bilingual German track − Comparison to Median Mean Exact R−Precision by Topic (Topics 166 to 200)
1
069 42.86 114 21.43 149 83.33 194 50.00 deydeRSenFS3456
0.5
080 53.33 115 33.33 160 100.00 195 31.03
Difference
081 64.00 116 58.06 161 0.00 196 66.67 0
082 77.78 117 20.00 162 58.33 197 57.38 −0.5
083 33.33 118 0.00 163 29.79 198 0.00 −1
166 167 168 169 170 171 172 173 174 175 176 177 178 179 190 191 192 193 194 195 196 197 198 199 200
084 0.00 119 47.92 164 1.39 199 75.00 Topic Identifier
085 62.16 130 60.00 165 25.00 200 18.18
268
daedalus esx011esRLitFW3FS3 AH-ROBUST-BILI-X2ES-CLEF2006
Overall statistics for 100 queries : Priority 1
Total number of documents over all queries Query Construction AUTOMATIC
Retrieved 100,000 Source Language Italian
Relevant 5,008 Topic Fields title, description, narrative
Relevant retrieved 3,504 Pooled false
Geometric Mean Average Precision 0.0826 Combined run
Binary Preference (BPREF) 0.4224
Interploated Recall (%) Precision Averages (%)
Ad−Hoc Robust Bilingual Spanish track − Interpolated Recall vs Average Precision
100%
0 58.36
esx011esRLitFW3FS3
10 50.66 90%
20 45.70
80%
30 40.62
40 36.79 70%
50 32.32
Average Precision
60%
60 27.58
70 23.52 50%
80 18.27 40%
90 11.89
30%
100 7.04
Average precision (non-interpolated) for all 20%
relevant documents (averaged over queries)
30.68 10%
0%
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
Interpolated Recall
Mean Average Precision Ad−Hoc Robust Bilingual Spanish track − Box plot of the Topics of the Experiment
Maximum 1.0000
Minimum 0.0000
First Quartile 0.0490
Second Quartile 0.2152
Third Quartile 0.5023
Interquartile range 0.4533
Mean 0.3068
Standard Deviation 0.2882
Lower Outlier Threshold 0.0000
Upper Outlier Threshold 1.0000 0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Mean Average Precision
Mean With No Outliers 0.3068
Std With No Outliers 0.2882
Ad−Hoc Robust Bilingual Spanish track − Distribution of the Topics of the Experiment
25
Number of Topics of the Experiment
20
15
10
5
0
0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Mean Average Precision
Precision averages (%) for individual Ad−Hoc Robust Bilingual Spanish track − Comparison to Median Mean Average Precision by Topic (Topics 041 to 085)
queries 1
esx011esRLitFW3FS3
0.5
041 0.48 086 25.18 131 1.07 166 9.61
Difference
0
042 35.25 087 42.78 132 50.25 167 43.43
−0.5
043 45.88 088 20.37 133 8.18 168 13.09
044 11.39 089 47.87 134 100.00 169 0.00 −1
041 042 043 044 045 046 047 048 049 060 061 062 063 064 065 066 067 068 069 080 081 082 083 084 085
Topic Identifier
045 59.67 090 7.50 135 2.27 170 16.29 Ad−Hoc Robust Bilingual Spanish track − Comparison to Median Mean Average Precision by Topic (Topics 086 to 130)
1
046 0.15 091 23.01 136 37.46 171 11.27 esx011esRLitFW3FS3
047 3.38 092 10.74 137 6.45 172 2.19 0.5
Difference
048 8.84 093 21.59 138 96.83 173 62.31 0
049 8.13 094 31.19 139 0.01 174 59.17 −0.5
060 43.15 095 8.83 140 9.52 175 100.00 −1
086 087 088 089 090 091 092 093 094 095 096 097 098 099 110 111 112 113 114 115 116 117 118 119 130
061 0.00 096 0.78 141 85.82 176 80.89 Topic Identifier
Ad−Hoc Robust Bilingual Spanish track − Comparison to Median Mean Average Precision by Topic (Topics 131 to 165)
062 21.02 097 68.15 142 68.39 177 0.00 1
esx011esRLitFW3FS3
063 73.74 098 46.32 143 22.48 178 3.21 0.5
Difference
064 100.00 099 24.46 144 6.51 179 16.45
0
065 23.92 110 71.49 145 59.72 190 1.40
−0.5
066 32.35 111 0.36 146 54.76 191 4.76
−1
067 5.04 112 39.25 147 2.32 192 56.85 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 160 161 162 163 164 165
Topic Identifier
068 0.03 113 1.02 148 14.10 193 9.30 Ad−Hoc Robust Bilingual Spanish track − Comparison to Median Mean Average Precision by Topic (Topics 166 to 200)
1
069 1.50 114 43.03 149 63.74 194 17.35 esx011esRLitFW3FS3
0.5
080 48.80 115 18.05 160 18.25 195 0.00
Difference
081 74.87 116 21.45 161 20.00 196 74.22 0
082 56.03 117 4.42 162 79.14 197 52.66 −0.5
083 30.80 118 2.31 163 50.21 198 41.06 −1
166 167 168 169 170 171 172 173 174 175 176 177 178 179 190 191 192 193 194 195 196 197 198 199 200
084 0.00 119 0.00 164 0.22 199 87.92 Topic Identifier
085 46.76 130 67.62 165 32.37 200 35.41
269
daedalus esx011esRLitFW3FS3 AH-ROBUST-BILI-X2ES-CLEF2006
Docs Cutoff Levels Precision at DCL (%)
Ad−Hoc Robust Bilingual Spanish track − Retrieved documents vs Precision
100%
5 docs 38.80
esx011esRLitFW3FS3
10 docs 34.90 90%
15 docs 32.33
80%
20 docs 29.60
30 docs 26.23 70%
100 docs 16.09
60%
200 docs 10.89
R−Precision
500 docs 6.03 50%
1000 docs 3.50 40%
R-Precision (precision after R document retrieved,
where R = Relevant retrieved) 30%
31.50
20%
10%
0%
5 10 15 20 30 100 200 500 1000
Retrieved Documents (logarithmic scale)
Exact R-Precision Ad−Hoc Robust Bilingual Spanish track − Box plot of the Topics of the Experiment
Maximum 1.0000
Minimum 0.0000
First Quartile 0.0909
Second Quartile 0.2679
Third Quartile 0.5021
Interquartile range 0.4112
Mean 0.3150
Standard Deviation 0.2713
Lower Outlier Threshold 0.0000
Upper Outlier Threshold 1.0000 0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Exact R−Precision
Mean With No Outliers 0.3150
Std With No Outliers 0.2713
Ad−Hoc Robust Bilingual Spanish track − Distribution of the Topics of the Experiment
20
Number of Topics of the Experiment
15
10
5
0
0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Exact R−Precision
Precision averages (%) for individual Ad−Hoc Robust Bilingual Spanish track − Comparison to Median Mean Exact R−Precision by Topic (Topics 041 to 085)
queries 1
esx011esRLitFW3FS3
0.5
041 0.00 086 33.93 131 4.55 166 14.29
Difference
0
042 39.85 087 44.32 132 57.69 167 40.00
−0.5
043 50.00 088 14.29 133 15.79 168 17.95
044 22.50 089 56.25 134 100.00 169 0.00 −1
041 042 043 044 045 046 047 048 049 060 061 062 063 064 065 066 067 068 069 080 081 082 083 084 085
Topic Identifier
045 57.59 090 6.67 135 9.62 170 20.00 Ad−Hoc Robust Bilingual Spanish track − Comparison to Median Mean Exact R−Precision by Topic (Topics 086 to 130)
1
046 0.00 091 29.03 136 50.00 171 25.00 esx011esRLitFW3FS3
047 5.56 092 16.28 137 10.00 172 0.00 0.5
Difference
048 15.62 093 28.57 138 85.71 173 50.00 0
049 10.53 094 31.82 139 0.57 174 54.55 −0.5
060 48.00 095 20.56 140 11.76 175 100.00 −1
086 087 088 089 090 091 092 093 094 095 096 097 098 099 110 111 112 113 114 115 116 117 118 119 130
061 0.00 096 1.08 141 84.62 176 75.32 Topic Identifier
Ad−Hoc Robust Bilingual Spanish track − Comparison to Median Mean Exact R−Precision by Topic (Topics 131 to 165)
062 20.00 097 68.00 142 67.86 177 0.00 1
esx011esRLitFW3FS3
063 69.49 098 40.00 143 32.64 178 10.23 0.5
Difference
064 100.00 099 35.71 144 9.09 179 22.12
0
065 41.89 110 57.14 145 53.19 190 0.00
−0.5
066 39.39 111 0.00 146 50.00 191 0.00
−1
067 10.20 112 37.50 147 11.11 192 54.55 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 160 161 162 163 164 165
Topic Identifier
068 0.00 113 6.33 148 9.09 193 11.76 Ad−Hoc Robust Bilingual Spanish track − Comparison to Median Mean Exact R−Precision by Topic (Topics 166 to 200)
1
069 0.00 114 44.44 149 61.54 194 12.50 esx011esRLitFW3FS3
0.5
080 53.06 115 21.05 160 0.00 195 0.00
Difference
081 68.97 116 14.29 161 20.00 196 73.33 0
082 57.89 117 7.14 162 74.42 197 50.42 −0.5
083 33.33 118 0.00 163 43.33 198 37.50 −1
166 167 168 169 170 171 172 173 174 175 176 177 178 179 190 191 192 193 194 195 196 197 198 199 200
084 0.00 119 0.00 164 1.39 199 82.35 Topic Identifier
085 43.93 130 53.85 165 33.33 200 44.68
270
daedalus esFSesRLit2S AH-ROBUST-BILI-X2ES-CLEF2006
Overall statistics for 100 queries : Priority 4
Total number of documents over all queries Query Construction AUTOMATIC
Retrieved 100,000 Source Language Italian
Relevant 5,008 Topic Fields title, description
Relevant retrieved 3,266 Pooled false
Geometric Mean Average Precision 0.0620 Mandatory TD run
Binary Preference (BPREF) 0.3784
Interploated Recall (%) Precision Averages (%)
Ad−Hoc Robust Bilingual Spanish track − Interpolated Recall vs Average Precision
100%
0 56.71
esFSesRLit2S
10 45.90 90%
20 40.91
80%
30 36.92
40 32.24 70%
50 27.99
Average Precision
60%
60 22.93
70 19.30 50%
80 14.19 40%
90 9.68
30%
100 5.53
Average precision (non-interpolated) for all 20%
relevant documents (averaged over queries)
26.89 10%
0%
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
Interpolated Recall
Mean Average Precision Ad−Hoc Robust Bilingual Spanish track − Box plot of the Topics of the Experiment
Maximum 1.0000
Minimum 0.0000
First Quartile 0.0330
Second Quartile 0.1548
Third Quartile 0.4623
Interquartile range 0.4293
Mean 0.2689
Standard Deviation 0.2779
Lower Outlier Threshold 0.0000
Upper Outlier Threshold 1.0000 0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Mean Average Precision
Mean With No Outliers 0.2689
Std With No Outliers 0.2779
Ad−Hoc Robust Bilingual Spanish track − Distribution of the Topics of the Experiment
30
Number of Topics of the Experiment
25
20
15
10
5
0
0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Mean Average Precision
Precision averages (%) for individual Ad−Hoc Robust Bilingual Spanish track − Comparison to Median Mean Average Precision by Topic (Topics 041 to 085)
queries 1
esFSesRLit2S
0.5
041 0.10 086 11.36 131 1.01 166 3.61
Difference
0
042 18.29 087 36.90 132 62.30 167 43.89
−0.5
043 26.20 088 30.55 133 5.48 168 14.42
044 10.54 089 45.34 134 100.00 169 0.00 −1
041 042 043 044 045 046 047 048 049 060 061 062 063 064 065 066 067 068 069 080 081 082 083 084 085
Topic Identifier
045 65.23 090 0.82 135 2.19 170 11.28 Ad−Hoc Robust Bilingual Spanish track − Comparison to Median Mean Average Precision by Topic (Topics 086 to 130)
1
046 0.13 091 23.39 136 34.41 171 15.26 esFSesRLit2S
047 0.40 092 7.69 137 8.75 172 4.17 0.5
Difference
048 10.82 093 15.74 138 86.69 173 41.23 0
049 11.22 094 23.98 139 0.00 174 34.31 −0.5
060 48.30 095 5.18 140 2.86 175 100.00 −1
086 087 088 089 090 091 092 093 094 095 096 097 098 099 110 111 112 113 114 115 116 117 118 119 130
061 0.00 096 1.02 141 51.48 176 70.43 Topic Identifier
Ad−Hoc Robust Bilingual Spanish track − Comparison to Median Mean Average Precision by Topic (Topics 131 to 165)
062 21.74 097 68.55 142 66.22 177 0.00 1
esFSesRLit2S
063 65.50 098 35.41 143 18.25 178 2.46 0.5
Difference
064 100.00 099 28.78 144 5.91 179 10.92
0
065 6.48 110 63.10 145 57.29 190 1.49
−0.5
066 28.62 111 0.27 146 50.90 191 5.26
−1
067 15.38 112 40.35 147 5.71 192 35.29 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 160 161 162 163 164 165
Topic Identifier
068 0.00 113 0.02 148 12.45 193 6.07 Ad−Hoc Robust Bilingual Spanish track − Comparison to Median Mean Average Precision by Topic (Topics 166 to 200)
1
069 0.28 114 2.99 149 22.87 194 10.67 esFSesRLit2S
0.5
080 49.70 115 20.53 160 50.82 195 0.00
Difference
081 75.83 116 10.93 161 0.67 196 75.74 0
082 64.10 117 14.65 162 82.80 197 62.46 −0.5
083 30.79 118 0.69 163 45.68 198 46.79 −1
166 167 168 169 170 171 172 173 174 175 176 177 178 179 190 191 192 193 194 195 196 197 198 199 200
084 0.00 119 0.10 164 0.36 199 83.34 Topic Identifier
085 53.54 130 15.58 165 23.21 200 4.25
271
daedalus esFSesRLit2S AH-ROBUST-BILI-X2ES-CLEF2006
Docs Cutoff Levels Precision at DCL (%)
Ad−Hoc Robust Bilingual Spanish track − Retrieved documents vs Precision
100%
5 docs 34.20
esFSesRLit2S
10 docs 30.30 90%
15 docs 28.07
80%
20 docs 25.60
30 docs 23.13 70%
100 docs 14.87
60%
200 docs 10.22
R−Precision
500 docs 5.52 50%
1000 docs 3.27 40%
R-Precision (precision after R document retrieved,
where R = Relevant retrieved) 30%
28.75
20%
10%
0%
5 10 15 20 30 100 200 500 1000
Retrieved Documents (logarithmic scale)
Exact R-Precision Ad−Hoc Robust Bilingual Spanish track − Box plot of the Topics of the Experiment
Maximum 1.0000
Minimum 0.0000
First Quartile 0.0592
Second Quartile 0.2315
Third Quartile 0.5000
Interquartile range 0.4408
Mean 0.2875
Standard Deviation 0.2624
Lower Outlier Threshold 0.0000
Upper Outlier Threshold 1.0000 0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Exact R−Precision
Mean With No Outliers 0.2875
Std With No Outliers 0.2624
Ad−Hoc Robust Bilingual Spanish track − Distribution of the Topics of the Experiment
25
Number of Topics of the Experiment
20
15
10
5
0
0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Exact R−Precision
Precision averages (%) for individual Ad−Hoc Robust Bilingual Spanish track − Comparison to Median Mean Exact R−Precision by Topic (Topics 041 to 085)
queries 1
esFSesRLit2S
0.5
041 0.00 086 23.21 131 5.45 166 0.00
Difference
0
042 27.97 087 42.05 132 53.85 167 40.00
−0.5
043 27.78 088 28.57 133 15.79 168 23.08
044 17.50 089 43.75 134 100.00 169 0.00 −1
041 042 043 044 045 046 047 048 049 060 061 062 063 064 065 066 067 068 069 080 081 082 083 084 085
Topic Identifier
045 65.45 090 0.00 135 8.97 170 20.00 Ad−Hoc Robust Bilingual Spanish track − Comparison to Median Mean Exact R−Precision by Topic (Topics 086 to 130)
1
046 0.00 091 27.96 136 50.00 171 25.00 esFSesRLit2S
047 2.78 092 16.28 137 10.00 172 7.69 0.5
Difference
048 16.67 093 21.43 138 71.43 173 33.33 0
049 10.53 094 45.45 139 0.57 174 36.36 −0.5
060 53.33 095 15.58 140 8.40 175 100.00 −1
086 087 088 089 090 091 092 093 094 095 096 097 098 099 110 111 112 113 114 115 116 117 118 119 130
061 0.00 096 2.15 141 53.85 176 68.83 Topic Identifier
Ad−Hoc Robust Bilingual Spanish track − Comparison to Median Mean Exact R−Precision by Topic (Topics 131 to 165)
062 20.00 097 60.00 142 67.86 177 0.00 1
esFSesRLit2S
063 64.41 098 40.00 143 24.31 178 7.95 0.5
Difference
064 100.00 099 25.00 144 9.09 179 13.27
0
065 16.22 110 57.14 145 53.19 190 0.00
−0.5
066 38.38 111 0.00 146 50.00 191 0.00
−1
067 22.45 112 37.50 147 11.11 192 45.45 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 160 161 162 163 164 165
Topic Identifier
068 0.00 113 0.63 148 9.09 193 11.76 Ad−Hoc Robust Bilingual Spanish track − Comparison to Median Mean Exact R−Precision by Topic (Topics 166 to 200)
1
069 0.00 114 2.22 149 23.08 194 25.00 esFSesRLit2S
0.5
080 53.47 115 21.05 160 50.00 195 0.00
Difference
081 72.41 116 0.00 161 0.00 196 73.33 0
082 65.79 117 23.21 162 76.74 197 55.42 −0.5
083 33.33 118 0.00 163 50.00 198 37.50 −1
166 167 168 169 170 171 172 173 174 175 176 177 178 179 190 191 192 193 194 195 196 197 198 199 200
084 0.00 119 1.90 164 0.00 199 77.65 Topic Identifier
085 56.07 130 30.77 165 33.33 200 6.38
272
daedalus esFSesRLit3S AH-ROBUST-BILI-X2ES-CLEF2006
Overall statistics for 100 queries : Priority 3
Total number of documents over all queries Query Construction AUTOMATIC
Retrieved 100,000 Source Language Italian
Relevant 5,008 Topic Fields title, description, narrative
Relevant retrieved 3,504 Pooled false
Geometric Mean Average Precision 0.0833 TDN run
Binary Preference (BPREF) 0.4129
Interploated Recall (%) Precision Averages (%)
Ad−Hoc Robust Bilingual Spanish track − Interpolated Recall vs Average Precision
100%
0 61.40
esFSesRLit3S
10 50.75 90%
20 45.51
80%
30 40.19
40 36.34 70%
50 31.85
Average Precision
60%
60 26.93
70 22.86 50%
80 17.61 40%
90 11.23
30%
100 6.37
Average precision (non-interpolated) for all 20%
relevant documents (averaged over queries)
30.37 10%
0%
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
Interpolated Recall
Mean Average Precision Ad−Hoc Robust Bilingual Spanish track − Box plot of the Topics of the Experiment
Maximum 1.0000
Minimum 0.0000
First Quartile 0.0493
Second Quartile 0.2249
Third Quartile 0.4950
Interquartile range 0.4457
Mean 0.3037
Standard Deviation 0.2790
Lower Outlier Threshold 0.0000
Upper Outlier Threshold 1.0000 0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Mean Average Precision
Mean With No Outliers 0.3037
Std With No Outliers 0.2790
Ad−Hoc Robust Bilingual Spanish track − Distribution of the Topics of the Experiment
25
Number of Topics of the Experiment
20
15
10
5
0
0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Mean Average Precision
Precision averages (%) for individual Ad−Hoc Robust Bilingual Spanish track − Comparison to Median Mean Average Precision by Topic (Topics 041 to 085)
queries 1
esFSesRLit3S
0.5
041 0.48 086 26.37 131 1.09 166 9.61
Difference
0
042 35.25 087 42.40 132 57.42 167 43.43
−0.5
043 45.88 088 22.75 133 8.18 168 13.09
044 13.14 089 47.87 134 100.00 169 0.00 −1
041 042 043 044 045 046 047 048 049 060 061 062 063 064 065 066 067 068 069 080 081 082 083 084 085
Topic Identifier
045 59.67 090 8.61 135 2.38 170 19.62 Ad−Hoc Robust Bilingual Spanish track − Comparison to Median Mean Average Precision by Topic (Topics 086 to 130)
1
046 0.15 091 23.01 136 37.46 171 11.27 esFSesRLit3S
047 3.54 092 10.74 137 11.45 172 2.19 0.5
Difference
048 8.87 093 21.59 138 96.83 173 62.31 0
049 10.76 094 31.19 139 0.01 174 59.17 −0.5
060 42.82 095 8.83 140 9.52 175 100.00 −1
086 087 088 089 090 091 092 093 094 095 096 097 098 099 110 111 112 113 114 115 116 117 118 119 130
061 0.00 096 0.78 141 85.82 176 80.60 Topic Identifier
Ad−Hoc Robust Bilingual Spanish track − Comparison to Median Mean Average Precision by Topic (Topics 131 to 165)
062 21.02 097 68.15 142 68.39 177 0.00 1
esFSesRLit3S
063 73.74 098 46.32 143 22.48 178 3.21 0.5
Difference
064 33.33 099 28.63 144 6.51 179 17.04
0
065 22.51 110 71.49 145 59.72 190 1.40
−0.5
066 31.06 111 0.36 146 54.76 191 4.76
−1
067 5.11 112 40.30 147 2.32 192 55.71 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 160 161 162 163 164 165
Topic Identifier
068 0.03 113 1.02 148 14.10 193 10.77 Ad−Hoc Robust Bilingual Spanish track − Comparison to Median Mean Average Precision by Topic (Topics 166 to 200)
1
069 1.50 114 43.03 149 63.74 194 9.01 esFSesRLit3S
0.5
080 48.80 115 20.68 160 26.59 195 0.00
Difference
081 74.87 116 21.45 161 20.00 196 74.22 0
082 54.71 117 4.42 162 79.14 197 52.66 −0.5
083 30.80 118 2.31 163 50.21 198 41.06 −1
166 167 168 169 170 171 172 173 174 175 176 177 178 179 190 191 192 193 194 195 196 197 198 199 200
084 0.00 119 0.00 164 0.22 199 87.92 Topic Identifier
085 46.76 130 67.62 165 34.96 200 40.20
273
daedalus esFSesRLit3S AH-ROBUST-BILI-X2ES-CLEF2006
Docs Cutoff Levels Precision at DCL (%)
Ad−Hoc Robust Bilingual Spanish track − Retrieved documents vs Precision
100%
5 docs 39.20
esFSesRLit3S
10 docs 35.00 90%
15 docs 32.13
80%
20 docs 29.55
30 docs 26.23 70%
100 docs 16.09
60%
200 docs 10.89
R−Precision
500 docs 6.03 50%
1000 docs 3.50 40%
R-Precision (precision after R document retrieved,
where R = Relevant retrieved) 30%
31.00
20%
10%
0%
5 10 15 20 30 100 200 500 1000
Retrieved Documents (logarithmic scale)
Exact R-Precision Ad−Hoc Robust Bilingual Spanish track − Box plot of the Topics of the Experiment
Maximum 1.0000
Minimum 0.0000
First Quartile 0.0909
Second Quartile 0.2679
Third Quartile 0.5000
Interquartile range 0.4091
Mean 0.3100
Standard Deviation 0.2630
Lower Outlier Threshold 0.0000
Upper Outlier Threshold 1.0000 0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Exact R−Precision
Mean With No Outliers 0.3100
Std With No Outliers 0.2630
Ad−Hoc Robust Bilingual Spanish track − Distribution of the Topics of the Experiment
20
Number of Topics of the Experiment
15
10
5
0
0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Exact R−Precision
Precision averages (%) for individual Ad−Hoc Robust Bilingual Spanish track − Comparison to Median Mean Exact R−Precision by Topic (Topics 041 to 085)
queries 1
esFSesRLit3S
0.5
041 0.00 086 33.93 131 4.55 166 14.29
Difference
0
042 39.85 087 44.32 132 57.69 167 40.00
−0.5
043 50.00 088 14.29 133 15.79 168 17.95
044 22.50 089 56.25 134 100.00 169 0.00 −1
041 042 043 044 045 046 047 048 049 060 061 062 063 064 065 066 067 068 069 080 081 082 083 084 085
Topic Identifier
045 57.59 090 6.67 135 9.62 170 20.00 Ad−Hoc Robust Bilingual Spanish track − Comparison to Median Mean Exact R−Precision by Topic (Topics 086 to 130)
1
046 0.00 091 29.03 136 50.00 171 25.00 esFSesRLit3S
047 5.56 092 16.28 137 10.00 172 0.00 0.5
Difference
048 15.62 093 28.57 138 85.71 173 50.00 0
049 10.53 094 31.82 139 0.57 174 54.55 −0.5
060 48.00 095 20.56 140 11.76 175 100.00 −1
086 087 088 089 090 091 092 093 094 095 096 097 098 099 110 111 112 113 114 115 116 117 118 119 130
061 0.00 096 1.08 141 84.62 176 75.32 Topic Identifier
Ad−Hoc Robust Bilingual Spanish track − Comparison to Median Mean Exact R−Precision by Topic (Topics 131 to 165)
062 20.00 097 68.00 142 67.86 177 0.00 1
esFSesRLit3S
063 69.49 098 40.00 143 32.64 178 10.23 0.5
Difference
064 0.00 099 35.71 144 9.09 179 22.12
0
065 41.89 110 57.14 145 53.19 190 0.00
−0.5
066 39.39 111 0.00 146 50.00 191 0.00
−1
067 10.20 112 37.50 147 11.11 192 54.55 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 160 161 162 163 164 165
Topic Identifier
068 0.00 113 6.33 148 9.09 193 11.76 Ad−Hoc Robust Bilingual Spanish track − Comparison to Median Mean Exact R−Precision by Topic (Topics 166 to 200)
1
069 0.00 114 44.44 149 61.54 194 12.50 esFSesRLit3S
0.5
080 53.06 115 21.05 160 50.00 195 0.00
Difference
081 68.97 116 14.29 161 20.00 196 73.33 0
082 57.89 117 7.14 162 74.42 197 50.42 −0.5
083 33.33 118 0.00 163 43.33 198 37.50 −1
166 167 168 169 170 171 172 173 174 175 176 177 178 179 190 191 192 193 194 195 196 197 198 199 200
084 0.00 119 0.00 164 1.39 199 82.35 Topic Identifier
085 43.93 130 53.85 165 33.33 200 44.68
274
daedalus esx021esRLitFW3FS3 AH-ROBUST-BILI-X2ES-CLEF2006
Overall statistics for 100 queries : Priority 2
Total number of documents over all queries Query Construction AUTOMATIC
Retrieved 100,000 Source Language Italian
Relevant 5,008 Topic Fields title, description, narrative
Relevant retrieved 3,504 Pooled false
Geometric Mean Average Precision 0.0812 Combined run
Binary Preference (BPREF) 0.4184
Interploated Recall (%) Precision Averages (%)
Ad−Hoc Robust Bilingual Spanish track − Interpolated Recall vs Average Precision
100%
0 57.19
esx021esRLitFW3FS3
10 50.34 90%
20 45.24
80%
30 40.32
40 36.65 70%
50 32.18
Average Precision
60%
60 27.57
70 23.52 50%
80 18.27 40%
90 11.89
30%
100 7.04
Average precision (non-interpolated) for all 20%
relevant documents (averaged over queries)
30.42 10%
0%
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
Interpolated Recall
Mean Average Precision Ad−Hoc Robust Bilingual Spanish track − Box plot of the Topics of the Experiment
Maximum 1.0000
Minimum 0.0000
First Quartile 0.0477
Second Quartile 0.2179
Third Quartile 0.4968
Interquartile range 0.4490
Mean 0.3042
Standard Deviation 0.2886
Lower Outlier Threshold 0.0000
Upper Outlier Threshold 1.0000 0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Mean Average Precision
Mean With No Outliers 0.3042
Std With No Outliers 0.2886
Ad−Hoc Robust Bilingual Spanish track − Distribution of the Topics of the Experiment
30
Number of Topics of the Experiment
25
20
15
10
5
0
0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Mean Average Precision
Precision averages (%) for individual Ad−Hoc Robust Bilingual Spanish track − Comparison to Median Mean Average Precision by Topic (Topics 041 to 085)
queries 1
esx021esRLitFW3FS3
0.5
041 0.48 086 24.70 131 1.05 166 8.87
Difference
0
042 35.25 087 43.06 132 46.18 167 43.43
−0.5
043 45.88 088 20.37 133 8.18 168 13.09
044 10.72 089 44.64 134 100.00 169 0.00 −1
041 042 043 044 045 046 047 048 049 060 061 062 063 064 065 066 067 068 069 080 081 082 083 084 085
Topic Identifier
045 59.10 090 6.95 135 2.38 170 14.62 Ad−Hoc Robust Bilingual Spanish track − Comparison to Median Mean Average Precision by Topic (Topics 086 to 130)
1
046 0.15 091 23.01 136 39.54 171 11.27 esx021esRLitFW3FS3
047 3.38 092 10.74 137 4.79 172 2.19 0.5
Difference
048 8.84 093 19.21 138 96.83 173 62.31 0
049 7.25 094 31.19 139 0.01 174 59.17 −0.5
060 42.38 095 8.71 140 9.52 175 100.00 −1
086 087 088 089 090 091 092 093 094 095 096 097 098 099 110 111 112 113 114 115 116 117 118 119 130
061 0.00 096 0.78 141 85.82 176 80.89 Topic Identifier
Ad−Hoc Robust Bilingual Spanish track − Comparison to Median Mean Average Precision by Topic (Topics 131 to 165)
062 22.14 097 68.15 142 68.39 177 0.00 1
esx021esRLitFW3FS3
063 73.74 098 46.32 143 22.59 178 3.19 0.5
Difference
064 100.00 099 22.26 144 5.45 179 16.20
0
065 23.92 110 71.49 145 59.72 190 1.40
−0.5
066 31.50 111 0.36 146 54.76 191 4.76
−1
067 4.96 112 39.25 147 2.20 192 57.61 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 160 161 162 163 164 165
Topic Identifier
068 0.03 113 1.02 148 14.10 193 8.81 Ad−Hoc Robust Bilingual Spanish track − Comparison to Median Mean Average Precision by Topic (Topics 166 to 200)
1
069 1.50 114 43.67 149 63.74 194 17.35 esx021esRLitFW3FS3
0.5
080 49.14 115 20.65 160 14.09 195 0.00
Difference
081 74.87 116 21.45 161 20.00 196 74.22 0
082 56.91 117 4.42 162 79.14 197 52.74 −0.5
083 30.80 118 2.31 163 50.21 198 36.55 −1
166 167 168 169 170 171 172 173 174 175 176 177 178 179 190 191 192 193 194 195 196 197 198 199 200
084 0.00 119 0.00 164 0.22 199 87.92 Topic Identifier
085 46.21 130 67.62 165 32.37 200 32.89
275
daedalus esx021esRLitFW3FS3 AH-ROBUST-BILI-X2ES-CLEF2006
Docs Cutoff Levels Precision at DCL (%)
Ad−Hoc Robust Bilingual Spanish track − Retrieved documents vs Precision
100%
5 docs 38.00
esx021esRLitFW3FS3
10 docs 34.70 90%
15 docs 32.27
80%
20 docs 29.50
30 docs 26.20 70%
100 docs 16.09
60%
200 docs 10.89
R−Precision
500 docs 6.03 50%
1000 docs 3.50 40%
R-Precision (precision after R document retrieved,
where R = Relevant retrieved) 30%
31.39
20%
10%
0%
5 10 15 20 30 100 200 500 1000
Retrieved Documents (logarithmic scale)
Exact R-Precision Ad−Hoc Robust Bilingual Spanish track − Box plot of the Topics of the Experiment
Maximum 1.0000
Minimum 0.0000
First Quartile 0.0812
Second Quartile 0.2679
Third Quartile 0.5021
Interquartile range 0.4209
Mean 0.3139
Standard Deviation 0.2724
Lower Outlier Threshold 0.0000
Upper Outlier Threshold 1.0000 0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Exact R−Precision
Mean With No Outliers 0.3139
Std With No Outliers 0.2724
Ad−Hoc Robust Bilingual Spanish track − Distribution of the Topics of the Experiment
25
Number of Topics of the Experiment
20
15
10
5
0
0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Exact R−Precision
Precision averages (%) for individual Ad−Hoc Robust Bilingual Spanish track − Comparison to Median Mean Exact R−Precision by Topic (Topics 041 to 085)
queries 1
esx021esRLitFW3FS3
0.5
041 0.00 086 33.93 131 4.55 166 14.29
Difference
0
042 39.85 087 44.32 132 57.69 167 40.00
−0.5
043 50.00 088 14.29 133 15.79 168 17.95
044 22.50 089 56.25 134 100.00 169 0.00 −1
041 042 043 044 045 046 047 048 049 060 061 062 063 064 065 066 067 068 069 080 081 082 083 084 085
Topic Identifier
045 57.59 090 6.67 135 9.62 170 20.00 Ad−Hoc Robust Bilingual Spanish track − Comparison to Median Mean Exact R−Precision by Topic (Topics 086 to 130)
1
046 0.00 091 29.03 136 50.00 171 25.00 esx021esRLitFW3FS3
047 5.56 092 16.28 137 10.00 172 0.00 0.5
Difference
048 15.62 093 28.57 138 85.71 173 50.00 0
049 10.53 094 31.82 139 0.57 174 54.55 −0.5
060 48.00 095 20.56 140 11.76 175 100.00 −1
086 087 088 089 090 091 092 093 094 095 096 097 098 099 110 111 112 113 114 115 116 117 118 119 130
061 0.00 096 1.08 141 84.62 176 75.32 Topic Identifier
Ad−Hoc Robust Bilingual Spanish track − Comparison to Median Mean Exact R−Precision by Topic (Topics 131 to 165)
062 20.00 097 68.00 142 67.86 177 0.00 1
esx021esRLitFW3FS3
063 69.49 098 40.00 143 32.64 178 10.23 0.5
Difference
064 100.00 099 35.71 144 9.09 179 22.12
0
065 41.89 110 57.14 145 53.19 190 0.00
−0.5
066 39.39 111 0.00 146 50.00 191 0.00
−1
067 10.20 112 37.50 147 0.00 192 54.55 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 160 161 162 163 164 165
Topic Identifier
068 0.00 113 6.33 148 9.09 193 11.76 Ad−Hoc Robust Bilingual Spanish track − Comparison to Median Mean Exact R−Precision by Topic (Topics 166 to 200)
1
069 0.00 114 44.44 149 61.54 194 12.50 esx021esRLitFW3FS3
0.5
080 53.06 115 21.05 160 0.00 195 0.00
Difference
081 68.97 116 14.29 161 20.00 196 73.33 0
082 57.89 117 7.14 162 74.42 197 50.42 −0.5
083 33.33 118 0.00 163 43.33 198 37.50 −1
166 167 168 169 170 171 172 173 174 175 176 177 178 179 190 191 192 193 194 195 196 197 198 199 200
084 0.00 119 0.00 164 1.39 199 82.35 Topic Identifier
085 43.93 130 53.85 165 33.33 200 44.68
276
dcu dcuitqynarrsp1005 AH-ROBUST-BILI-X2ES-CLEF2006
Overall statistics for 100 queries : Priority 1
Total number of documents over all queries Query Construction AUTOMATIC
Retrieved 100,000 Source Language Italian
Relevant 5,008 Topic Fields title, description, narrative
Relevant retrieved 3,856 Pooled false
Geometric Mean Average Precision 0.1117 spanish bilingual standard
Binary Preference (BPREF) 0.4735
Interploated Recall (%) Precision Averages (%)
Ad−Hoc Robust Bilingual Spanish track − Interpolated Recall vs Average Precision
100%
0 62.34
dcuitqynarrsp1005
10 54.70 90%
20 48.41
80%
30 43.49
40 39.59 70%
50 36.11
Average Precision
60%
60 32.50
70 28.49 50%
80 24.06 40%
90 16.66
30%
100 9.83
Average precision (non-interpolated) for all 20%
relevant documents (averaged over queries)
34.65 10%
0%
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
Interpolated Recall
Mean Average Precision Ad−Hoc Robust Bilingual Spanish track − Box plot of the Topics of the Experiment
Maximum 1.0000
Minimum 0.0000
First Quartile 0.0652
Second Quartile 0.2581
Third Quartile 0.5831
Interquartile range 0.5179
Mean 0.3465
Standard Deviation 0.3098
Lower Outlier Threshold 0.0000
Upper Outlier Threshold 1.0000 0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Mean Average Precision
Mean With No Outliers 0.3465
Std With No Outliers 0.3098
Ad−Hoc Robust Bilingual Spanish track − Distribution of the Topics of the Experiment
25
Number of Topics of the Experiment
20
15
10
5
0
0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Mean Average Precision
Precision averages (%) for individual Ad−Hoc Robust Bilingual Spanish track − Comparison to Median Mean Average Precision by Topic (Topics 041 to 085)
queries 1
dcuitqynarrsp1005
0.5
041 2.49 086 21.67 131 0.27 166 0.94
Difference
0
042 50.65 087 58.71 132 0.29 167 51.98
−0.5
043 4.28 088 30.71 133 22.98 168 23.28
044 17.58 089 52.39 134 91.67 169 0.00 −1
041 042 043 044 045 046 047 048 049 060 061 062 063 064 065 066 067 068 069 080 081 082 083 084 085
Topic Identifier
045 66.05 090 1.05 135 22.49 170 88.33 Ad−Hoc Robust Bilingual Spanish track − Comparison to Median Mean Average Precision by Topic (Topics 086 to 130)
1
046 2.33 091 7.76 136 36.11 171 9.61 dcuitqynarrsp1005
047 10.97 092 21.91 137 1.26 172 6.98 0.5
Difference
048 7.18 093 22.36 138 100.00 173 67.16 0
049 15.52 094 58.09 139 1.85 174 87.38 −0.5
060 20.98 095 10.80 140 26.67 175 100.00 −1
086 087 088 089 090 091 092 093 094 095 096 097 098 099 110 111 112 113 114 115 116 117 118 119 130
061 0.00 096 2.69 141 88.06 176 97.74 Topic Identifier
Ad−Hoc Robust Bilingual Spanish track − Comparison to Median Mean Average Precision by Topic (Topics 131 to 165)
062 34.12 097 66.99 142 90.25 177 0.76 1
dcuitqynarrsp1005
063 80.59 098 54.32 143 58.53 178 26.34 0.5
Difference
064 1.33 099 0.62 144 36.89 179 1.56
0
065 20.21 110 69.48 145 55.27 190 0.26
−0.5
066 33.80 111 1.08 146 29.17 191 50.00
−1
067 6.61 112 54.38 147 12.89 192 64.58 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 160 161 162 163 164 165
Topic Identifier
068 0.00 113 1.53 148 25.28 193 8.12 Ad−Hoc Robust Bilingual Spanish track − Comparison to Median Mean Average Precision by Topic (Topics 166 to 200)
1
069 1.98 114 64.60 149 79.40 194 9.91 dcuitqynarrsp1005
0.5
080 38.36 115 18.14 160 0.13 195 0.00
Difference
081 38.29 116 39.29 161 20.08 196 69.25 0
082 82.35 117 6.43 162 89.33 197 44.92 −0.5
083 53.51 118 34.05 163 80.97 198 20.96 −1
166 167 168 169 170 171 172 173 174 175 176 177 178 179 190 191 192 193 194 195 196 197 198 199 200
084 0.15 119 12.64 164 27.91 199 88.33 Topic Identifier
085 47.93 130 80.99 165 66.11 200 52.30
277
dcu dcuitqynarrsp1005 AH-ROBUST-BILI-X2ES-CLEF2006
Docs Cutoff Levels Precision at DCL (%)
Ad−Hoc Robust Bilingual Spanish track − Retrieved documents vs Precision
100%
5 docs 42.00
dcuitqynarrsp1005
10 docs 40.10 90%
15 docs 36.87
80%
20 docs 33.45
30 docs 29.13 70%
100 docs 17.63
60%
200 docs 11.90
R−Precision
500 docs 6.60 50%
1000 docs 3.86 40%
R-Precision (precision after R document retrieved,
where R = Relevant retrieved) 30%
34.17
20%
10%
0%
5 10 15 20 30 100 200 500 1000
Retrieved Documents (logarithmic scale)
Exact R-Precision Ad−Hoc Robust Bilingual Spanish track − Box plot of the Topics of the Experiment
Maximum 1.0000
Minimum 0.0000
First Quartile 0.0629
Second Quartile 0.3195
Third Quartile 0.5556
Interquartile range 0.4927
Mean 0.3417
Standard Deviation 0.2880
Lower Outlier Threshold 0.0000
Upper Outlier Threshold 1.0000 0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Exact R−Precision
Mean With No Outliers 0.3417
Std With No Outliers 0.2880
Ad−Hoc Robust Bilingual Spanish track − Distribution of the Topics of the Experiment
25
Number of Topics of the Experiment
20
15
10
5
0
0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Exact R−Precision
Precision averages (%) for individual Ad−Hoc Robust Bilingual Spanish track − Comparison to Median Mean Exact R−Precision by Topic (Topics 041 to 085)
queries 1
dcuitqynarrsp1005
0.5
041 6.25 086 28.57 131 1.82 166 0.00
Difference
0
042 49.81 087 57.95 132 0.00 167 50.00
−0.5
043 5.56 088 14.29 133 31.58 168 25.64
044 25.00 089 56.25 134 75.00 169 0.00 −1
041 042 043 044 045 046 047 048 049 060 061 062 063 064 065 066 067 068 069 080 081 082 083 084 085
Topic Identifier
045 67.54 090 0.00 135 32.69 170 80.00 Ad−Hoc Robust Bilingual Spanish track − Comparison to Median Mean Exact R−Precision by Topic (Topics 086 to 130)
1
046 0.00 091 13.98 136 50.00 171 0.00 dcuitqynarrsp1005
047 16.67 092 25.58 137 0.00 172 7.69 0.5
Difference
048 12.50 093 21.43 138 100.00 173 33.33 0
049 21.05 094 68.18 139 2.30 174 81.82 −0.5
060 33.33 095 21.81 140 36.13 175 100.00 −1
086 087 088 089 090 091 092 093 094 095 096 097 098 099 110 111 112 113 114 115 116 117 118 119 130
061 0.00 096 8.60 141 84.62 176 92.21 Topic Identifier
Ad−Hoc Robust Bilingual Spanish track − Comparison to Median Mean Exact R−Precision by Topic (Topics 131 to 165)
062 26.67 097 60.00 142 85.71 177 0.00 1
dcuitqynarrsp1005
063 77.97 098 40.00 143 52.78 178 35.23 0.5
Difference
064 0.00 099 0.00 144 54.55 179 3.54
0
065 36.49 110 57.14 145 48.94 190 0.00
−0.5
066 32.32 111 0.00 146 50.00 191 0.00
−1
067 6.12 112 50.00 147 11.11 192 61.36 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 160 161 162 163 164 165
Topic Identifier
068 0.00 113 6.33 148 27.27 193 8.82 Ad−Hoc Robust Bilingual Spanish track − Comparison to Median Mean Exact R−Precision by Topic (Topics 166 to 200)
1
069 0.00 114 55.56 149 69.23 194 12.50 dcuitqynarrsp1005
0.5
080 41.63 115 26.32 160 0.00 195 0.00
Difference
081 37.93 116 42.86 161 20.00 196 66.67 0
082 84.21 117 8.93 162 81.40 197 47.92 −0.5
083 55.56 118 20.00 163 73.33 198 37.50 −1
166 167 168 169 170 171 172 173 174 175 176 177 178 179 190 191 192 193 194 195 196 197 198 199 200
084 0.00 119 21.90 164 36.11 199 82.35 Topic Identifier
085 42.06 130 61.54 165 66.67 200 53.19
278
dcu dcuitqydescsp12075 AH-ROBUST-BILI-X2ES-CLEF2006
Overall statistics for 100 queries : Priority 2
Total number of documents over all queries Query Construction AUTOMATIC
Retrieved 100,000 Source Language Italian
Relevant 5,008 Topic Fields title, description
Relevant retrieved 3,759 Pooled false
Geometric Mean Average Precision 0.1044 spanish bilingual with desc+title
Binary Preference (BPREF) 0.4587
Interploated Recall (%) Precision Averages (%)
Ad−Hoc Robust Bilingual Spanish track − Interpolated Recall vs Average Precision
100%
0 59.97
dcuitqydescsp12075
10 52.91 90%
20 46.68
80%
30 42.72
40 39.25 70%
50 34.59
Average Precision
60%
60 30.34
70 26.65 50%
80 21.88 40%
90 15.67
30%
100 9.27
Average precision (non-interpolated) for all 20%
relevant documents (averaged over queries)
33.22 10%
0%
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
Interpolated Recall
Mean Average Precision Ad−Hoc Robust Bilingual Spanish track − Box plot of the Topics of the Experiment
Maximum 1.0000
Minimum 0.0000
First Quartile 0.0485
Second Quartile 0.2314
Third Quartile 0.5543
Interquartile range 0.5058
Mean 0.3322
Standard Deviation 0.3084
Lower Outlier Threshold 0.0000
Upper Outlier Threshold 1.0000 0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Mean Average Precision
Mean With No Outliers 0.3322
Std With No Outliers 0.3084
Ad−Hoc Robust Bilingual Spanish track − Distribution of the Topics of the Experiment
30
Number of Topics of the Experiment
25
20
15
10
5
0
0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Mean Average Precision
Precision averages (%) for individual Ad−Hoc Robust Bilingual Spanish track − Comparison to Median Mean Average Precision by Topic (Topics 041 to 085)
queries 1
dcuitqydescsp12075
0.5
041 0.89 086 22.79 131 0.22 166 3.62
Difference
0
042 39.37 087 48.89 132 0.48 167 46.79
−0.5
043 4.83 088 36.21 133 29.60 168 17.24
044 17.53 089 52.50 134 100.00 169 0.00 −1
041 042 043 044 045 046 047 048 049 060 061 062 063 064 065 066 067 068 069 080 081 082 083 084 085
Topic Identifier
045 50.68 090 0.98 135 18.24 170 83.45 Ad−Hoc Robust Bilingual Spanish track − Comparison to Median Mean Average Precision by Topic (Topics 086 to 130)
1
046 0.64 091 8.17 136 39.50 171 5.67 dcuitqydescsp12075
047 4.83 092 20.10 137 13.96 172 7.47 0.5
Difference
048 8.94 093 23.50 138 100.00 173 67.16 0
049 12.96 094 63.99 139 1.37 174 89.08 −0.5
060 23.95 095 10.42 140 25.54 175 100.00 −1
086 087 088 089 090 091 092 093 094 095 096 097 098 099 110 111 112 113 114 115 116 117 118 119 130
061 0.00 096 1.22 141 87.42 176 96.70 Topic Identifier
Ad−Hoc Robust Bilingual Spanish track − Comparison to Median Mean Average Precision by Topic (Topics 131 to 165)
062 27.78 097 68.49 142 89.52 177 0.43 1
dcuitqydescsp12075
063 75.77 098 48.54 143 31.53 178 17.38 0.5
Difference
064 0.35 099 2.05 144 19.93 179 1.49
0
065 21.94 110 69.05 145 52.79 190 1.00
−0.5
066 19.95 111 0.29 146 27.38 191 33.33
−1
067 6.71 112 43.24 147 4.66 192 58.08 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 160 161 162 163 164 165
Topic Identifier
068 0.00 113 1.73 148 28.55 193 6.77 Ad−Hoc Robust Bilingual Spanish track − Comparison to Median Mean Average Precision by Topic (Topics 166 to 200)
1
069 1.77 114 65.09 149 74.93 194 4.87 dcuitqydescsp12075
0.5
080 36.40 115 21.88 160 0.06 195 0.00
Difference
081 68.36 116 45.22 161 20.06 196 68.36 0
082 79.31 117 6.39 162 89.38 197 66.39 −0.5
083 51.93 118 28.53 163 77.00 198 10.91 −1
166 167 168 169 170 171 172 173 174 175 176 177 178 179 190 191 192 193 194 195 196 197 198 199 200
084 0.43 119 13.73 164 21.17 199 87.08 Topic Identifier
085 48.67 130 79.86 165 59.36 200 49.53
279
dcu dcuitqydescsp12075 AH-ROBUST-BILI-X2ES-CLEF2006
Docs Cutoff Levels Precision at DCL (%)
Ad−Hoc Robust Bilingual Spanish track − Retrieved documents vs Precision
100%
5 docs 41.60
dcuitqydescsp12075
10 docs 37.40 90%
15 docs 35.13
80%
20 docs 31.85
30 docs 27.90 70%
100 docs 17.09
60%
200 docs 11.57
R−Precision
500 docs 6.40 50%
1000 docs 3.76 40%
R-Precision (precision after R document retrieved,
where R = Relevant retrieved) 30%
33.55
20%
10%
0%
5 10 15 20 30 100 200 500 1000
Retrieved Documents (logarithmic scale)
Exact R-Precision Ad−Hoc Robust Bilingual Spanish track − Box plot of the Topics of the Experiment
Maximum 1.0000
Minimum 0.0000
First Quartile 0.0714
Second Quartile 0.2857
Third Quartile 0.5312
Interquartile range 0.4598
Mean 0.3355
Standard Deviation 0.2866
Lower Outlier Threshold 0.0000
Upper Outlier Threshold 1.0000 0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Exact R−Precision
Mean With No Outliers 0.3355
Std With No Outliers 0.2866
Ad−Hoc Robust Bilingual Spanish track − Distribution of the Topics of the Experiment
25
Number of Topics of the Experiment
20
15
10
5
0
0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Exact R−Precision
Precision averages (%) for individual Ad−Hoc Robust Bilingual Spanish track − Comparison to Median Mean Exact R−Precision by Topic (Topics 041 to 085)
queries 1
dcuitqydescsp12075
0.5
041 6.25 086 28.57 131 0.91 166 0.00
Difference
0
042 44.44 087 48.86 132 0.00 167 50.00
−0.5
043 5.56 088 28.57 133 36.84 168 20.51
044 22.50 089 56.25 134 100.00 169 0.00 −1
041 042 043 044 045 046 047 048 049 060 061 062 063 064 065 066 067 068 069 080 081 082 083 084 085
Topic Identifier
045 49.21 090 0.00 135 29.49 170 80.00 Ad−Hoc Robust Bilingual Spanish track − Comparison to Median Mean Exact R−Precision by Topic (Topics 086 to 130)
1
046 0.00 091 12.90 136 50.00 171 0.00 dcuitqydescsp12075
047 10.19 092 25.58 137 10.00 172 15.38 0.5
Difference
048 16.67 093 21.43 138 100.00 173 33.33 0
049 21.05 094 72.73 139 1.15 174 81.82 −0.5
060 33.33 095 23.99 140 34.45 175 100.00 −1
086 087 088 089 090 091 092 093 094 095 096 097 098 099 110 111 112 113 114 115 116 117 118 119 130
061 0.00 096 3.23 141 84.62 176 92.21 Topic Identifier
Ad−Hoc Robust Bilingual Spanish track − Comparison to Median Mean Exact R−Precision by Topic (Topics 131 to 165)
062 16.67 097 64.00 142 82.14 177 0.00 1
dcuitqydescsp12075
063 71.19 098 40.00 143 42.36 178 28.41 0.5
Difference
064 0.00 099 7.14 144 27.27 179 3.54
0
065 39.19 110 57.14 145 48.94 190 0.00
−0.5
066 29.29 111 0.00 146 50.00 191 0.00
−1
067 6.12 112 50.00 147 11.11 192 56.82 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 160 161 162 163 164 165
Topic Identifier
068 0.00 113 8.86 148 36.36 193 8.82 Ad−Hoc Robust Bilingual Spanish track − Comparison to Median Mean Exact R−Precision by Topic (Topics 166 to 200)
1
069 0.00 114 57.78 149 69.23 194 12.50 dcuitqydescsp12075
0.5
080 41.63 115 31.58 160 0.00 195 0.00
Difference
081 63.22 116 42.86 161 20.00 196 66.67 0
082 76.32 117 7.14 162 82.95 197 62.50 −0.5
083 44.44 118 20.00 163 63.33 198 12.50 −1
166 167 168 169 170 171 172 173 174 175 176 177 178 179 190 191 192 193 194 195 196 197 198 199 200
084 0.00 119 21.90 164 25.00 199 80.00 Topic Identifier
085 42.99 130 61.54 165 66.67 200 44.68
280
reina reinaIT2EStdtest AH-ROBUST-BILI-X2ES-CLEF2006
Overall statistics for 100 queries : Priority 1
Total number of documents over all queries Query Construction AUTOMATIC
Retrieved 100,000 Source Language Italian
Relevant 5,008 Topic Fields title, description
Relevant retrieved 3,863 Pooled false
Geometric Mean Average Precision 0.1342 MT + local expansion with term windows, title+desc
Binary Preference (BPREF) 0.4754 fields
Interploated Recall (%) Precision Averages (%)
Ad−Hoc Robust Bilingual Spanish track − Interpolated Recall vs Average Precision
100%
0 66.09
reinaIT2EStdtest
10 56.89 90%
20 51.61
80%
30 46.99
40 43.63 70%
50 39.75
Average Precision
60%
60 34.59
70 29.70 50%
80 23.19 40%
90 16.77
30%
100 9.95
Average precision (non-interpolated) for all 20%
relevant documents (averaged over queries)
36.93 10%
0%
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
Interpolated Recall
Mean Average Precision Ad−Hoc Robust Bilingual Spanish track − Box plot of the Topics of the Experiment
Maximum 1.0000
Minimum 0.0000
First Quartile 0.0734
Second Quartile 0.2632
Third Quartile 0.6421
Interquartile range 0.5687
Mean 0.3693
Standard Deviation 0.3222
Lower Outlier Threshold 0.0000
Upper Outlier Threshold 1.0000 0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Mean Average Precision
Mean With No Outliers 0.3693
Std With No Outliers 0.3222
Ad−Hoc Robust Bilingual Spanish track − Distribution of the Topics of the Experiment
20
Number of Topics of the Experiment
15
10
5
0
0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Mean Average Precision
Precision averages (%) for individual Ad−Hoc Robust Bilingual Spanish track − Comparison to Median Mean Average Precision by Topic (Topics 041 to 085)
queries 1
reinaIT2EStdtest
0.5
041 7.49 086 26.23 131 15.85 166 1.02
Difference
0
042 44.71 087 64.33 132 47.05 167 58.57
−0.5
043 14.21 088 2.42 133 5.76 168 14.63
044 25.68 089 48.79 134 84.09 169 0.00 −1
041 042 043 044 045 046 047 048 049 060 061 062 063 064 065 066 067 068 069 080 081 082 083 084 085
Topic Identifier
045 64.09 090 0.67 135 66.63 170 83.93 Ad−Hoc Robust Bilingual Spanish track − Comparison to Median Mean Average Precision by Topic (Topics 086 to 130)
1
046 9.07 091 3.92 136 49.38 171 40.17 reinaIT2EStdtest
047 5.93 092 23.13 137 16.15 172 2.99 0.5
Difference
048 11.16 093 33.82 138 88.02 173 74.35 0
049 6.56 094 4.97 139 0.48 174 91.91 −0.5
060 26.41 095 20.73 140 11.87 175 100.00 −1
086 087 088 089 090 091 092 093 094 095 096 097 098 099 110 111 112 113 114 115 116 117 118 119 130
061 0.00 096 5.97 141 94.57 176 98.69 Topic Identifier
Ad−Hoc Robust Bilingual Spanish track − Comparison to Median Mean Average Precision by Topic (Topics 131 to 165)
062 29.47 097 73.45 142 85.64 177 7.19 1
reinaIT2EStdtest
063 79.68 098 75.71 143 54.92 178 0.93 0.5
Difference
064 100.00 099 2.56 144 59.86 179 49.72
0
065 14.50 110 59.52 145 58.63 190 1.07
−0.5
066 85.33 111 2.20 146 17.42 191 100.00
−1
067 23.51 112 20.53 147 11.61 192 28.14 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 160 161 162 163 164 165
Topic Identifier
068 0.58 113 0.02 148 13.25 193 11.65 Ad−Hoc Robust Bilingual Spanish track − Comparison to Median Mean Average Precision by Topic (Topics 166 to 200)
1
069 0.60 114 10.55 149 58.77 194 44.80 reinaIT2EStdtest
0.5
080 44.37 115 18.14 160 12.50 195 0.00
Difference
081 61.65 116 11.52 161 0.05 196 70.39 0
082 82.83 117 18.41 162 87.67 197 73.02 −0.5
083 39.14 118 42.86 163 34.51 198 48.92 −1
166 167 168 169 170 171 172 173 174 175 176 177 178 179 190 191 192 193 194 195 196 197 198 199 200
084 13.27 119 36.07 164 4.91 199 87.48 Topic Identifier
085 79.89 130 72.62 165 73.76 200 0.48
281
reina reinaIT2EStdtest AH-ROBUST-BILI-X2ES-CLEF2006
Docs Cutoff Levels Precision at DCL (%)
Ad−Hoc Robust Bilingual Spanish track − Retrieved documents vs Precision
100%
5 docs 44.40
reinaIT2EStdtest
10 docs 40.80 90%
15 docs 37.93
80%
20 docs 34.70
30 docs 30.80 70%
100 docs 19.29
60%
200 docs 13.00
R−Precision
500 docs 6.81 50%
1000 docs 3.86 40%
R-Precision (precision after R document retrieved,
where R = Relevant retrieved) 30%
36.73
20%
10%
0%
5 10 15 20 30 100 200 500 1000
Retrieved Documents (logarithmic scale)
Exact R-Precision Ad−Hoc Robust Bilingual Spanish track − Box plot of the Topics of the Experiment
Maximum 1.0000
Minimum 0.0000
First Quartile 0.1064
Second Quartile 0.3207
Third Quartile 0.6046
Interquartile range 0.4982
Mean 0.3673
Standard Deviation 0.2941
Lower Outlier Threshold 0.0000
Upper Outlier Threshold 1.0000 0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Exact R−Precision
Mean With No Outliers 0.3673
Std With No Outliers 0.2941
Ad−Hoc Robust Bilingual Spanish track − Distribution of the Topics of the Experiment
25
Number of Topics of the Experiment
20
15
10
5
0
0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Exact R−Precision
Precision averages (%) for individual Ad−Hoc Robust Bilingual Spanish track − Comparison to Median Mean Exact R−Precision by Topic (Topics 041 to 085)
queries 1
reinaIT2EStdtest
0.5
041 12.50 086 32.14 131 30.00 166 0.00
Difference
0
042 49.04 087 62.50 132 42.31 167 40.00
−0.5
043 11.11 088 0.00 133 21.05 168 17.95
044 25.00 089 56.25 134 75.00 169 0.00 −1
041 042 043 044 045 046 047 048 049 060 061 062 063 064 065 066 067 068 069 080 081 082 083 084 085
Topic Identifier
045 64.92 090 0.00 135 63.46 170 60.00 Ad−Hoc Robust Bilingual Spanish track − Comparison to Median Mean Exact R−Precision by Topic (Topics 086 to 130)
1
046 20.00 091 10.75 136 50.00 171 50.00 reinaIT2EStdtest
047 13.89 092 29.07 137 20.00 172 7.69 0.5
Difference
048 16.67 093 35.71 138 71.43 173 66.67 0
049 10.53 094 0.00 139 1.15 174 90.91 −0.5
060 32.00 095 31.78 140 17.65 175 100.00 −1
086 087 088 089 090 091 092 093 094 095 096 097 098 099 110 111 112 113 114 115 116 117 118 119 130
061 0.00 096 12.90 141 92.31 176 94.81 Topic Identifier
Ad−Hoc Robust Bilingual Spanish track − Comparison to Median Mean Exact R−Precision by Topic (Topics 131 to 165)
062 33.33 097 64.00 142 75.00 177 13.04 1
reinaIT2EStdtest
063 72.88 098 60.00 143 53.47 178 3.41 0.5
Difference
064 100.00 099 3.57 144 59.09 179 53.98
0
065 27.03 110 57.14 145 55.32 190 0.00
−0.5
066 78.79 111 0.00 146 0.00 191 100.00
−1
067 28.57 112 25.00 147 11.11 192 38.64 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 160 161 162 163 164 165
Topic Identifier
068 0.00 113 0.00 148 9.09 193 20.59 Ad−Hoc Robust Bilingual Spanish track − Comparison to Median Mean Exact R−Precision by Topic (Topics 166 to 200)
1
069 0.00 114 4.44 149 61.54 194 50.00 reinaIT2EStdtest
0.5
080 47.35 115 26.32 160 0.00 195 0.00
Difference
081 60.92 116 0.00 161 0.00 196 66.67 0
082 71.05 117 23.21 162 78.29 197 66.67 −0.5
083 55.56 118 40.00 163 43.33 198 50.00 −1
166 167 168 169 170 171 172 173 174 175 176 177 178 179 190 191 192 193 194 195 196 197 198 199 200
084 25.00 119 38.10 164 5.56 199 82.35 Topic Identifier
085 75.70 130 61.54 165 55.56 200 2.13
282
reina reinaIT2ESttest AH-ROBUST-BILI-X2ES-CLEF2006
Overall statistics for 100 queries : Priority 2
Total number of documents over all queries Query Construction AUTOMATIC
Retrieved 100,000 Source Language Italian
Relevant 5,008 Topic Fields title
Relevant retrieved 3,529 Pooled false
Geometric Mean Average Precision 0.0600 MT + local expansion with term windows, only title
Binary Preference (BPREF) 0.4030 field
Interploated Recall (%) Precision Averages (%)
Ad−Hoc Robust Bilingual Spanish track − Interpolated Recall vs Average Precision
100%
0 54.05
reinaIT2ESttest
10 47.46 90%
20 42.89
80%
30 38.89
40 34.68 70%
50 32.52
Average Precision
60%
60 28.71
70 24.78 50%
80 19.87 40%
90 14.04
30%
100 6.89
Average precision (non-interpolated) for all 20%
relevant documents (averaged over queries)
30.39 10%
0%
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
Interpolated Recall
Mean Average Precision Ad−Hoc Robust Bilingual Spanish track − Box plot of the Topics of the Experiment
Maximum 1.0000
Minimum 0.0000
First Quartile 0.0237
Second Quartile 0.1654
Third Quartile 0.5446
Interquartile range 0.5209
Mean 0.3039
Standard Deviation 0.3157
Lower Outlier Threshold 0.0000
Upper Outlier Threshold 1.0000 0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Mean Average Precision
Mean With No Outliers 0.3039
Std With No Outliers 0.3157
Ad−Hoc Robust Bilingual Spanish track − Distribution of the Topics of the Experiment
40
Number of Topics of the Experiment
35
30
25
20
15
10
5
0
0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Mean Average Precision
Precision averages (%) for individual Ad−Hoc Robust Bilingual Spanish track − Comparison to Median Mean Average Precision by Topic (Topics 041 to 085)
queries 1
reinaIT2ESttest
0.5
041 18.95 086 29.95 131 3.70 166 1.47
Difference
0
042 42.68 087 63.47 132 56.21 167 46.21
−0.5
043 16.80 088 0.00 133 1.34 168 10.24
044 13.16 089 18.62 134 80.88 169 0.00 −1
041 042 043 044 045 046 047 048 049 060 061 062 063 064 065 066 067 068 069 080 081 082 083 084 085
Topic Identifier
045 69.00 090 0.62 135 71.09 170 31.06 Ad−Hoc Robust Bilingual Spanish track − Comparison to Median Mean Average Precision by Topic (Topics 086 to 130)
1
046 3.61 091 0.00 136 55.45 171 15.46 reinaIT2ESttest
047 10.68 092 14.02 137 10.46 172 3.32 0.5
Difference
048 1.04 093 26.20 138 60.19 173 68.89 0
049 2.80 094 0.00 139 0.22 174 87.95 −0.5
060 30.34 095 2.63 140 13.84 175 100.00 −1
086 087 088 089 090 091 092 093 094 095 096 097 098 099 110 111 112 113 114 115 116 117 118 119 130
061 0.00 096 1.66 141 93.25 176 98.19 Topic Identifier
Ad−Hoc Robust Bilingual Spanish track − Comparison to Median Mean Average Precision by Topic (Topics 131 to 165)
062 29.58 097 78.17 142 83.86 177 1.48 1
reinaIT2ESttest
063 83.73 098 80.54 143 41.98 178 2.58 0.5
Difference
064 100.00 099 10.55 144 34.49 179 55.73
0
065 1.55 110 0.07 145 39.85 190 0.24
−0.5
066 84.96 111 1.99 146 7.34 191 16.67
−1
067 47.58 112 10.57 147 2.15 192 26.86 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 160 161 162 163 164 165
Topic Identifier
068 6.58 113 0.00 148 16.41 193 22.32 Ad−Hoc Robust Bilingual Spanish track − Comparison to Median Mean Average Precision by Topic (Topics 166 to 200)
1
069 0.05 114 5.56 149 53.47 194 0.66 reinaIT2ESttest
0.5
080 45.62 115 18.69 160 0.00 195 0.00
Difference
081 13.49 116 14.48 161 0.21 196 62.79 0
082 87.24 117 7.72 162 83.61 197 44.07 −0.5
083 40.48 118 0.78 163 15.64 198 52.75 −1
166 167 168 169 170 171 172 173 174 175 176 177 178 179 190 191 192 193 194 195 196 197 198 199 200
084 29.17 119 7.15 164 3.37 199 84.95 Topic Identifier
085 82.14 130 75.64 165 45.43 200 0.02
283
reina reinaIT2ESttest AH-ROBUST-BILI-X2ES-CLEF2006
Docs Cutoff Levels Precision at DCL (%)
Ad−Hoc Robust Bilingual Spanish track − Retrieved documents vs Precision
100%
5 docs 38.00
reinaIT2ESttest
10 docs 33.60 90%
15 docs 29.87
80%
20 docs 28.90
30 docs 25.90 70%
100 docs 16.94
60%
200 docs 11.37
R−Precision
500 docs 6.18 50%
1000 docs 3.53 40%
R-Precision (precision after R document retrieved,
where R = Relevant retrieved) 30%
31.64
20%
10%
0%
5 10 15 20 30 100 200 500 1000
Retrieved Documents (logarithmic scale)
Exact R-Precision Ad−Hoc Robust Bilingual Spanish track − Box plot of the Topics of the Experiment
Maximum 1.0000
Minimum 0.0000
First Quartile 0.0081
Second Quartile 0.2388
Third Quartile 0.5447
Interquartile range 0.5367
Mean 0.3164
Standard Deviation 0.2988
Lower Outlier Threshold 0.0000
Upper Outlier Threshold 1.0000 0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Exact R−Precision
Mean With No Outliers 0.3164
Std With No Outliers 0.2988
Ad−Hoc Robust Bilingual Spanish track − Distribution of the Topics of the Experiment
30
Number of Topics of the Experiment
25
20
15
10
5
0
0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Exact R−Precision
Precision averages (%) for individual Ad−Hoc Robust Bilingual Spanish track − Comparison to Median Mean Exact R−Precision by Topic (Topics 041 to 085)
queries 1
reinaIT2ESttest
0.5
041 37.50 086 32.14 131 10.91 166 0.00
Difference
0
042 45.21 087 59.09 132 50.00 167 30.00
−0.5
043 16.67 088 0.00 133 0.00 168 12.82
044 15.00 089 31.25 134 75.00 169 0.00 −1
041 042 043 044 045 046 047 048 049 060 061 062 063 064 065 066 067 068 069 080 081 082 083 084 085
Topic Identifier
045 69.11 090 0.00 135 64.10 170 40.00 Ad−Hoc Robust Bilingual Spanish track − Comparison to Median Mean Exact R−Precision by Topic (Topics 086 to 130)
1
046 0.00 091 0.00 136 62.50 171 25.00 reinaIT2ESttest
047 18.52 092 24.42 137 20.00 172 7.69 0.5
Difference
048 1.04 093 42.86 138 57.14 173 50.00 0
049 5.26 094 0.00 139 0.57 174 90.91 −0.5
060 38.67 095 9.03 140 21.01 175 100.00 −1
086 087 088 089 090 091 092 093 094 095 096 097 098 099 110 111 112 113 114 115 116 117 118 119 130
061 0.00 096 8.60 141 84.62 176 94.81 Topic Identifier
Ad−Hoc Robust Bilingual Spanish track − Comparison to Median Mean Exact R−Precision by Topic (Topics 131 to 165)
062 33.33 097 72.00 142 78.57 177 4.35 1
reinaIT2ESttest
063 77.97 098 80.00 143 41.67 178 6.82 0.5
Difference
064 100.00 099 17.86 144 50.00 179 58.41
0
065 0.00 110 0.00 145 40.43 190 0.00
−0.5
066 76.77 111 0.00 146 0.00 191 0.00
−1
067 55.10 112 0.00 147 11.11 192 29.55 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 160 161 162 163 164 165
Topic Identifier
068 22.22 113 0.00 148 18.18 193 29.41 Ad−Hoc Robust Bilingual Spanish track − Comparison to Median Mean Exact R−Precision by Topic (Topics 166 to 200)
1
069 0.00 114 2.22 149 53.85 194 0.00 reinaIT2ESttest
0.5
080 50.20 115 21.05 160 0.00 195 0.00
Difference
081 25.29 116 14.29 161 0.00 196 60.00 0
082 81.58 117 12.50 162 79.84 197 49.58 −0.5
083 44.44 118 0.00 163 23.33 198 50.00 −1
166 167 168 169 170 171 172 173 174 175 176 177 178 179 190 191 192 193 194 195 196 197 198 199 200
084 37.50 119 15.24 164 5.56 199 81.18 Topic Identifier
085 76.64 130 69.23 165 55.56 200 0.00
284
daedalus nlFSnlRLfr2S AH-ROBUST-BILI-X2NL-CLEF2006
Overall statistics for 100 queries : Priority 4
Total number of documents over all queries Query Construction AUTOMATIC
Retrieved 100,000 Source Language French
Relevant 2,717 Topic Fields title, description
Relevant retrieved 2,084 Pooled false
Geometric Mean Average Precision 0.0975 Mandatory TD run
Binary Preference (BPREF) 0.3575
Interploated Recall (%) Precision Averages (%)
Ad−Hoc Robust Bilingual Dutch track − Interpolated Recall vs Average Precision
100%
0 65.23
nlFSnlRLfr2S
10 58.61 90%
20 50.93
80%
30 43.67
40 40.00 70%
50 35.69
Average Precision
60%
60 31.27
70 26.93 50%
80 21.86 40%
90 16.62
30%
100 12.36
Average precision (non-interpolated) for all 20%
relevant documents (averaged over queries)
35.37 10%
0%
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
Interpolated Recall
Mean Average Precision Ad−Hoc Robust Bilingual Dutch track − Box plot of the Topics of the Experiment
Maximum 1.0000
Minimum 0.0000
First Quartile 0.0547
Second Quartile 0.2635
Third Quartile 0.6360
Interquartile range 0.5813
Mean 0.3537
Standard Deviation 0.3203
Lower Outlier Threshold 0.0000
Upper Outlier Threshold 1.0000 0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Mean Average Precision
Mean With No Outliers 0.3537
Std With No Outliers 0.3203
Ad−Hoc Robust Bilingual Dutch track − Distribution of the Topics of the Experiment
25
Number of Topics of the Experiment
20
15
10
5
0
0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Mean Average Precision
Precision averages (%) for individual Ad−Hoc Robust Bilingual Dutch track − Comparison to Median Mean Average Precision by Topic (Topics 041 to 085)
queries 1
nlFSnlRLfr2S
0.5
041 4.52 086 33.83 131 28.27 166 0.00
Difference
0
042 15.38 087 67.64 132 63.85 167 66.67
−0.5
043 51.67 088 36.07 133 8.08 168 21.80
044 27.75 089 94.49 134 100.00 169 2.86 −1
041 042 043 044 045 046 047 048 049 060 061 062 063 064 065 066 067 068 069 080 081 082 083 084 085
Topic Identifier
045 38.81 090 37.48 135 65.71 170 65.85 Ad−Hoc Robust Bilingual Dutch track − Comparison to Median Mean Average Precision by Topic (Topics 086 to 130)
1
046 17.25 091 12.46 136 100.00 171 1.81 nlFSnlRLfr2S
047 5.12 092 43.11 137 0.00 172 2.18 0.5
Difference
048 2.38 093 78.97 138 12.03 173 83.33 0
049 6.85 094 12.95 139 39.35 174 92.57 −0.5
060 21.68 095 27.10 140 17.80 175 0.00 −1
086 087 088 089 090 091 092 093 094 095 096 097 098 099 110 111 112 113 114 115 116 117 118 119 130
061 25.60 096 2.60 141 25.18 176 90.89 Topic Identifier
Ad−Hoc Robust Bilingual Dutch track − Comparison to Median Mean Average Precision by Topic (Topics 131 to 165)
062 57.19 097 84.88 142 71.02 177 7.80 1
nlFSnlRLfr2S
063 10.50 098 88.19 143 63.35 178 13.69 0.5
Difference
064 75.00 099 27.40 144 100.00 179 39.96
0
065 1.60 110 100.00 145 30.31 190 6.29
−0.5
066 74.81 111 8.33 146 33.33 191 0.00
−1
067 1.11 112 50.11 147 0.57 192 27.18 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 160 161 162 163 164 165
Topic Identifier
068 25.16 113 0.82 148 65.14 193 62.77 Ad−Hoc Robust Bilingual Dutch track − Comparison to Median Mean Average Precision by Topic (Topics 166 to 200)
1
069 6.23 114 25.55 149 83.62 194 0.00 nlFSnlRLfr2S
0.5
080 19.27 115 18.97 160 0.00 195 0.46
Difference
081 66.70 116 61.57 161 0.00 196 42.70 0
082 0.32 117 2.15 162 72.23 197 91.43 −0.5
083 4.43 118 23.98 163 5.81 198 0.67 −1
166 167 168 169 170 171 172 173 174 175 176 177 178 179 190 191 192 193 194 195 196 197 198 199 200
084 54.07 119 61.18 164 1.16 199 80.99 Topic Identifier
085 57.74 130 59.43 165 16.92 200 1.30
285
daedalus nlFSnlRLfr2S AH-ROBUST-BILI-X2NL-CLEF2006
Docs Cutoff Levels Precision at DCL (%)
Ad−Hoc Robust Bilingual Dutch track − Retrieved documents vs Precision
100%
5 docs 46.80
nlFSnlRLfr2S
10 docs 39.20 90%
15 docs 34.20
80%
20 docs 30.70
30 docs 25.43 70%
100 docs 12.87
60%
200 docs 8.05
R−Precision
500 docs 3.78 50%
1000 docs 2.08 40%
R-Precision (precision after R document retrieved,
where R = Relevant retrieved) 30%
34.18
20%
10%
0%
5 10 15 20 30 100 200 500 1000
Retrieved Documents (logarithmic scale)
Exact R-Precision Ad−Hoc Robust Bilingual Dutch track − Box plot of the Topics of the Experiment
Maximum 1.0000
Minimum 0.0000
First Quartile 0.0492
Second Quartile 0.2950
Third Quartile 0.5834
Interquartile range 0.5342
Mean 0.3418
Standard Deviation 0.2926
Lower Outlier Threshold 0.0000
Upper Outlier Threshold 1.0000 0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Exact R−Precision
Mean With No Outliers 0.3418
Std With No Outliers 0.2926
Ad−Hoc Robust Bilingual Dutch track − Distribution of the Topics of the Experiment
25
Number of Topics of the Experiment
20
15
10
5
0
0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Exact R−Precision
Precision averages (%) for individual Ad−Hoc Robust Bilingual Dutch track − Comparison to Median Mean Exact R−Precision by Topic (Topics 041 to 085)
queries 1
nlFSnlRLfr2S
0.5
041 4.35 086 40.00 131 37.50 166 0.00
Difference
0
042 25.00 087 46.15 132 61.54 167 50.00
−0.5
043 25.00 088 25.00 133 14.29 168 36.36
044 33.33 089 86.05 134 100.00 169 0.00 −1
041 042 043 044 045 046 047 048 049 060 061 062 063 064 065 066 067 068 069 080 081 082 083 084 085
Topic Identifier
045 37.50 090 38.46 135 66.13 170 50.00 Ad−Hoc Robust Bilingual Dutch track − Comparison to Median Mean Exact R−Precision by Topic (Topics 086 to 130)
1
046 22.92 091 11.11 136 100.00 171 0.00 nlFSnlRLfr2S
047 4.84 092 50.00 137 0.00 172 0.00 0.5
Difference
048 10.91 093 80.00 138 11.11 173 50.00 0
049 7.14 094 26.32 139 39.13 174 91.30 −0.5
060 30.77 095 35.55 140 26.39 175 0.00 −1
086 087 088 089 090 091 092 093 094 095 096 097 098 099 110 111 112 113 114 115 116 117 118 119 130
061 26.09 096 0.00 141 25.00 176 80.00 Topic Identifier
Ad−Hoc Robust Bilingual Dutch track − Comparison to Median Mean Exact R−Precision by Topic (Topics 131 to 165)
062 50.00 097 68.75 142 72.73 177 0.00 1
nlFSnlRLfr2S
063 20.00 098 66.67 143 62.07 178 28.57 0.5
Difference
064 75.00 099 25.00 144 100.00 179 40.32
0
065 7.25 110 100.00 145 28.57 190 12.90
−0.5
066 67.86 111 21.43 146 0.00 191 0.00
−1
067 5.00 112 50.00 147 0.00 192 26.67 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 160 161 162 163 164 165
Topic Identifier
068 30.43 113 6.06 148 61.90 193 47.37 Ad−Hoc Robust Bilingual Dutch track − Comparison to Median Mean Exact R−Precision by Topic (Topics 166 to 200)
1
069 0.00 114 31.37 149 66.67 194 0.00 nlFSnlRLfr2S
0.5
080 18.28 115 30.77 160 0.00 195 0.00
Difference
081 66.67 116 58.62 161 0.00 196 44.44 0
082 0.00 117 0.00 162 63.64 197 82.35 −0.5
083 0.00 118 36.36 163 21.43 198 0.00 −1
166 167 168 169 170 171 172 173 174 175 176 177 178 179 190 191 192 193 194 195 196 197 198 199 200
084 60.00 119 58.06 164 4.00 199 75.00 Topic Identifier
085 61.11 130 55.56 165 0.00 200 3.57
286
daedalus nlFSnlRLfr6S AH-ROBUST-BILI-X2NL-CLEF2006
Overall statistics for 100 queries : Priority 1
Total number of documents over all queries Query Construction AUTOMATIC
Retrieved 100,000 Source Language French
Relevant 2,717 Topic Fields title, description, narrative
Relevant retrieved 2,112 Pooled false
Geometric Mean Average Precision 0.1078 TDN run
Binary Preference (BPREF) 0.3712
Interploated Recall (%) Precision Averages (%)
Ad−Hoc Robust Bilingual Dutch track − Interpolated Recall vs Average Precision
100%
0 66.96
nlFSnlRLfr6S
10 60.16 90%
20 53.85
80%
30 46.19
40 42.64 70%
50 37.40
Average Precision
60%
60 33.28
70 28.74 50%
80 22.90 40%
90 17.75
30%
100 13.00
Average precision (non-interpolated) for all 20%
relevant documents (averaged over queries)
37.20 10%
0%
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
Interpolated Recall
Mean Average Precision Ad−Hoc Robust Bilingual Dutch track − Box plot of the Topics of the Experiment
Maximum 1.0000
Minimum 0.0000
First Quartile 0.0599
Second Quartile 0.3134
Third Quartile 0.6538
Interquartile range 0.5939
Mean 0.3720
Standard Deviation 0.3227
Lower Outlier Threshold 0.0000
Upper Outlier Threshold 1.0000 0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Mean Average Precision
Mean With No Outliers 0.3720
Std With No Outliers 0.3227
Ad−Hoc Robust Bilingual Dutch track − Distribution of the Topics of the Experiment
25
Number of Topics of the Experiment
20
15
10
5
0
0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Mean Average Precision
Precision averages (%) for individual Ad−Hoc Robust Bilingual Dutch track − Comparison to Median Mean Average Precision by Topic (Topics 041 to 085)
queries 1
nlFSnlRLfr6S
0.5
041 4.73 086 41.21 131 29.54 166 0.00
Difference
0
042 14.34 087 70.27 132 66.88 167 64.29
−0.5
043 47.25 088 36.37 133 11.87 168 29.50
044 32.03 089 93.95 134 100.00 169 2.79 −1
041 042 043 044 045 046 047 048 049 060 061 062 063 064 065 066 067 068 069 080 081 082 083 084 085
Topic Identifier
045 39.77 090 51.98 135 62.88 170 70.02 Ad−Hoc Robust Bilingual Dutch track − Comparison to Median Mean Average Precision by Topic (Topics 086 to 130)
1
046 13.66 091 6.15 136 100.00 171 1.28 nlFSnlRLfr6S
047 10.07 092 40.60 137 0.00 172 2.39 0.5
Difference
048 3.90 093 79.98 138 31.14 173 83.33 0
049 5.84 094 9.37 139 39.24 174 95.50 −0.5
060 8.51 095 28.09 140 17.70 175 0.00 −1
086 087 088 089 090 091 092 093 094 095 096 097 098 099 110 111 112 113 114 115 116 117 118 119 130
061 25.92 096 1.50 141 25.51 176 91.32 Topic Identifier
Ad−Hoc Robust Bilingual Dutch track − Comparison to Median Mean Average Precision by Topic (Topics 131 to 165)
062 59.79 097 90.22 142 70.89 177 8.31 1
nlFSnlRLfr6S
063 16.81 098 88.19 143 53.77 178 25.04 0.5
Difference
064 75.00 099 33.81 144 100.00 179 40.37
0
065 2.12 110 100.00 145 34.60 190 8.23
−0.5
066 74.15 111 8.01 146 29.17 191 0.00
−1
067 1.54 112 56.45 147 0.64 192 36.12 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 160 161 162 163 164 165
Topic Identifier
068 26.05 113 0.75 148 66.47 193 61.00 Ad−Hoc Robust Bilingual Dutch track − Comparison to Median Mean Average Precision by Topic (Topics 166 to 200)
1
069 7.88 114 42.26 149 86.90 194 0.00 nlFSnlRLfr6S
0.5
080 73.73 115 22.09 160 0.00 195 0.82
Difference
081 73.37 116 63.33 161 0.00 196 42.41 0
082 1.18 117 2.58 162 73.06 197 91.18 −0.5
083 4.86 118 31.55 163 5.63 198 7.76 −1
166 167 168 169 170 171 172 173 174 175 176 177 178 179 190 191 192 193 194 195 196 197 198 199 200
084 50.99 119 51.74 164 0.88 199 82.29 Topic Identifier
085 55.08 130 72.24 165 16.67 200 1.64
287
daedalus nlFSnlRLfr6S AH-ROBUST-BILI-X2NL-CLEF2006
Docs Cutoff Levels Precision at DCL (%)
Ad−Hoc Robust Bilingual Dutch track − Retrieved documents vs Precision
100%
5 docs 48.40
nlFSnlRLfr6S
10 docs 40.80 90%
15 docs 36.27
80%
20 docs 31.75
30 docs 26.73 70%
100 docs 13.85
60%
200 docs 8.42
R−Precision
500 docs 3.94 50%
1000 docs 2.11 40%
R-Precision (precision after R document retrieved,
where R = Relevant retrieved) 30%
35.63
20%
10%
0%
5 10 15 20 30 100 200 500 1000
Retrieved Documents (logarithmic scale)
Exact R-Precision Ad−Hoc Robust Bilingual Dutch track − Box plot of the Topics of the Experiment
Maximum 1.0000
Minimum 0.0000
First Quartile 0.0417
Second Quartile 0.3205
Third Quartile 0.6000
Interquartile range 0.5583
Mean 0.3563
Standard Deviation 0.2992
Lower Outlier Threshold 0.0000
Upper Outlier Threshold 1.0000 0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Exact R−Precision
Mean With No Outliers 0.3563
Std With No Outliers 0.2992
Ad−Hoc Robust Bilingual Dutch track − Distribution of the Topics of the Experiment
30
Number of Topics of the Experiment
25
20
15
10
5
0
0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Exact R−Precision
Precision averages (%) for individual Ad−Hoc Robust Bilingual Dutch track − Comparison to Median Mean Exact R−Precision by Topic (Topics 041 to 085)
queries 1
nlFSnlRLfr6S
0.5
041 4.35 086 44.00 131 35.00 166 0.00
Difference
0
042 25.00 087 46.15 132 61.54 167 50.00
−0.5
043 25.00 088 25.00 133 17.86 168 45.45
044 40.00 089 88.37 134 100.00 169 0.00 −1
041 042 043 044 045 046 047 048 049 060 061 062 063 064 065 066 067 068 069 080 081 082 083 084 085
Topic Identifier
045 35.00 090 46.15 135 63.71 170 64.29 Ad−Hoc Robust Bilingual Dutch track − Comparison to Median Mean Exact R−Precision by Topic (Topics 086 to 130)
1
046 16.67 091 11.11 136 100.00 171 0.00 nlFSnlRLfr6S
047 14.52 092 47.37 137 0.00 172 0.00 0.5
Difference
048 10.91 093 80.00 138 33.33 173 50.00 0
049 0.00 094 15.79 139 39.13 174 91.30 −0.5
060 15.38 095 39.20 140 26.39 175 0.00 −1
086 087 088 089 090 091 092 093 094 095 096 097 098 099 110 111 112 113 114 115 116 117 118 119 130
061 30.43 096 0.00 141 25.00 176 80.00 Topic Identifier
Ad−Hoc Robust Bilingual Dutch track − Comparison to Median Mean Exact R−Precision by Topic (Topics 131 to 165)
062 55.00 097 81.25 142 72.73 177 0.00 1
nlFSnlRLfr6S
063 20.00 098 66.67 143 58.62 178 28.57 0.5
Difference
064 75.00 099 25.00 144 100.00 179 40.32
0
065 8.70 110 100.00 145 28.57 190 12.90
−0.5
066 64.29 111 21.43 146 0.00 191 0.00
−1
067 5.00 112 60.00 147 0.00 192 40.00 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 160 161 162 163 164 165
Topic Identifier
068 30.43 113 3.03 148 61.90 193 47.37 Ad−Hoc Robust Bilingual Dutch track − Comparison to Median Mean Exact R−Precision by Topic (Topics 166 to 200)
1
069 0.00 114 43.14 149 66.67 194 0.00 nlFSnlRLfr6S
0.5
080 70.97 115 30.77 160 0.00 195 0.00
Difference
081 74.07 116 55.17 161 0.00 196 44.44 0
082 0.00 117 0.00 162 60.61 197 83.19 −0.5
083 0.00 118 45.45 163 21.43 198 0.00 −1
166 167 168 169 170 171 172 173 174 175 176 177 178 179 190 191 192 193 194 195 196 197 198 199 200
084 60.00 119 54.84 164 4.00 199 78.12 Topic Identifier
085 61.11 130 55.56 165 0.00 200 3.57
288
daedalus nlx021nlRLfrFW4FS6 AH-ROBUST-BILI-X2NL-CLEF2006
Overall statistics for 100 queries : Priority 3
Total number of documents over all queries Query Construction AUTOMATIC
Retrieved 100,000 Source Language French
Relevant 2,717 Topic Fields title, description, narrative
Relevant retrieved 2,112 Pooled false
Geometric Mean Average Precision 0.1075 Combined run
Binary Preference (BPREF) 0.3679
Interploated Recall (%) Precision Averages (%)
Ad−Hoc Robust Bilingual Dutch track − Interpolated Recall vs Average Precision
100%
0 65.28
nlx021nlRLfrFW4FS6
10 58.74 90%
20 53.47
80%
30 46.17
40 42.65 70%
50 37.41
Average Precision
60%
60 33.34
70 28.80 50%
80 22.97 40%
90 17.82
30%
100 13.07
Average precision (non-interpolated) for all 20%
relevant documents (averaged over queries)
36.92 10%
0%
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
Interpolated Recall
Mean Average Precision Ad−Hoc Robust Bilingual Dutch track − Box plot of the Topics of the Experiment
Maximum 1.0000
Minimum 0.0000
First Quartile 0.0561
Second Quartile 0.3105
Third Quartile 0.6311
Interquartile range 0.5749
Mean 0.3692
Standard Deviation 0.3183
Lower Outlier Threshold 0.0000
Upper Outlier Threshold 1.0000 0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Mean Average Precision
Mean With No Outliers 0.3692
Std With No Outliers 0.3183
Ad−Hoc Robust Bilingual Dutch track − Distribution of the Topics of the Experiment
25
Number of Topics of the Experiment
20
15
10
5
0
0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Mean Average Precision
Precision averages (%) for individual Ad−Hoc Robust Bilingual Dutch track − Comparison to Median Mean Average Precision by Topic (Topics 041 to 085)
queries 1
nlx021nlRLfrFW4FS6
0.5
041 4.73 086 41.21 131 26.96 166 0.00
Difference
0
042 14.34 087 70.27 132 66.88 167 30.95
−0.5
043 47.25 088 36.37 133 11.87 168 30.01
044 28.70 089 93.95 134 100.00 169 2.79 −1
041 042 043 044 045 046 047 048 049 060 061 062 063 064 065 066 067 068 069 080 081 082 083 084 085
Topic Identifier
045 38.93 090 52.62 135 62.88 170 70.02 Ad−Hoc Robust Bilingual Dutch track − Comparison to Median Mean Average Precision by Topic (Topics 086 to 130)
1
046 13.66 091 4.30 136 100.00 171 1.28 nlx021nlRLfrFW4FS6
047 10.07 092 39.28 137 0.00 172 2.39 0.5
Difference
048 3.90 093 79.98 138 31.14 173 58.33 0
049 5.84 094 6.65 139 39.24 174 95.50 −0.5
060 8.19 095 28.09 140 17.70 175 0.00 −1
086 087 088 089 090 091 092 093 094 095 096 097 098 099 110 111 112 113 114 115 116 117 118 119 130
061 27.37 096 1.50 141 25.51 176 91.32 Topic Identifier
Ad−Hoc Robust Bilingual Dutch track − Comparison to Median Mean Average Precision by Topic (Topics 131 to 165)
062 59.79 097 90.22 142 70.89 177 8.31 1
nlx021nlRLfrFW4FS6
063 16.81 098 88.19 143 53.77 178 22.66 0.5
Difference
064 75.00 099 29.79 144 100.00 179 40.37
0
065 2.12 110 100.00 145 34.60 190 10.92
−0.5
066 72.37 111 8.01 146 45.00 191 0.00
−1
067 1.42 112 56.45 147 0.64 192 33.89 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 160 161 162 163 164 165
Topic Identifier
068 21.53 113 0.75 148 67.07 193 61.00 Ad−Hoc Robust Bilingual Dutch track − Comparison to Median Mean Average Precision by Topic (Topics 166 to 200)
1
069 7.88 114 42.26 149 86.90 194 0.00 nlx021nlRLfrFW4FS6
0.5
080 70.76 115 18.88 160 0.00 195 0.82
Difference
081 73.37 116 63.33 161 0.00 196 42.41 0
082 1.18 117 2.58 162 73.06 197 91.18 −0.5
083 4.86 118 31.55 163 5.39 198 50.62 −1
166 167 168 169 170 171 172 173 174 175 176 177 178 179 190 191 192 193 194 195 196 197 198 199 200
084 50.99 119 51.74 164 0.88 199 82.29 Topic Identifier
085 55.08 130 72.24 165 16.67 200 1.64
289
daedalus nlx021nlRLfrFW4FS6 AH-ROBUST-BILI-X2NL-CLEF2006
Docs Cutoff Levels Precision at DCL (%)
Ad−Hoc Robust Bilingual Dutch track − Retrieved documents vs Precision
100%
5 docs 48.20
nlx021nlRLfrFW4FS6
10 docs 40.60 90%
15 docs 36.20
80%
20 docs 31.60
30 docs 26.70 70%
100 docs 13.85
60%
200 docs 8.42
R−Precision
500 docs 3.94 50%
1000 docs 2.11 40%
R-Precision (precision after R document retrieved,
where R = Relevant retrieved) 30%
36.13
20%
10%
0%
5 10 15 20 30 100 200 500 1000
Retrieved Documents (logarithmic scale)
Exact R-Precision Ad−Hoc Robust Bilingual Dutch track − Box plot of the Topics of the Experiment
Maximum 1.0000
Minimum 0.0000
First Quartile 0.0467
Second Quartile 0.3417
Third Quartile 0.6000
Interquartile range 0.5533
Mean 0.3613
Standard Deviation 0.2973
Lower Outlier Threshold 0.0000
Upper Outlier Threshold 1.0000 0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Exact R−Precision
Mean With No Outliers 0.3613
Std With No Outliers 0.2973
Ad−Hoc Robust Bilingual Dutch track − Distribution of the Topics of the Experiment
25
Number of Topics of the Experiment
20
15
10
5
0
0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Exact R−Precision
Precision averages (%) for individual Ad−Hoc Robust Bilingual Dutch track − Comparison to Median Mean Exact R−Precision by Topic (Topics 041 to 085)
queries 1
nlx021nlRLfrFW4FS6
0.5
041 4.35 086 44.00 131 35.00 166 0.00
Difference
0
042 25.00 087 46.15 132 61.54 167 0.00
−0.5
043 25.00 088 25.00 133 17.86 168 45.45
044 40.00 089 88.37 134 100.00 169 0.00 −1
041 042 043 044 045 046 047 048 049 060 061 062 063 064 065 066 067 068 069 080 081 082 083 084 085
Topic Identifier
045 35.00 090 46.15 135 63.71 170 64.29 Ad−Hoc Robust Bilingual Dutch track − Comparison to Median Mean Exact R−Precision by Topic (Topics 086 to 130)
1
046 16.67 091 11.11 136 100.00 171 0.00 nlx021nlRLfrFW4FS6
047 14.52 092 47.37 137 0.00 172 0.00 0.5
Difference
048 10.91 093 80.00 138 33.33 173 50.00 0
049 0.00 094 15.79 139 39.13 174 91.30 −0.5
060 15.38 095 39.20 140 26.39 175 0.00 −1
086 087 088 089 090 091 092 093 094 095 096 097 098 099 110 111 112 113 114 115 116 117 118 119 130
061 30.43 096 0.00 141 25.00 176 80.00 Topic Identifier
Ad−Hoc Robust Bilingual Dutch track − Comparison to Median Mean Exact R−Precision by Topic (Topics 131 to 165)
062 55.00 097 81.25 142 72.73 177 0.00 1
nlx021nlRLfrFW4FS6
063 20.00 098 66.67 143 58.62 178 28.57 0.5
Difference
064 75.00 099 25.00 144 100.00 179 40.32
0
065 8.70 110 100.00 145 28.57 190 12.90
−0.5
066 64.29 111 21.43 146 50.00 191 0.00
−1
067 5.00 112 60.00 147 0.00 192 40.00 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 160 161 162 163 164 165
Topic Identifier
068 30.43 113 3.03 148 61.90 193 47.37 Ad−Hoc Robust Bilingual Dutch track − Comparison to Median Mean Exact R−Precision by Topic (Topics 166 to 200)
1
069 0.00 114 43.14 149 66.67 194 0.00 nlx021nlRLfrFW4FS6
0.5
080 70.97 115 30.77 160 0.00 195 0.00
Difference
081 74.07 116 55.17 161 0.00 196 44.44 0
082 0.00 117 0.00 162 60.61 197 83.19 −0.5
083 0.00 118 45.45 163 21.43 198 50.00 −1
166 167 168 169 170 171 172 173 174 175 176 177 178 179 190 191 192 193 194 195 196 197 198 199 200
084 60.00 119 54.84 164 4.00 199 78.12 Topic Identifier
085 61.11 130 55.56 165 0.00 200 3.57
290
daedalus nlx011nlRLfrFW4FS6 AH-ROBUST-BILI-X2NL-CLEF2006
Overall statistics for 100 queries : Priority 2
Total number of documents over all queries Query Construction AUTOMATIC
Retrieved 100,000 Source Language French
Relevant 2,717 Topic Fields title, description, narrative
Relevant retrieved 2,112 Pooled false
Geometric Mean Average Precision 0.1080 Combined run
Binary Preference (BPREF) 0.3683
Interploated Recall (%) Precision Averages (%)
Ad−Hoc Robust Bilingual Dutch track − Interpolated Recall vs Average Precision
100%
0 65.60
nlx011nlRLfrFW4FS6
10 59.69 90%
20 53.82
80%
30 46.18
40 42.65 70%
50 37.41
Average Precision
60%
60 33.28
70 28.74 50%
80 22.90 40%
90 17.75
30%
100 13.00
Average precision (non-interpolated) for all 20%
relevant documents (averaged over queries)
36.99 10%
0%
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
Interpolated Recall
Mean Average Precision Ad−Hoc Robust Bilingual Dutch track − Box plot of the Topics of the Experiment
Maximum 1.0000
Minimum 0.0000
First Quartile 0.0561
Second Quartile 0.3134
Third Quartile 0.6311
Interquartile range 0.5749
Mean 0.3699
Standard Deviation 0.3178
Lower Outlier Threshold 0.0000
Upper Outlier Threshold 1.0000 0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Mean Average Precision
Mean With No Outliers 0.3699
Std With No Outliers 0.3178
Ad−Hoc Robust Bilingual Dutch track − Distribution of the Topics of the Experiment
25
Number of Topics of the Experiment
20
15
10
5
0
0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Mean Average Precision
Precision averages (%) for individual Ad−Hoc Robust Bilingual Dutch track − Comparison to Median Mean Average Precision by Topic (Topics 041 to 085)
queries 1
nlx011nlRLfrFW4FS6
0.5
041 4.73 086 41.21 131 29.54 166 0.00
Difference
0
042 14.34 087 70.27 132 66.88 167 39.29
−0.5
043 47.25 088 36.37 133 11.87 168 29.50
044 28.70 089 93.95 134 100.00 169 2.79 −1
041 042 043 044 045 046 047 048 049 060 061 062 063 064 065 066 067 068 069 080 081 082 083 084 085
Topic Identifier
045 39.77 090 51.98 135 62.88 170 70.02 Ad−Hoc Robust Bilingual Dutch track − Comparison to Median Mean Average Precision by Topic (Topics 086 to 130)
1
046 13.66 091 4.30 136 100.00 171 1.28 nlx011nlRLfrFW4FS6
047 10.07 092 39.28 137 0.00 172 2.39 0.5
Difference
048 3.90 093 79.98 138 31.14 173 58.33 0
049 5.84 094 7.61 139 39.24 174 95.50 −0.5
060 8.51 095 28.09 140 17.70 175 0.00 −1
086 087 088 089 090 091 092 093 094 095 096 097 098 099 110 111 112 113 114 115 116 117 118 119 130
061 27.37 096 1.50 141 25.51 176 91.32 Topic Identifier
Ad−Hoc Robust Bilingual Dutch track − Comparison to Median Mean Average Precision by Topic (Topics 131 to 165)
062 59.79 097 90.22 142 70.89 177 8.31 1
nlx011nlRLfrFW4FS6
063 16.81 098 88.19 143 53.77 178 25.04 0.5
Difference
064 75.00 099 33.81 144 100.00 179 40.37
0
065 2.12 110 100.00 145 34.60 190 9.84
−0.5
066 72.37 111 8.01 146 29.17 191 0.00
−1
067 1.42 112 56.45 147 0.64 192 36.12 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 160 161 162 163 164 165
Topic Identifier
068 23.35 113 0.75 148 66.47 193 61.00 Ad−Hoc Robust Bilingual Dutch track − Comparison to Median Mean Average Precision by Topic (Topics 166 to 200)
1
069 7.88 114 42.26 149 86.90 194 0.00 nlx011nlRLfrFW4FS6
0.5
080 71.62 115 20.17 160 0.00 195 0.82
Difference
081 73.37 116 63.33 161 0.00 196 42.41 0
082 1.18 117 2.58 162 73.06 197 91.18 −0.5
083 4.86 118 31.55 163 5.39 198 50.62 −1
166 167 168 169 170 171 172 173 174 175 176 177 178 179 190 191 192 193 194 195 196 197 198 199 200
084 50.99 119 51.74 164 0.88 199 82.29 Topic Identifier
085 55.08 130 72.24 165 16.67 200 1.64
291
daedalus nlx011nlRLfrFW4FS6 AH-ROBUST-BILI-X2NL-CLEF2006
Docs Cutoff Levels Precision at DCL (%)
Ad−Hoc Robust Bilingual Dutch track − Retrieved documents vs Precision
100%
5 docs 48.20
nlx011nlRLfrFW4FS6
10 docs 40.70 90%
15 docs 36.27
80%
20 docs 31.65
30 docs 26.70 70%
100 docs 13.85
60%
200 docs 8.42
R−Precision
500 docs 3.94 50%
1000 docs 2.11 40%
R-Precision (precision after R document retrieved,
where R = Relevant retrieved) 30%
36.13
20%
10%
0%
5 10 15 20 30 100 200 500 1000
Retrieved Documents (logarithmic scale)
Exact R-Precision Ad−Hoc Robust Bilingual Dutch track − Box plot of the Topics of the Experiment
Maximum 1.0000
Minimum 0.0000
First Quartile 0.0467
Second Quartile 0.3417
Third Quartile 0.6000
Interquartile range 0.5533
Mean 0.3613
Standard Deviation 0.2973
Lower Outlier Threshold 0.0000
Upper Outlier Threshold 1.0000 0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Exact R−Precision
Mean With No Outliers 0.3613
Std With No Outliers 0.2973
Ad−Hoc Robust Bilingual Dutch track − Distribution of the Topics of the Experiment
25
Number of Topics of the Experiment
20
15
10
5
0
0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Exact R−Precision
Precision averages (%) for individual Ad−Hoc Robust Bilingual Dutch track − Comparison to Median Mean Exact R−Precision by Topic (Topics 041 to 085)
queries 1
nlx011nlRLfrFW4FS6
0.5
041 4.35 086 44.00 131 35.00 166 0.00
Difference
0
042 25.00 087 46.15 132 61.54 167 50.00
−0.5
043 25.00 088 25.00 133 17.86 168 45.45
044 40.00 089 88.37 134 100.00 169 0.00 −1
041 042 043 044 045 046 047 048 049 060 061 062 063 064 065 066 067 068 069 080 081 082 083 084 085
Topic Identifier
045 35.00 090 46.15 135 63.71 170 64.29 Ad−Hoc Robust Bilingual Dutch track − Comparison to Median Mean Exact R−Precision by Topic (Topics 086 to 130)
1
046 16.67 091 11.11 136 100.00 171 0.00 nlx011nlRLfrFW4FS6
047 14.52 092 47.37 137 0.00 172 0.00 0.5
Difference
048 10.91 093 80.00 138 33.33 173 50.00 0
049 0.00 094 15.79 139 39.13 174 91.30 −0.5
060 15.38 095 39.20 140 26.39 175 0.00 −1
086 087 088 089 090 091 092 093 094 095 096 097 098 099 110 111 112 113 114 115 116 117 118 119 130
061 30.43 096 0.00 141 25.00 176 80.00 Topic Identifier
Ad−Hoc Robust Bilingual Dutch track − Comparison to Median Mean Exact R−Precision by Topic (Topics 131 to 165)
062 55.00 097 81.25 142 72.73 177 0.00 1
nlx011nlRLfrFW4FS6
063 20.00 098 66.67 143 58.62 178 28.57 0.5
Difference
064 75.00 099 25.00 144 100.00 179 40.32
0
065 8.70 110 100.00 145 28.57 190 12.90
−0.5
066 64.29 111 21.43 146 0.00 191 0.00
−1
067 5.00 112 60.00 147 0.00 192 40.00 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 160 161 162 163 164 165
Topic Identifier
068 30.43 113 3.03 148 61.90 193 47.37 Ad−Hoc Robust Bilingual Dutch track − Comparison to Median Mean Exact R−Precision by Topic (Topics 166 to 200)
1
069 0.00 114 43.14 149 66.67 194 0.00 nlx011nlRLfrFW4FS6
0.5
080 70.97 115 30.77 160 0.00 195 0.00
Difference
081 74.07 116 55.17 161 0.00 196 44.44 0
082 0.00 117 0.00 162 60.61 197 83.19 −0.5
083 0.00 118 45.45 163 21.43 198 50.00 −1
166 167 168 169 170 171 172 173 174 175 176 177 178 179 190 191 192 193 194 195 196 197 198 199 200
084 60.00 119 54.84 164 4.00 199 78.12 Topic Identifier
085 61.11 130 55.56 165 0.00 200 3.57
292