Introduction 3
Results for CLEF 2007 Domain Specific Track 3. Individual Experiment Results and Graphs This section provides the individual results for each official experiment. For each experiment the following tables and graphs are shown: - Overall statistics and information - Interpolated recall vs precision averages plot - Average precision statistics and box plot - Average precision comparison to median plot - Document cutoff levels vs precision at DCL plot - R-Precision statistics and box plot - R-Precision comparison to median plot Topics are identified with DOIs, as well as the experiments. The prefix for the DOI of a topic is 10.2452. The following example shows how to build the DOI for a topic given its number: for topic 200-AH, the corresponding DOI is 10.2452/200-AH List of Submitted Experiments 7
CLEF2007.CHEMNITZ.CUT_DS_MONO_DE_STRUCT 10.2415/DS-MONO-DECLEF2007.CHEMNITZ.CUT_DS_MONO_DE_UNSTRUCT 10.2415/DS-MONO-DE
10.2415/DS-MONO-DE
10.2415/DS-MONO-DE
10.2415/DS-MONO-DE
10.2415/DS-MONO-DE
10.2415/DS-MONO-DE
10.2415/DS-MONO-DE
10.2415/DS-MONO-DE
10.2415/DS-MONO-ENCLEF2007.CHEMNITZ.CUT_DS_MONO_EN_MERGED 10.2415/DS-MONO-ENCLEF2007.CHEMNITZ.CUT_DS_MONO_EN_MERGED_S
10.2415/DS-MONO-ENCLEF2007.CHEMNITZ.CUT_DS_MONO_EN_MERGED_U
10.2415/DS-MONO-EN
10.2415/DS-MONO-EN
10.2415/DS-MONO-EN
10.2415/DS-MONO-EN
10.2415/DS-MONO-EN
10.2415/DS-MONO-EN
10.2415/DS-MONO-EN
10.2415/DS-MONO-EN
10.2415/DS-MONO-EN
10.2415/DS-MONO-RUCLEF2007.CHEMNITZ.CUT_DS_MONO_RU_MERGED 10.2415/DS-MONO-RUCLEF2007.CHEMNITZ.CUT_DS_MONO_RU_STRUCT 10.2415/DS-MONO-RUCLEF2007.CHEMNITZ.CUT_DS_MONO_RU_UNSTRUCT
10.2415/DS-MONO-RU
10.2415/DS-MONO-RU
10.2415/DS-MONO-RU
10.2415/DS-MONO-RU
chemnitz 10.2415/DS-BILI-X2DECLEF2007.CHEMNITZ.CUT_DS_BILI_EN2DE_MERGE
10.2415/DS-BILI-X2DECLEF2007.CHEMNITZ.CUT_DS_BILI_RU2DE_MERGE D en ru ru en en ru ru en en en en en en de de ru de de ru ru ru ru de de de de de en de de en de de en en en en de en ru
TD TD TD TD TD TD TD TD TD TD TD TD TD TD TD TD TD TD TD TD TD TD TD TD TD TD TD TD TD TD TD TD TD TD TD TD TD TD TD TD Track Overview Results and Graphs 12 30% 20% 10% 0%
0% 0.8 0.6 0.4 0.2 e c n re 0 e iff D −0.2 −0.4 −0.6 −0.8 −1
cheshire [Experiment BERKMDETHP7; MAP 31.99%; Pooled] cheshire [Experiment BERKMDECCP15; MAP 31.50%; Pooled] chemnitz [Experiment CUT_DS_MONO_DE_MERGED; MAP 29.91%; Pooled] chemnitz [Experiment CUT_DS_MONO_DE_UNSTRUCT; MAP 28.87%; Pooled] chemnitz [Experiment CUT_DS_MONO_DE_STRUCT; MAP 26.31%; Pooled] 0% 10% 20% 30% 70% 80% 10.2455/TUKEY_T_TEST.67D6904A15764192AAB8A24F7DD2C621
Domain−Specific Monolingual German Task − Tukey T test with "top group" highlighted
Domain−Specific Monolingual German Task Top 5 Participants − Retrieved documents vs Mean Precision 100% xerox [Experiment XRCEPRFLEDE; R−Prec 50.08%; Pooled] unine [Experiment UNINEDE4; R−Prec 39.61%; Pooled] cheshire [Experiment BERKMDETHP7; R−Prec 35.75%; Pooled] chemnitz [Experiment CUT_DS_MONO_DE_MERGED; R−Prec 33.84%; Pooled] 90%
30 100 200 Retrieved Documents (logarithmic scale) 500 1000 1 Domain−Specific Monolingual German Task Top 5 Participants − Comparison to Median R−Precision by Topic (Topics 176−DS to 200−DS) xerox [Experiment XRCEPRFLEDE; R−Prec 50.08%; Pooled] unine [Experiment UNINEDE4; R−Prec 39.61%; Pooled] cheshire [Experiment BERKMDETHP7; R−Prec 35.75%; Pooled] chemnitz [Experiment CUT_DS_MONO_DE_MERGED; R−Prec 33.84%; Pooled] chemnitz [Experiment CUT_DS_MONO_DE_UNSTRUCT; R−Prec 32.97%; Pooled] chemnitz [Experiment CUT_DS_MONO_DE_STRUCT; R−Prec 30.37%; Pooled] 0% 10% 20% 30% 40%
50% R−Precision 60% 70% 80%
Domain−Specific Monolingual German Task − Tukey T test with "top group" highlighted XRCEPRFLEDE
Domain−Specific Bilingual Russian Task − Distribution of the Topics of the Experiment 00% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Average Precision 1 0.8 0.6 0.4 0.2 0%5
Domain−Specific Bilingual Russian Task − Distribution of the Topics of the Experiment
Domain−Specific Bilingual Russian Task − Box plot of the Topics of the Experiment
Domain−Specific Bilingual Russian Task − Distribution of the Topics of the Experiment 00% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Average Precision 1 0.8 0.6 0.4 0.2 0%5 Domain−Specific Bilingual Russian Task − Box plot of the Topics of the Experiment
Domain−Specific Bilingual Russian Task − Distribution of the Topics of the Experiment 15 t n e m ir e p xE10 e h ft so cp i o fT 5 o r e b m u N
Domain−Specific Multilingual Task − Box plot of the Topics of the Experiment
Domain−Specific Multilingual Task − Distribution of the Topics of the Experiment 00% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Average Precision 1 0.8 0.6 0.4 0.2 15 t n e m ir e p xE10 e h ft so cp i o fT 5 o r e b m u
N −1176−DS177−DS178−DS179−DS180−DS181−DS182−DS183−DS184−DS185−DS186−DS187−DS188−DS189−DS190−DS191−DS192−DS193−DS194−DS195−DS196−DS197−DS198−DS199−DS200−DS
Multilingual merged unstructured indizes translated with Google translate (Beta), PROMT and the provided thesaurus 0%5 Domain−Specific Multilingual Task − Box plot of the Topics of the Experiment
Domain−Specific Multilingual Task − Distribution of the Topics of the Experiment 6 t iren5 m e xp E4 e h ftcspo3 i o f2 T o r e bu1 m N 00% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
−1176−DS177−DS178−DS179−DS180−DS181−DS182−DS183−DS184−DS185−DS186−DS187−DS188−DS189−DS190−DS191−DS192−DS193−DS194−DS195−DS196−DS197−DS198−DS199−DS200−DS
Priority
Query Construction 25,000 Source Language 9,690 Topic Fields 2,662 Pooled
Domain−Specific Multilingual Task − Box plot of the Topics of the Experiment
Domain−Specific Multilingual Task − Distribution of the Topics of the Experiment 00% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100% Average Precision 1 0.8 0.6 0.4 0.2 −1176−DS177−DS178−DS179−DS180−DS181−DS182−DS183−DS184−DS185−DS186−DS187−DS188−DS189−DS190−DS191−DS192−DS193−DS194−DS195−DS196−DS197−DS198−DS199−DS200−DS
Multilingual merged unstructured indizes translated with Google translate (Beta) and the provided thesaurus 0%5 Domain−Specific Multilingual Task − Box plot of the Topics of the Experiment R_PRECISION
Domain−Specific Multilingual Task − Distribution of the Topics of the Experiment 6 t iren5 m e xp E4 e h ftscop3 i o f2 T o r e ub1 m N 00% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
−1176−DS177−DS178−DS179−DS180−DS181−DS182−DS183−DS184−DS185−DS186−DS187−DS188−DS189−DS190−DS191−DS192−DS193−DS194−DS195−DS196−DS197−DS198−DS199−DS200−DS
Priority
Query Construction 25,000 Source Language 9,690 Topic Fields 1,725 Pooled
Domain−Specific Multilingual Task − Box plot of the Topics of the Experiment
Domain−Specific Multilingual Task − Distribution of the Topics of the Experiment 00% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100% Average Precision 1 0.8 0.6 0.4 0.2 −0.2 −0.4 −0.6 −0.8 −1176−DS177−DS178−DS179−DS180−DS181−DS182−DS183−DS184−DS185−DS186−DS187−DS188−DS189−DS190−DS191−DS192−DS193−DS194−DS195−DS196−DS197−DS198−DS199−DS200−DS
Domain−Specific Multilingual Task − Retrieved documents vs Mean Precision
CUT_DS_MULTI_RU2X_MERGED 100% 90% 80% 70% 60% 40% 30% 20% 10% 0%5 Multilingual merged unstructured indizes translated with Google translate (Beta), PROMT and the provided thesaurus
Domain−Specific Multilingual Task − Distribution of the Topics of the Experiment 00% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Precision averages (%) for individual queries −1176−DS177−DS178−DS179−DS180−DS181−DS182−DS183−DS184−DS185−DS186−DS187−DS188−DS189−DS190−DS191−DS192−DS193−DS194−DS195−DS196−DS197−DS198−DS199−DS200−DS
Priority
Query Construction 25,000 Source Language 9,690 Topic Fields 1,892 Pooled 100% 90% 80% 70% 60% 40% 30% 20% 10% 0%0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
Domain−Specific Multilingual Task − Box plot of the Topics of the Experiment
Domain−Specific Multilingual Task − Distribution of the Topics of the Experiment 00% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100% Average Precision 1 0.8 0.6 0.4 0.2 −0.2 −0.4 −0.6 −0.8 Precision averages (%) for individual queries Domain−SpecificMultilingualTask−ComparisontoMedianR−PrecisionbyTopic(Topics176−DSto200−DS)
Domain−Specific Multilingual Task − Box plot of the Topics of the Experiment 0%5
Domain−Specific Multilingual Task − Distribution of the Topics of the Experiment 10 t n e irm 8 e xp E the 6 f so c iop 4 T f o rbe 2 m u N 00% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Overall statistics for 25 queries : Total number of documents over all queries Retrieved Relevant Relevant retrieved Priority
Query Construction 25,000 Source Language 9,690 Topic Fields 1,966 Pooled 4 AUTOMATIC German title, description false 100% 90% 80% 70% 60% 40% 30% 20% 10% 0%0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
Domain−Specific Multilingual Task − Box plot of the Topics of the Experiment 0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Domain−Specific Multilingual Task − Distribution of the Topics of the Experiment 00% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100% Average Precision 1 0.8 0.6 0.4 0.2 −0.2 −0.4 −0.6 −0.8 −1 176−DS177−DS178−DS179−DS180−DS181−DS182−DS183−DS184−DS185−DS186−DS187−DS188−DS189−DS190−DS191−DS192−DS193−DS194−DS195−DS196−DS197−DS198−DS199−DS200−DS
Precision averages (%) for individual queries Domain−SpecificMultilingualTask−ComparisontoMedianR−PrecisionbyTopic(Topics176−DSto200−DS)
Domain−Specific Multilingual Task − Box plot of the Topics of the Experiment 20% 10% 0%5 −0.2 −0.4 −0.6 −0.8
Domain−Specific Multilingual Task − Distribution of the Topics of the Experiment 00% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
−1 176−DS177−DS178−DS179−DS180−DS181−DS182−DS183−DS184−DS185−DS186−DS187−DS188−DS189−DS190−DS191−DS192−DS193−DS194−DS195−DS196−DS197−DS198−DS199−DS200−DS
Overall statistics for 25 queries : Total number of documents over all queries Retrieved Relevant Relevant retrieved Priority
Query Construction 25,000 Source Language 9,690 Topic Fields 3,323 Pooled 100% 90% 80% 70% 60% 40% 30% 20% 10% 0%0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
Domain−Specific Multilingual Task − Box plot of the Topics of the Experiment 0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Domain−Specific Multilingual Task − Distribution of the Topics of the Experiment 00% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100% Average Precision 1 0.8 0.6 0.4 0.2 −0.2 −0.4 −0.6 −0.8 Docs Cutoff Levels Precision at DCL (%) 5 docs 24.80 10 docs 23.60 15 docs 26.93 20 docs 28.60 30 docs 27.87 100 docs 27.80 200 docs 24.58 500 docs 18.44 1000 docs 13.29 R-Precision (precision after R document retrieved, where R = Relevant retrieved) 18.52
Domain−Specific Multilingual Task − Box plot of the Topics of the Experiment 0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Domain−Specific Multilingual Task − Distribution of the Topics of the Experiment 5 t n e ie4 m r xp E fth3 e so c ip2 o T f o r e b1 m u N 00% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
−0.2 −0.4 −0.6 −0.8 Precision averages (%) for individual queries 100% 90% 80% 70% 60% 40% 30% 20% 10% 0%5 Overall statistics for 25 queries : Total number of documents over all queries Retrieved Relevant Relevant retrieved Priority
Query Construction 25,000 Source Language 9,690 Topic Fields 3,237 Pooled 5 AUTOMATIC English title, description false 100% 90% 80% 70% 60% 40% 30% 20% 10% 0%0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
Domain−Specific Multilingual Task − Box plot of the Topics of the Experiment 0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Domain−Specific Multilingual Task − Distribution of the Topics of the Experiment 00% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100% Average Precision 1 0.8 0.6 0.4 0.2 ce reen 0 if D −0.2 −0.4 −0.6 −0.8 −1 176−DS177−DS178−DS179−DS180−DS181−DS182−DS183−DS184−DS185−DS186−DS187−DS188−DS189−DS190−DS191−DS192−DS193−DS194−DS195−DS196−DS197−DS198−DS199−DS200−DS
Docs Cutoff Levels Precision at DCL (%) 5 docs 24.00 10 docs 23.60 15 docs 28.27 20 docs 28.20 30 docs 27.47 100 docs 26.64 200 docs 23.62 500 docs 17.80 1000 docs 12.95 R-Precision (precision after R document retrieved, where R = Relevant retrieved) 17.84
Domain−Specific Multilingual Task − Retrieved documents vs Mean Precision
Domain−Specific Multilingual Task − Box plot of the Topics of the Experiment 0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Domain−Specific Multilingual Task − Distribution of the Topics of the Experiment 5 t n e ie4 m r xp E tfh3 e so c ip2 o T f o r e b1 m u N 00% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
2.59 19.49 11.00 7.25 39.11 18.45 36.34 0.00 4.46 34.63 8.86 17.73 1 0.8 0.6 0.4 0.2 ce reen 0 if D −0.2 −0.4 −0.6 −0.8 Precision averages (%) for individual queries −1 176−DS177−DS178−DS179−DS180−DS181−DS182−DS183−DS184−DS185−DS186−DS187−DS188−DS189−DS190−DS191−DS192−DS193−DS194−DS195−DS196−DS197−DS198−DS199−DS200−DS
TopicIdentifier 100% 90% 80% 70% 60% 40% 30% 20% 10% 0%5 Overall statistics for 25 queries : Total number of documents over all queries Retrieved Relevant Relevant retrieved Priority
Query Construction 25,000 Source Language 9,690 Topic Fields 2,125 Pooled 0.0167 Multilingual from Russian using Powertrans and 0.1259 Class Cluster Lookup 100% 90% 80% 70% 60% 40% 30% 20% 10% 0%0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
Domain−Specific Multilingual Task − Box plot of the Topics of the Experiment 0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Domain−Specific Multilingual Task − Distribution of the Topics of the Experiment 00% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100% Average Precision 1 0.8 0.6 0.4 0.2 ce reen 0 if D −0.2 −0.4 −0.6 −0.8 20 t n e m i rpe15 x E e h ftso10 cp i o T f o re 5 b m u
N −1 176−DS177−DS 178−DS179−DS180−DS181−DS182−DS 183−DS184−DS185−DS186−DS187−DS 188−DS189−DS190−DS191−DS192−DS 193−DS194−DS195−DS196−DS197−DS 198−DS199−DS200−DS
Docs Cutoff Levels Precision at DCL (%) 5 docs 22.40 10 docs 17.60 15 docs 17.33 20 docs 16.20 30 docs 15.60 100 docs 15.44 200 docs 14.44 500 docs 11.29 1000 docs 8.50 R-Precision (precision after R document retrieved, where R = Relevant retrieved) 12.38
Domain−Specific Multilingual Task − Retrieved documents vs Mean Precision
Domain−Specific Multilingual Task − Box plot of the Topics of the Experiment 0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Domain−Specific Multilingual Task − Distribution of the Topics of the Experiment 8 tn7 e m ire6 xp E ftscope45 h i o3 T f o r2 e b m u1 N 00% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
0.00 27.94 18.00 8.70 13.87 16.14 17.40 4.23 11.14 12.16 0.63 13.04 1 0.8 0.6 0.4 0.2 ce reen 0 if D −0.2 −0.4 −0.6 −0.8 Precision averages (%) for individual queries −1 176−DS 177−DS 178−DS 179−DS 180−DS 181−DS 182−DS 183−DS 184−DS 185−DS 186−DS 187−DS 188−DS 189−DS 190−DS 191−DS 192−DS 193−DS 194−DS 195−DS 196−DS 197−DS 198−DS 199−DS 200−DS
TopicIdentifier 100% 90% 80% 70% 60% 40% 30% 20% 10% 0%5 Overall statistics for 25 queries : Total number of documents over all queries Retrieved Relevant Relevant retrieved Priority
Query Construction 25,000 Source Language 9,690 Topic Fields 2,087 Pooled 6 AUTOMATIC Russian title, description false 0.0156 Multilingual from Russian using Powertrans and 0.1230 Thesaurus Lookup 100% 90% 80% 70% 60% 40% 30% 20% 10% 0%0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
Domain−Specific Multilingual Task − Box plot of the Topics of the Experiment 0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Domain−Specific Multilingual Task − Distribution of the Topics of the Experiment 00% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100% Average Precision 1 0.8 0.6 0.4 0.2 ce reen 0 if D −0.2 −0.4 −0.6 −0.8 20 t n e m i rpe15 x E e h tfso10 cp i o T f o re 5 b m u
N −1 176−DS177−DS178−DS179−DS180−DS181−DS182−DS183−DS184−DS185−DS186−DS187−DS188−DS189−DS190−DS191−DS192−DS193−DS194−DS195−DS196−DS197−DS198−DS199−DS200−DS
0.04 9.89 5.53 1.34 2.70 5.57 4.19 0.37 1.70 3.31 0.03 2.81 Precision averages (%) for individual queries Domain−Specific Multilingual Task − Retrieved documents vs Mean Precision Domain−Specific Multilingual Task − Box plot of the Topics of the Experiment 100% 90% 80% 70% 60% 40% 30% 20% 10% Interquartile range Mean Mean With No Outliers Std With No Outliers 176-DS 177-DS 178-DS 179-DS 180-DS 181-DS 182-DS 183-DS 184-DS 185-DS 186-DS 187-DS 188-DS 0.55 2.26 1.64 29.31 24.34 22.58 2.96 10.25 3.44 9.48 3.03 22.97 27.92 189-DS 190-DS 191-DS 192-DS 193-DS 194-DS 195-DS 196-DS 197-DS 198-DS 199-DS 200-DS 1