Appendix D Results of the GeoCLEF Track

-

Introduction 3

Results for CLEF 2007 GeoCLEF Track 3. Individual Experiment Results and Graphs This section provides the individual results for each official experiment. For each experiment the following tables and graphs are shown: - Overall statistics and information - Interpolated recall vs precision averages plot - Average precision statistics and box plot - Average precision comparison to median plot - Document cutoff levels vs precision at DCL plot - R-Precision statistics and box plot - R-Precision comparison to median plot Topics are identified with DOIs, as well as the experiments. The prefix for the DOI of a topic is 10.2452. The following example shows how to build the DOI for a topic given its number: for topic 200-AH, the corresponding DOI is 10.2452/200-AH List of Submitted Experiments 7

GC-MONO-DE-CLEF2007

de de de de de de de de de de de de de es es es en en es en en en en en en en en en en en en en en es es en en en en en TD TD TDN TD TD TDN TDN TD TD TD TDN TDN TD TD TDN TD TD TD TDN TDN TDN TD TD TDN TD TD TDN TDN T TD TDN TDN TDN TD TDN TD TD TDN TDN TDN no no no yes yes yes yes no TDN MANUAL no icl linguit linguit linguit linguit moscow moscow msasia msasia msasia msasia msasia valencia valencia valencia valencia valencia valencia valencia valencia valencia valencia valencia valencia en en en en en es en en en en en en en en en en en en en en en en en en en en en en en pt es pt pt pt pt pt pt pt pt pt T TD T T TD TN TD TDN TD TDN TD TD TDN TD TDN TD TDN TD TD TD TD TD TD TD T T T T TD TD TD TDN TD TDN TD T T T T yes yes no no no yes yes yes yes yes

AUTOMATIC

AUTOMATIC

AUTOMATIC

AUTOMATIC

AUTOMATIC

csusm

GC-BILI-X2DE-CLEF2007

GC-BILI-X2DE-CLEF2007

GC-BILI-X2DE-CLEF2007

GC-BILI-X2DE-CLEF2007

GC-BILI-X2DE-CLEF2007

GC-BILI-X2DE-CLEF2007

GC-BILI-X2DE-CLEF2007

GC-BILI-X2DE-CLEF2007

Track Overview Results and Graphs 12 30% 20% 10% 0%

0% 1 0.8 0.6 0.4 0.2 e c n re 0 e iff D −0.2 −0.4 −0.6 −0.8 −1 51−GC 52−GC 53−GC 54−GC 55−GC 56−GC 57−GC 58−GC 59−GC 60−GC 61−GC 62−GC 63−GC 64−GC 65−GC 66−GC 67−GC 68−GC 69−GC 70−GC 71−GC 72−GC 73−GC 74−GC 75−GC Topic Identifier GeoCLEF Monolingual German Task − Box Plot of the Topics hildesheim [Experiment HIMODENE2NA; MAP 20.67%; Pooled] csusm [Experiment GEOMODE6; MAP 20.38%; Pooled] csusm [Experiment GEOMODE1; MAP 20.38%; Pooled] s t n e m i r e p x E hildesheim [Experiment HIMODEBASE; MAP 20.19%; Pooled]

hagen [Experiment FUHTD6DE; MAP 19.57%; Pooled] hildesheim [Experiment HIMODENE2; MAP 19.53%; Pooled] hagen [Experiment FUHTD2DE; MAP 19.11%; Pooled] hagen [Experiment FUHTD3DE; MAP 19.09%; Pooled] hildesheim [Experiment HIMODENE3; MAP 17.95%; Pooled] cheshire [Experiment BERKMODEBASE; MAP 13.92%; Pooled] csusm [Experiment GEOMODE3; MAP 11.70%; Pooled] csusm [Experiment GEOMODE2; MAP 11.70%; Pooled] 0% 10% 20% 30% 70% 80% 90% 100% 40% 50% 60%

Average Precision 10.2455/TUKEY_T_TEST.40A773D08F2E0F044438139E14B327C3

GeoCLEF Monolingual German Task − Tukey T test with "top group" highlighted FUHTDN5DE FUHTDN4DE GEOMODE4 GEOMODE5 GEOMODE1

GEOMODE6 s t n e m i r e p x E

HIMODENE2NA

FUHTD6DE HIMODEBASE

FUHTD3DE HIMODENE2

FUHTD2DE HIMODENE3 GEOMODE2

GEOMODE3 BERKMODEBASE

30 100 200 Retrieved Documents (logarithmic scale) 500 1000 GeoCLEF Monolingual German Task Top 5 Participants − Comparison to Median R−Precision by Topic (Topics 51−GC to 75−GC) hagen [Experiment FUHTDN5DE; R−Prec 26.58%; Pooled] csusm [Experiment GEOMODE5; R−Prec 22.99%; Pooled] hildesheim [Experiment HIMODENE2NA; R−Prec 23.41%; Pooled] cheshire [Experiment BERKMODEBASE; R−Prec 16.36%; Pooled] −1 51−GC 52−GC 53−GC 54−GC 55−GC 56−GC 57−GC 58−GC 59−GC 60−GC 61−GC 62−GC 63−GC 64−GC 65−GC 66−GC 67−GC 68−GC 69−GC 70−GC 71−GC 72−GC 73−GC 74−GC 75−GC Topic Identifier hildesheim [Experiment HIMODENE2NA; R−Prec 23.41%; Pooled] s t n e m i r e p x E csusm [Experiment GEOMODE5; R−Prec 22.99%; Pooled] csusm [Experiment GEOMODE4; R−Prec 22.99%; Pooled] hagen [Experiment FUHTD6DE; R−Prec 22.68%; Pooled] csusm [Experiment GEOMODE6; R−Prec 22.10%; Pooled] csusm [Experiment GEOMODE1; R−Prec 22.10%; Pooled] hagen [Experiment FUHTD2DE; R−Prec 21.66%; Pooled] hagen [Experiment FUHTD3DE; R−Prec 21.58%; Pooled] hildesheim [Experiment HIMODEBASE; R−Prec 19.93%; Pooled] hildesheim [Experiment HIMODENE2; R−Prec 19.48%; Pooled] hildesheim [Experiment HIMODENE3; R−Prec 18.60%; Pooled] cheshire [Experiment BERKMODEBASE; R−Prec 16.36%; Pooled] csusm [Experiment GEOMODE3; R−Prec 14.75%; Pooled] csusm [Experiment GEOMODE2; R−Prec 14.75%; Pooled] 0% 10% 20% 30% 40% 60% 70% 80% 90%

100% 50%

R−Precision

GeoCLEF Monolingual German Task − Tukey T test with "top group" highlighted s t n e m i r e p x E Average precision (non-interpolated) for all relevant documents (averaged over queries) depok

GeoCLEF Bilingual English Task − Box plot of the Topics of the Experiment

GeoCLEF Bilingual English Task − Distribution of the Topics of the Experiment 00% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%

Average Precision Precision averages (%) for individual queries 0%5

GeoCLEF Bilingual English Task − Distribution of the Topics of the Experiment 00% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%

R−Precision 20 30 40 50 60 70 80 90 100 Average precision (non-interpolated) for all relevant documents (averaged over queries) depok 0.0198 Query translation using machine translation. query 0.1716 expansion is done by georeference processing 100% 90% 80% 70% 60% 40% 30% 20% 10%

GeoCLEF Bilingual English Task − Distribution of the Topics of the Experiment 15 Precision averages (%) for individual queries depok

Docs Cutoff Levels Precision at DCL (%)

0%5

GeoCLEF Bilingual English Task − Box plot of the Topics of the Experiment

GeoCLEF Bilingual English Task − Distribution of the Topics of the Experiment 15 t n e m ir e p xE10 e h ft so cp i o fT 5 o r e b m u N depok GeoCLEF Bilingual English Task − Box plot of the Topics of the Experiment 0 10 20 30 40 50 60 70 80 90 100 Average precision (non-interpolated) for all relevant documents (averaged over queries) Precision averages (%) for individual queries

GeoCLEF Bilingual English Task − Distribution of the Topics of the Experiment 00% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100% Average Precision Precision averages (%) for individual queries depok Query translation using machine translation. query expansion is done by georeference processing and pseudo relevance feeddback on geo terms 0%5

GeoCLEF Bilingual English Task − Box plot of the Topics of the Experiment

GeoCLEF Bilingual English Task − Distribution of the Topics of the Experiment 00% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100% R−Precision depok

GeoCLEF Bilingual English Task − Box plot of the Topics of the Experiment

GeoCLEF Bilingual English Task − Distribution of the Topics of the Experiment 00% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100% Average Precision Query translation using machine translation. query expansion is done by georeference processing and common pseudo relevance Feedback 0%5 depok Precision averages (%) for individual queries GeoCLEFBilingualEnglishTask−ComparisontoMedianR−PrecisionbyTopic(Topics51−GCto75−GC)

UIBITGPPF5

GeoCLEF Bilingual English Task − Box plot of the Topics of the Experiment

GeoCLEF Bilingual English Task − Distribution of the Topics of the Experiment 00% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100% R−Precision

Priority

Query Construction 25,000 Source Language 762 Topic Fields 352 Pooled

1

AUTOMATIC

German

title, description true 0%0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%

Recall

GeoCLEF Bilingual Portuguese Task − Box plot of the Topics of the Experiment

GeoCLEF Bilingual Portuguese Task − Distribution of the Topics of the Experiment 00% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100% Average Precision 1 0.8 0.6 0.4 0.2 −0.2 −0.4 −0.6 −0.8 20 t n e m i rpe15 x E e h ftso10 cp i o T f o re 5 b m u

N Precision averages (%) for individual queries GeoCLEF Bilingual Portuguese Task − Box plot of the Topics of the Experiment 0%5

GeoCLEF Bilingual Portuguese Task − Distribution of the Topics of the Experiment 00% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100% R−Precision

Priority

Query Construction 25,000 Source Language 762 Topic Fields 402 Pooled

2

AUTOMATIC

English

title, description true

GeoCLEF Bilingual Portuguese Task − Box plot of the Topics of the Experiment

GeoCLEF Bilingual Portuguese Task − Distribution of the Topics of the Experiment 8 tn7 e m ire6 xp E tfsceop54 h i o3 T f o r2 e b m u1 N 00% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100% Average Precision Precision averages (%) for individual queries GeoCLEF Bilingual Portuguese Task − Box plot of the Topics of the Experiment 0%5

GeoCLEF Bilingual Portuguese Task − Distribution of the Topics of the Experiment 00% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100% R−Precision −1 51−GC 52−GC 53−GC 54−GC 55−GC 56−GC 57−GC 58−GC 59−GC 60−GC 61−GC 62−GC 63−GC 64−GC 65−GC 66−GC 67−GC 68−GC 69−GC 70−GC 71−GC 72−GC 73−GC 74−GC 75−GC

TopicIdentifier

Priority

Query Construction 25,000 Source Language 762 Topic Fields 465 Pooled

3

AUTOMATIC

Spanish; Castilian

title, description true 0%0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%

Recall

GeoCLEF Bilingual Portuguese Task − Box plot of the Topics of the Experiment

GeoCLEF Bilingual Portuguese Task − Distribution of the Topics of the Experiment 00% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100% Average Precision 1 0.8 0.6 0.4 0.2 −0.2 −0.4 −0.6 −0.8 Baseline Spanish to Portuguese using Powertrans and searching using TREC2 ranking with Blind Feedback Precision averages (%) for individual queries GeoCLEF Bilingual Portuguese Task − Box plot of the Topics of the Experiment 20% 10% 0%5 −0.2 −0.4 −0.6 −0.8

GeoCLEF Bilingual Portuguese Task − Distribution of the Topics of the Experiment 00% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100% R−Precision −1 51−GC 52−GC 53−GC 54−GC 55−GC 56−GC 57−GC 58−GC 59−GC 60−GC 61−GC 62−GC 63−GC 64−GC 65−GC 66−GC 67−GC 68−GC 69−GC 70−GC 71−GC 72−GC 73−GC 74−GC 75−GC

TopicIdentifier 10 20 30 40 50 60 70 80 90 100 Average precision (non-interpolated) for all relevant documents (averaged over queries) csusm Overall statistics for 25 queries : Total number of documents over all queries

Retrieved

Relevant

Relevant retrieved

1

AUTOMATIC

Spanish; Castilian

title, description true 0.0011 Spanish topics, English docs, title & description 0.0408 −0.2 −0.4 −0.6 −0.8 100% 90% 80% 70% 60% 40% 30% 20% 10%

GeoCLEF Bilingual Portuguese Task − Box plot of the Topics of the Experiment 0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%

Average Precision

GeoCLEF Bilingual Portuguese Task − Distribution of the Topics of the Experiment 20 Precision averages (%) for individual queries csusm

Docs Cutoff Levels Precision at DCL (%)

−0.2 −0.4 −0.6 −0.8 100% 90% 80% 70% 60% 40% 30% 20% 10% 0%5

GeoCLEF Bilingual Portuguese Task − Box plot of the Topics of the Experiment 0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%

R−Precision

GeoCLEF Bilingual Portuguese Task − Distribution of the Topics of the Experiment 20 00% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100% R−Precision −1 51−GC 52−GC 53−GC 54−GC 55−GC 56−GC 57−GC 58−GC 59−GC 60−GC 61−GC 62−GC 63−GC 64−GC 65−GC 66−GC 67−GC 68−GC 69−GC 70−GC 71−GC 72−GC 73−GC 74−GC 75−GC

TopicIdentifier 10 20 30 40 50 60 70 80 90 100 Average precision (non-interpolated) for all relevant documents (averaged over queries) csusm Overall statistics for 25 queries : Total number of documents over all queries

Retrieved

Relevant

Relevant retrieved

2

AUTOMATIC

Spanish; Castilian

title, description true 0.0010 Spanish topic's title & description with relevance 0.0407 feedback −0.2 −0.4 −0.6 −0.8 100% 90% 80% 70% 60% 40% 30% 20% 10% −1 51−GC 52−GC 53−GC 54−GC 55−GC 56−GC 57−GC 58−GC 59−GC 60−GC 61−GC 62−GC 63−GC 64−GC 65−GC 66−GC 67−GC 68−GC 69−GC 70−GC 71−GC 72−GC 73−GC 74−GC 75−GC

TopicIdentifier

GeoCLEF Bilingual Portuguese Task − Box plot of the Topics of the Experiment 0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%

Average Precision

GeoCLEF Bilingual Portuguese Task − Distribution of the Topics of the Experiment 20 Precision averages (%) for individual queries csusm

Docs Cutoff Levels Precision at DCL (%)

−0.2 −0.4 −0.6 −0.8 100% 90% 80% 70% 60% 40% 30% 20% 10% 0%5

GeoCLEF Bilingual Portuguese Task − Box plot of the Topics of the Experiment 0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%

R−Precision

GeoCLEF Bilingual Portuguese Task − Distribution of the Topics of the Experiment 20 00% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100% R−Precision −1 51−GC 52−GC 53−GC 54−GC 55−GC 56−GC 57−GC 58−GC 59−GC 60−GC 61−GC 62−GC 63−GC 64−GC 65−GC 66−GC 67−GC 68−GC 69−GC 70−GC 71−GC 72−GC 73−GC 74−GC 75−GC

TopicIdentifier 10 20 30 40 50 60 70 80 90 100 Average precision (non-interpolated) for all relevant documents (averaged over queries) csusm Overall statistics for 25 queries : Total number of documents over all queries

Retrieved

Relevant

Relevant retrieved

3

AUTOMATIC

Spanish; Castilian

title, description, narrative true 0.0010 Spanish topic's title, description, & narrative 0.0407 ce reen 0 if D −0.2 −0.4 −0.6 −0.8 100% 90% 80% 70% 60% 40% 30% 20% 10% −1 51−GC 52−GC 53−GC 54−GC 55−GC 56−GC 57−GC 58−GC 59−GC 60−GC 61−GC 62−GC 63−GC 64−GC 65−GC 66−GC 67−GC 68−GC 69−GC 70−GC 71−GC 72−GC 73−GC 74−GC 75−GC

TopicIdentifier

GeoCLEF Bilingual Portuguese Task − Box plot of the Topics of the Experiment 0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%

Average Precision

GeoCLEF Bilingual Portuguese Task − Distribution of the Topics of the Experiment 20 Precision averages (%) for individual queries csusm

Docs Cutoff Levels

Precision at DCL (%)

ce reen 0 if D −0.2 −0.4 −0.6 −0.8 100% 90% 80% 70% 60% 40% 30% 20% 10% 0%5

GeoCLEF Bilingual Portuguese Task − Box plot of the Topics of the Experiment 0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%

R−Precision

GeoCLEF Bilingual Portuguese Task − Distribution of the Topics of the Experiment 20 00% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100% R−Precision −1 51−GC 52−GC 53−GC 54−GC 55−GC 56−GC 57−GC 58−GC 59−GC 60−GC 61−GC 62−GC 63−GC 64−GC 65−GC 66−GC 67−GC 68−GC 69−GC 70−GC 71−GC 72−GC 73−GC 74−GC 75−GC

TopicIdentifier 10 20 30 40 50 60 70 80 90 100 Average precision (non-interpolated) for all relevant documents (averaged over queries) csusm Overall statistics for 25 queries : Total number of documents over all queries

Retrieved

Relevant

Relevant retrieved

4

AUTOMATIC

Spanish; Castilian

title, description, narrative true 0.0011 Spanish topic's title, description, & narrative 0.0408 with relevance feedback ce reen 0 if D −0.2 −0.4 −0.6 −0.8 100% 90% 80% 70% 60% 40% 30% 20% 10% −1 51−GC 52−GC 53−GC 54−GC 55−GC 56−GC 57−GC 58−GC 59−GC 60−GC 61−GC 62−GC 63−GC 64−GC 65−GC 66−GC 67−GC 68−GC 69−GC 70−GC 71−GC 72−GC 73−GC 74−GC 75−GC

TopicIdentifier

GeoCLEF Bilingual Portuguese Task − Box plot of the Topics of the Experiment 0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%

Average Precision

GeoCLEF Bilingual Portuguese Task − Distribution of the Topics of the Experiment 20 t n e m i rpe15 x E e h ftso10 cp i o T f o re 5 b m u N

Interquartile range

Mean

Mean With No Outliers

Std With No Outliers 51-GC 52-GC 53-GC 54-GC 55-GC 56-GC 57-GC 58-GC 59-GC 60-GC 61-GC 62-GC 63-GC R-Precision (precision after R document retrieved, where R = Relevant retrieved) Precision averages (%) for individual queries GEOBIESPT4 1 100% 90% 80% 70% 40% 30% 20% 10% 500 1000

GeoCLEF Bilingual Portuguese Task − Box plot of the Topics of the Experiment

GeoCLEF Bilingual Portuguese Task − Distribution of the Topics of the Experiment

References