Appendix E Results of the GeoCLEF Track

-

Introduction 3

Results for CLEF 2008 GeoCLEF Track 3. Individual Experiment Results and Graphs This section provides the individual results for each official experiment. For each experiment the following tables and graphs are shown: - Overall statistics and information - Interpolated recall vs precision averages plot - Average precision statistics and box plot - Average precision comparison to median plot - Document cutoff levels vs precision at DCL plot - R-Precision statistics and box plot - R-Precision comparison to median plot Topics are identified with DOIs, as well as the experiments. The prefix for the DOI of a topic is 10.2452. The following example shows how to build the DOI for a topic given its number: for topic 200-AH, the corresponding DOI is 10.2452/200-AH List of Submitted Experiments 7 alivale alivale alivale alivale alivale alivale alivale alivale alivale United States 10.2415/GC-MONO-EN

CLEF2008.CHESHIRE.BERKGCMOENTD

United States 10.2415/GC-MONO-EN

CLEF2008.CHESHIRE.BERKGCMOENTDN

United States 10.2415/GC-MONO-EN

CLEF2008.CHESHIRE.BERKGCMOENTDNPIV United States 10.2415/GC-MONO-EN-CLEF2008.CSUSM.MONEN1 en en en en en en en en en en en en en en en en en en en en en en en en en en en en en en en en en en en en en en en en

TDN TDN TDN TDN TDN TDN TD TDN TDN TD TDN TDN TD TDN TDN TD TDN TD TD TD TD TD TD TD TD TD TD TD TD TD TD TD TD TD TD TD TD TD TDN TDN

AUTOMATIC

AUTOMATIC

AUTOMATIC

AUTOMATIC

AUTOMATIC

AUTOMATIC

AUTOMATIC yes yes yes yes yes yes yes jaen jaen jaen jaen jaen jaen pittsburgh pittsburgh pittsburgh pittsburgh valencia valencia valencia valencia valencia valencia xldb xldb xldb xldb xldb xldb xldb xldb xldb xldb xldb xldb 10.2415/GC-MONO-DE-CLEF2008.HAGEN.FUHTD20 GC-MONO-DE-CLEF2008 10.2415/GC-MONO-DE

CLEF2008.HAGEN.FUHTD01M

10.2415/GC-MONO-DE

CLEF2008.HAGEN.FUHTD20M

10.2415/GC-MONO-DE

CLEF2008.HAGEN.FUHTDN20

United States 10.2415/GC-MONO-PT

CLEF2008.CHESHIRE.BERKGCMOPTTD

United States 10.2415/GC-MONO-PT

CLEF2008.CHESHIRE.BERKGCMOPTTDN United States 10.2415/GC-MONO-PTCLEF2008.CHESHIRE.BERKGCMOPTTDNPIV

10.2415/GC-MONO-EN-CLEF2008.JAEN.EXP5 en en en en en en en en en en en en en en en en en en en en en en en en en en en en de de de de de de de de en pt en pt

TDN TDN TDN TD TDN TDN TDN TD T TN TDN TDN TDN TD TD TDN T T T T T T T T T T T T TD TDN TDN TD TD TD TD TD TDN TD TDN TDN

AUTOMATIC

AUTOMATIC

AUTOMATIC

AUTOMATIC

AUTOMATIC

yes yes yes yes yes United States 10.2415/GC-MONO-PT-CLEF2008.CSUSM.MONPT1 en pt pt pt pt pt pt pt pt pt pt pt pt pt de de de pt pt pt de en en en en en en en en pt pt en en en en en en en en pt

TD T T T T T T T T T T T T T TD TDN TDN TD TDN TDN TD TD TDN TDN TD TDN TDN TD TDN TD TDN TDN TDN TD TD TDN TD TD TDN TD hagen hagen hagen hagen csusm Track Overview Results and Graphs 13

Track Overview Results and

GeoCLEF Monolingual English

Task

Top 5 Participants − Standard Recall Levels vs Mean Interpolated 0.8 0.6 0.4 0.2 −0.2 −0.4 −0.6 −0.8 −1 Track Overview Results and Graphs 16

Track Overview Results and

GeoCLEF Monolingual English

Task

Top 5 Participants − Retrieved documents vs Mean 0.8 0.6 0.4 0.2 −0.2 −0.4 −0.6 −0.8 −1 Track Overview Results and Graphs 20 hagen hagen R_PRECISION

GeoCLEF Bilingual German Task − Distribution of the Topics of the Experiment 5 t n e ie4 m r xp E tfh3 e so c ip2 o T f o r e b1 m u N 00% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%

R−Precision hagen GeoCLEF Bilingual German Task − Distribution of the Topics of the Experiment 0%5 hagen R_PRECISION Precision averages (%) for individual queries

GeoCLEF Bilingual German Task − Distribution of the Topics of the Experiment 6 t iren5 m e xp E4 e h ftscop3 i o f2 T o r e bu1 m N 00% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%

R−Precision hagen

GeoCLEF Bilingual German Task − Box plot of the Topics of the Experiment GeoCLEF Bilingual German Task − Distribution of the Topics of the Experiment 0%5 hagen Precision averages (%) for individual queries GeoCLEFBilngualGermanTask−ComparisontoMedianR−PrecisionbyTopic(Topics100−GCto99−GC FUHPTGTDN20

GeoCLEF Bilingual German Task − Box plot of the Topics of the Experiment

GeoCLEF Bilingual German Task − Distribution of the Topics of the Experiment 00% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%

R−Precision 1 0.8 0.6 0.4 0.2 hagen

GeoCLEF Bilingual German Task − Box plot of the Topics of the Experiment GeoCLEF Bilingual German Task − Distribution of the Topics of the Experiment 00% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100% Average Precision Precision averages (%) for individual queries 0%5 hagen Precision averages (%) for individual queries

GeoCLEF Bilingual German Task − Box plot of the Topics of the Experiment

GeoCLEF Bilingual German Task − Distribution of the Topics of the Experiment 00% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%

R−Precision 1 0.8 0.6 0.4 0.2 Priority

Query Construction 25,000 Source Language 1,245 Topic Fields 894 Pooled 5 AUTOMATIC English title, description, narrative true

GeoCLEF Bilingual Portuguese Task − Box plot of the Topics of the Experiment GeoCLEF Bilingual Portuguese Task − Distribution of the Topics of the Experiment 00% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100% Average Precision Precision averages (%) for individual queries BERKBIENPTTDNPIV −1 10 −GC 76−GC 7 −GC 78−GC 79−GC 80−GC 81−GC 82−GC 83−GC 84−GC 85−GC 86−GC TopicIdentifier 8 −GC 89−GC 90−GC 91−GC 92−GC 93−GC 94−GC 95−GC 96−GC 97−GC 98−GC 9 −GC 87−GC

GeoCLEF Bilingual Portuguese Task − Retrieved documents vs Mean Precision

BERKBIENPTTDNPIV Precision averages (%) for individual queries 0%5 GeoCLEF Bilingual Portuguese Task − Box plot of the Topics of the Experiment Bilingual English to Portuguese using TREC2 with Blind Feedback merged with OKAPI on title and description and Narrative

GeoCLEF Bilingual Portuguese Task − Distribution of the Topics of the Experiment 00% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%

R−Precision 1 0.8 0.6 0.4 0.2 10 t n e irm8 e xp E the 6 f so c iop 4 T f o rbe 2 m u

N −1 10 −GC 76−GC 7 −GC 78−GC 79−GC 80−GC 81−GC 82−GC 83−GC 84−GC 85−GC 86−GC TopicIdentifier 8 −GC 89−GC 90−GC 91−GC 92−GC 93−GC 94−GC 95−GC 96−GC 97−GC 98−GC 9 −GC 87−GC Priority

Query Construction 25,000 Source Language 1,245 Topic Fields 679 Pooled 1 AUTOMATIC German title, description true 0%0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%

Recall

GeoCLEF Bilingual Portuguese Task − Box plot of the Topics of the Experiment GeoCLEF Bilingual Portuguese Task − Distribution of the Topics of the Experiment 00% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100% Average Precision Precision averages (%) for individual queries BERKGCBIDEPTTD −1 10 −GC 76−GC 7 −GC 78−GC 79−GC 80−GC 81−GC 82−GC 83−GC 84−GC 85−GC 86−GC TopicIdentifier 8 −GC 89−GC 90−GC 91−GC 92−GC 93−GC 94−GC 95−GC 96−GC 97−GC 98−GC 9 −GC 87−GC Docs Cutoff Levels Precision at DCL (%) 5 docs 19.20 10 docs 20.00 15 docs 19.20 20 docs 18.60 30 docs 17.20 100 docs 13.72 200 docs 9.58 500 docs 4.88 1000 docs 2.72 R-Precision (precision after R document retrieved, where R = Relevant retrieved) 16.05 100% 90% 80% 70% 60% 0%5 Precision averages (%) for individual queries

GeoCLEF Bilingual Portuguese Task − Box plot of the Topics of the Experiment

GeoCLEF Bilingual Portuguese Task − Distribution of the Topics of the Experiment 00% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%

R−Precision 1 0.8 0.6 0.4 0.2 15 t n e m ir e p xE10 e h tf so cp i o fT 5 o r e b m u

N −1 10 −GC 76−GC 7 −GC 78−GC 79−GC 80−GC 81−GC 82−GC 83−GC 84−GC 85−GC 86−GC TopicIdentifier 8 −GC 89−GC 90−GC 91−GC 92−GC 93−GC 94−GC 95−GC 96−GC 97−GC 98−GC 9 −GC 87−GC Priority

Query Construction 25,000 Source Language 1,245 Topic Fields 708 Pooled 2 AUTOMATIC German title, description, narrative true 0%0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%

Recall

GeoCLEF Bilingual Portuguese Task − Box plot of the Topics of the Experiment GeoCLEF Bilingual Portuguese Task − Distribution of the Topics of the Experiment 00% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100% Average Precision Precision averages (%) for individual queries BERKGCBIDEPTTDN −1 10 −GC 76−GC 7 −GC 78−GC 79−GC 80−GC 81−GC 82−GC 83−GC 84−GC 85−GC 86−GC TopicIdentifier 8 −GC 89−GC 90−GC 91−GC 92−GC 93−GC 94−GC 95−GC 96−GC 97−GC 98−GC 9 −GC 87−GC

GeoCLEF Bilingual Portuguese Task − Retrieved documents vs Mean Precision

BERKGCBIDEPTTDN Precision averages (%) for individual queries 20% 10% 0%5 GeoCLEF Bilingual Portuguese Task − Box plot of the Topics of the Experiment Bilingual German to Portuguese using TREC2 with Blind Feedback on title and description and narrative

GeoCLEF Bilingual Portuguese Task − Distribution of the Topics of the Experiment 00% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%

R−Precision 1 0.8 0.6 0.4 0.2 −0.2 −0.4 −0.6 −0.8 15 t n e m ir e p xE10 e h ft so cp i o fT 5 o r e b m u

N −1 10 −GC 76−GC 7 −GC 78−GC 79−GC 80−GC 81−GC 82−GC 83−GC 84−GC 85−GC 86−GC TopicIdentifier 8 −GC 89−GC 90−GC 91−GC 92−GC 93−GC 94−GC 95−GC 96−GC 97−GC 98−GC 9 −GC 87−GC Overall statistics for 25 queries : Total number of documents over all queries Retrieved Relevant Relevant retrieved Priority

Query Construction 25,000 Source Language 1,245 Topic Fields 716 Pooled 6 AUTOMATIC English title, description, narrative true 0%0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%

Recall

GeoCLEF Bilingual Portuguese Task − Box plot of the Topics of the Experiment 0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%

Average Precision GeoCLEF Bilingual Portuguese Task − Distribution of the Topics of the Experiment −0.2 −0.4 −0.6 −0.8 15 t n e m ir e p xE10 e h tf so cp i o fT 5 o r e b m u N 00% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100% Average Precision Precision averages (%) for individual queries BERKGCBIDEPTTDNPIV −1 10 −GC 76−GC 7 −GC 78−GC 79−GC 80−GC 81−GC 82−GC 83−GC 84−GC 85−GC 86−GC TopicIdentifier 8 −GC 89−GC 90−GC 91−GC 92−GC 93−GC 94−GC 95−GC 96−GC 97−GC 98−GC 9 −GC 87−GC

GeoCLEF Bilingual Portuguese Task − Retrieved documents vs Mean Precision

BERKGCBIDEPTTDNPIV Precision averages (%) for individual queries 20% 10% 0%5 GeoCLEF Bilingual Portuguese Task − Box plot of the Topics of the Experiment Bilingual German to Portuguese using TREC2 with Blind Feedback merged with OKAPI on title and description and narrative

GeoCLEF Bilingual Portuguese Task − Distribution of the Topics of the Experiment 00% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%

R−Precision 1 0.8 0.6 0.4 0.2 ce reen 0 if D −0.2 −0.4 −0.6 −0.8 15 t n e m ir e p xE10 e h tf so cp i o fT 5 o r e b m u

N −1 10 −GC 76−GC 7 −GC 78−GC 79−GC 80−GC 81−GC 82−GC 83−GC 84−GC 85−GC 86−GC TopicIdentifier 8 −GC 89−GC 90−GC 91−GC 92−GC 93−GC 94−GC 95−GC 96−GC 97−GC 98−GC 9 −GC 87−GC Overall statistics for 25 queries : Total number of documents over all queries Retrieved Relevant Relevant retrieved Priority

Query Construction 25,000 Source Language 1,245 Topic Fields 818 Pooled 3 AUTOMATIC English title, description true

GeoCLEF Bilingual Portuguese Task − Standard Recall Levels vs Mean Interpolated Precision BERKGCBIENPTTD 0%0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%

Recall

GeoCLEF Bilingual Portuguese Task − Box plot of the Topics of the Experiment 0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%

Average Precision GeoCLEF Bilingual Portuguese Task − Distribution of the Topics of the Experiment −0.2 −0.4 −0.6 −0.8 15 t n e m ir e p xE10 e h tf so cp i o fT 5 o r e b m u N 00% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100% Average Precision Precision averages (%) for individual queries BERKGCBIENPTTD −1 10 −GC 76−GC 7 −GC 78−GC 79−GC 80−GC 81−GC 82−GC 83−GC 84−GC 85−GC 86−GC TopicIdentifier 8 −GC 89−GC 90−GC 91−GC 92−GC 93−GC 94−GC 95−GC 96−GC 97−GC 98−GC 9 −GC 87−GC Docs Cutoff Levels Precision at DCL (%) 5 docs 24.00 10 docs 24.40 15 docs 22.93 20 docs 22.00 30 docs 21.47 100 docs 17.12 200 docs 12.06 500 docs 5.86 1000 docs 3.27 R-Precision (precision after R document retrieved, where R = Relevant retrieved) 20.31 100% 90% 80% 70% 60% 40% 30% 20% 10% 0%5 21.25 0.00 15.00 25.00 0.00 89.13 0.00 33.33 5.00 7.14 46.67 0.00 Precision averages (%) for individual queries GeoCLEFBilngualPortugueseTask−ComparisontoMedianR−PrecisionbyTopic(Topics100−GCto99−GC BERKGCBIENPTTD

GeoCLEF Bilingual Portuguese Task − Box plot of the Topics of the Experiment 0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%

R−Precision

GeoCLEF Bilingual Portuguese Task − Distribution of the Topics of the Experiment 00% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%

R−Precision 1 0.8 0.6 0.4 0.2 ce reen 0 if D −0.2 −0.4 −0.6 −0.8 −1 10 −GC 76−GC 7 −GC 78−GC 79−GC 80−GC 81−GC 82−GC 83−GC 84−GC 85−GC 86−GC TopicIdentifier 8 −GC 89−GC 90−GC 91−GC 92−GC 93−GC 94−GC 95−GC 96−GC 97−GC 98−GC 9 −GC 87−GC Overall statistics for 25 queries : Total number of documents over all queries Retrieved Relevant Relevant retrieved Priority

Query Construction 25,000 Source Language 1,245 Topic Fields 844 Pooled 4 AUTOMATIC English title, description, narrative true 0%0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%

Recall

GeoCLEF Bilingual Portuguese Task − Box plot of the Topics of the Experiment 0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%

Average Precision GeoCLEF Bilingual Portuguese Task − Distribution of the Topics of the Experiment 15.47 0.18 2.34 46.31 0.03 88.20 0.00 21.29 6.22 11.65 36.55 0.01 1 0.8 0.6 0.4 0.2 ce reen 0 if D −0.2 −0.4 −0.6 −0.8 00% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100% Average Precision Precision averages (%) for individual queries BERKGCBIENPTTDN −1 10 −GC 76−GC 7 −GC 78−GC 79−GC 80−GC 81−GC 82−GC 83−GC 84−GC 85−GC 86−GC TopicIdentifier 8 −GC 89−GC 90−GC 91−GC 92−GC 93−GC 94−GC 95−GC 96−GC 97−GC 98−GC 9 −GC 87−GC

GeoCLEF Bilingual Portuguese Task − Retrieved documents vs Mean Precision

BERKGCBIENPTTDN Precision averages (%) for individual queries BERKGCBIENPTTDN 100% 90% 80% 70% 60% 40% 30% 20% 10% 0%5 GeoCLEF Bilingual Portuguese Task − Box plot of the Topics of the Experiment Bilingual English to Portuguese using TREC2 with Blind Feedback on title and description and Narrative 21.25 0.00 7.50 50.00 0.00 89.13 0.00 16.67 12.50 19.64 50.00 0.00 0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%

R−Precision

GeoCLEF Bilingual Portuguese Task − Distribution of the Topics of the Experiment 00% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%

R−Precision 1 0.8 0.6 0.4 0.2 ce reen 0 if D −0.2 −0.4 −0.6 −0.8 10 t n e irm8 e xp E the 6 f so c iop 4 T f o rbe 2 m u

N −1 10 −GC 76−GC 7 −GC 78−GC 79−GC 80−GC 81−GC 82−GC 83−GC 84−GC 85−GC 86−GC TopicIdentifier 8 −GC 89−GC 90−GC 91−GC 92−GC 93−GC 94−GC 95−GC 96−GC 97−GC 98−GC 9 −GC 87−GC Overall statistics for 25 queries : Total number of documents over all queries Retrieved Relevant Relevant retrieved Priority

Query Construction 25,000 Source Language 1,245 Topic Fields 275 Pooled 1 AUTOMATIC English title, description true 0.0003 English topics Portuguese docs 0.0213 20% 10% 0%0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%

Recall

GeoCLEF Bilingual Portuguese Task − Box plot of the Topics of the Experiment 0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%

Average Precision GeoCLEF Bilingual Portuguese Task − Distribution of the Topics of the Experiment 0.03 0.00 0.11 0.00 0.43 0.00 0.00 0.00 0.76 0.45 1.11 0.00 1 0.8 0.6 0.4 0.2 ce reen 0 if D −0.2 −0.4 −0.6 −0.8 25 t n e irm20 e xp E the15 f so c iop10 T f o rbe 5 m u N 00% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100% Average Precision Precision averages (%) for individual queries −1 10 −GC 76−GC 7 −GC 78−GC 79−GC 80−GC 81−GC 82−GC 83−GC 84−GC 85−GC 86−GC TopicIdentifier 8 −GC 89−GC 90−GC 91−GC 92−GC 93−GC 94−GC 95−GC 96−GC 97−GC 98−GC 9 −GC 87−GC Docs Cutoff Levels Precision at DCL (%) 5 docs 3.20 10 docs 3.60 15 docs 3.73 20 docs 3.20 30 docs 3.07 100 docs 3.36 200 docs 2.98 500 docs 1.89 1000 docs 1.10 R-Precision (precision after R document retrieved, where R = Relevant retrieved) 100% 90% 80% 70% 60% Interquartile range Mean Mean With No Outliers Std With No Outliers 100-GC 76-GC 77-GC 78-GC 79-GC 80-GC 81-GC 82-GC 83-GC 84-GC 85-GC 86-GC 87-GC EN2PT1

GeoCLEF Bilingual Portuguese Task − Box plot of the Topics of the Experiment

GeoCLEF Bilingual Portuguese Task − Distribution of the Topics of the Experiment 1 25

References

0.0025 Bilingual German to Portuguese using TREC2 with 0.1441 Blind Feedback on title and description 0.0188 Bilingual English to Portuguese using TREC2 with 0.1811 Blind Feedback on title and description