Introduction 3
Results for CLEF 2008 Ad-hoc Persian@CLEF Track 3. Individual Experiment Results and Graphs This section provides the individual results for each official experiment. For each experiment the following tables and graphs are shown: - Overall statistics and information - Interpolated recall vs precision averages plot - Average precision statistics and box plot - Average precision comparison to median plot - Document cutoff levels vs precision at DCL plot - R-Precision statistics and box plot - R-Precision comparison to median plot Topics are identified with DOIs, as well as the experiments. The prefix for the DOI of a topic is 10.2452. The following example shows how to build the DOI for a topic given its number: for topic 200-AH, the corresponding DOI is 10.2452/200-AH List of Submitted Experiments 7 AH-PERSIAN-MONO-FA-CLEF2008
TDN MANUAL
yes fa fa fa en fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa en en en en en en en
TD T T T T T TDN TDN TDN TDN TDN TDN TDN TDN TDN TDN TDN T T T TD TD TDN TD TD TD TD T T T AH-PERSIAN-MONO-FA-CLEF2008 tehran-NLPDB Iran, Islamic 10.2415/AH-PERSIAN-BILI-X2FA
Republic Of CLEF2008.TEHRAN-NLPDB.UTNLPDB1BT4G tehran-NLPDB Iran, Islamic 10.2415/AH-PERSIAN-BILI-X2FA
Republic Of CLEF2008.TEHRAN-NLPDB.UTNLPDB1BT5 Track Overview Results and Graphs 12 100% 90% 80% 70% 60% 50% 40% 30% 20% 0.8 0.6 0.4 0.2 −0.2 −0.4 −0.6 −0.8 −1 1 0.8 0.6 0.4 0.2 −0.2 −0.4 −0.6 −0.8 −1 Recall Levels vs Mean Interpolated
Track Overview Results and Graphs 15 100% 90% 80% 70% 60% 50% 40% 30% 20% 0.8 0.6 0.4 0.2 −0.2 −0.4 −0.6 −0.8 −1 1 0.8 0.6 0.4 0.2 −0.2 −0.4 −0.6 −0.8 −1
Track Overview Results and Graphs 19
Track Overview Results and Graphs 20 1 0.8 0.6 0.4 0.2 ce reen 0 if D−0.2 −0.4 −0.6 −0.8 1 0.8 0.6 0.4 0.2 ce reen 0 if D−0.2 −0.4 −0.6 −0.8
Ad−Hoc Persian Bilingual Persian Task − Distribution of the Topics of the Experiment 5 t n e ie4 m r xp E fth3 e so c ip2 o T f o r e b1 m u N 00% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Average Precision 1 0.8 0.6 0.4 0.2 ce reen 0 if D−0.2 −0.4 −0.6 −0.8 1 0.8 0.6 0.4 0.2 ce reen 0 if D−0.2 −0.4 −0.6 −0.8
Ad−Hoc Persian Bilingual Persian Task − Distribution of the Topics of the Experiment 8 tn7 e m ire6 xp E ftcspoe45 h i o3 T f o r2 e b m u1 N 00% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
R−Precision 1 0.8 0.6 0.4 0.2 ce reen 0 if D−0.2 −0.4 −0.6 −0.8 1 0.8 0.6 0.4 0.2 ce reen 0 if D−0.2 −0.4 −0.6 −0.8
Ad−Hoc Persian Bilingual Persian Task − Distribution of the Topics of the Experiment 8 tn7 e m ire6 xp E tfsceop54 h i o3 T f o r2 e b m u1 N 00% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Average Precision 0%5 Ad−Hoc Persian Bilingual Persian Task − Box plot of the Topics of the Experiment Precision averages (%) for individual queries 1 0.8 0.6 0.4 0.2 ce reen 0 if D−0.2 −0.4 −0.6 −0.8 1 0.8 0.6 0.4 0.2 ce reen 0 if D−0.2 −0.4 −0.6 −0.8
Ad−Hoc Persian Bilingual Persian Task − Distribution of the Topics of the Experiment 8 tn7 e m ire6 xp E ftscoep45 h i o3 T f o r2 e b m u1 N 00% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
R−Precision 0.0054 Bilingual Persian-English query translation using 0.1021 all meaning Precision averages (%) for individual queries 1 0.8 0.6 0.4 0.2 ce reen 0 if D−0.2 −0.4 −0.6 −0.8 1 0.8 0.6 0.4 0.2 ce reen 0 if D−0.2 −0.4 −0.6 −0.8 40
Ad−Hoc Persian Bilingual Persian Task − Box plot of the Topics of the Experiment
Ad−Hoc Persian Bilingual Persian Task − Distribution of the Topics of the Experiment 00% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Average Precision 0%5 Ad−Hoc Persian Bilingual Persian Task − Box plot of the Topics of the Experiment Precision averages (%) for individual queries 1 0.8 0.6 0.4 0.2 ce reen 0 if D−0.2 −0.4 −0.6 −0.8 1 0.8 0.6 0.4 0.2 ce reen 0 if D−0.2 −0.4 −0.6 −0.8 25 t n e irm20 e xp E the15 f so c iop10 T f o rbe 5 m u N
Ad−Hoc Persian Bilingual Persian Task − Distribution of the Topics of the Experiment Precision averages (%) for individual queries 1 0.8 0.6 0.4 0.2 ce reen 0 if D−0.2 −0.4 −0.6 −0.8 1 0.8 0.6 0.4 0.2 ce reen 0 if D−0.2 −0.4 −0.6 −0.8
Ad−Hoc Persian Bilingual Persian Task − Box plot of the Topics of the Experiment
Ad−Hoc Persian Bilingual Persian Task − Distribution of the Topics of the Experiment 00% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Average Precision
0%5 Ad−Hoc Persian Bilingual Persian Task − Box plot of the Topics of the Experiment
Ad−Hoc Persian Bilingual Persian Task − Distribution of the Topics of the Experiment 00% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100% R−Precision 1 0.8 0.6 0.4 0.2 ce reen 0 if D−0.2 −0.4 −0.6 −0.8 1 0.8 0.6 0.4 0.2 ce reen 0 if D−0.2 −0.4 −0.6 −0.8 15 Ad−HocPersianBilngualPersianTask−ComparisontoMedianR−PrecisionbyTopic(Topics576−AHto600−AH)
0.0095 Bilingual Persian-English query translation using 0.1537 first meaning Precision averages (%) for individual queries 1 0.8 0.6 0.4 0.2 ce reen 0 if D−0.2 −0.4 −0.6 −0.8 1 0.8 0.6 0.4 0.2 ce reen 0 if D−0.2 −0.4 −0.6 −0.8
Ad−Hoc Persian Bilingual Persian Task − Box plot of the Topics of the Experiment
Ad−Hoc Persian Bilingual Persian Task − Distribution of the Topics of the Experiment 00% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Average Precision
0%5 Ad−Hoc Persian Bilingual Persian Task − Box plot of the Topics of the Experiment R_PRECISION Precision averages (%) for individual queries 1 0.8 0.6 0.4 0.2 ce reen 0 if D−0.2 −0.4 −0.6 −0.8 1 0.8 0.6 0.4 0.2 ce reen 0 if D−0.2 −0.4 −0.6 −0.8 20 t n e m i rpe15 x E e h ftso10 cp i o T f o re 5 b m u N
Ad−Hoc Persian Bilingual Persian Task − Distribution of the Topics of the Experiment 00% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
R−Precision Precision averages (%) for individual queries 1 0.8 0.6 0.4 0.2 ce reen 0 if D−0.2 −0.4 −0.6 −0.8 1 0.8 0.6 0.4 0.2 ce reen 0 if D−0.2 −0.4 −0.6 −0.8 20 t n e m i rpe15 x E e h tfso10 cp i o T f o re 5 b m u N
Ad−Hoc Persian Bilingual Persian Task − Box plot of the Topics of the Experiment
Ad−Hoc Persian Bilingual Persian Task − Distribution of the Topics of the Experiment 00% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Average Precision
0%5 Ad−Hoc Persian Bilingual Persian Task − Box plot of the Topics of the Experiment
Ad−Hoc Persian Bilingual Persian Task − Distribution of the Topics of the Experiment 00% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100% R−Precision 1 0.8 0.6 0.4 0.2 ce reen 0 if D−0.2 −0.4 −0.6 −0.8 1 0.8 0.6 0.4 0.2 ce reen 0 if D−0.2 −0.4 −0.6 −0.8 15 Ad−HocPersianBilngualPersianTask−ComparisontoMedianR−PrecisionbyTopic(Topics576−AHto600−AH)
0.0175 Bilingual Persian-English query translation using 0.1474 top 5 meaning Precision averages (%) for individual queries 1 0.8 0.6 0.4 0.2 ce reen 0 if D−0.2 −0.4 −0.6 −0.8 1 0.8 0.6 0.4 0.2 ce reen 0 if D−0.2 −0.4 −0.6 −0.8 30 t n iem25 r e p xE20 e h ftso15 cp i o fT10 o r e b m5 u N
Ad−Hoc Persian Bilingual Persian Task − Box plot of the Topics of the Experiment
Ad−Hoc Persian Bilingual Persian Task − Distribution of the Topics of the Experiment 00% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Average Precision
20% 10% 0%5 Ad−Hoc Persian Bilingual Persian Task − Box plot of the Topics of the Experiment R_PRECISION Precision averages (%) for individual queries 1 0.8 0.6 0.4 0.2 ce reen 0 if D−0.2 −0.4 −0.6 −0.8 1 0.8 0.6 0.4 0.2 ce reen 0 if D−0.2 −0.4 −0.6 −0.8 15 0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
R−Precision
Ad−Hoc Persian Bilingual Persian Task − Distribution of the Topics of the Experiment 00% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
R−Precision Overall statistics for 50 queries : Total number of documents over all queries Retrieved Relevant Relevant retrieved 0.0215 Bilingual Persian-English query translation using 0.1847 top 5 meaning and 5-grams after SWD Precision averages (%) for individual queries 1 0.8 0.6 0.4 0.2 ce reen 0 if D−0.2 −0.4 −0.6 −0.8 1 0.8 0.6 0.4 0.2 ce reen 0 if D−0.2 −0.4 −0.6 −0.8
Ad−Hoc Persian Bilingual Persian Task − Box plot of the Topics of the Experiment 0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Average Precision
Ad−Hoc Persian Bilingual Persian Task − Distribution of the Topics of the Experiment 00% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Average Precision
Ad−Hoc Persian Bilingual Persian Task − Retrieved documents vs Mean Precision 20% 10% 0%5 Ad−Hoc Persian Bilingual Persian Task − Box plot of the Topics of the Experiment
Ad−Hoc Persian Bilingual Persian Task − Distribution of the Topics of the Experiment 00% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100% R−Precision 1 0.8 0.6 0.4 0.2 ce reen 0 if D−0.2 −0.4 −0.6 −0.8 1 0.8 0.6 0.4 0.2 ce reen 0 if D−0.2 −0.4 −0.6 −0.8 15 Ad−HocPersianBilngualPersianTask−ComparisontoMedianR−PrecisionbyTopic(Topics576−AHto600−AH)
Overall statistics for 50 queries : Total number of documents over all queries Retrieved Relevant Relevant retrieved Priority
Query Construction 5,000 Source Language 5,161 Topic Fields 1,234 Pooled 10 AUTOMATIC English title, description true 0.0390 CrossLanguage Document Translation with Description 0.1721 Run Precision averages (%) for individual queries 1 0.8 0.6 0.4 0.2 ce reen 0 if D−0.2 −0.4 −0.6 −0.8 1 0.8 0.6 0.4 0.2 ce reen 0 if D−0.2 −0.4 −0.6 −0.8 20 t n e m i rpe15 x E e h ftso10 cp i o T f o re 5 b m u N
Ad−Hoc Persian Bilingual Persian Task − Box plot of the Topics of the Experiment 0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Average Precision
Ad−Hoc Persian Bilingual Persian Task − Distribution of the Topics of the Experiment 00% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Average Precision
Docs Cutoff Levels Precision at DCL (%) 5 docs 40.00 10 docs 40.40 15 docs 38.67 20 docs 36.80 30 docs 33.80 100 docs 24.68 200 docs 12.34 500 docs 4.94 1000 docs 2.47 R-Precision (precision after R document retrieved, where R = Relevant retrieved) 20.18 100% 90% 80% 70% 60% 40% 30% 20% 10% 0%5 Ad−Hoc Persian Bilingual Persian Task − Box plot of the Topics of the Experiment
Ad−Hoc Persian Bilingual Persian Task − Retrieved documents vs Mean Precision
Precision averages (%) for individual queries 1 0.8 0.6 0.4 0.2 ce reen 0 if D−0.2 −0.4 −0.6 −0.8 1 0.8 0.6 0.4 0.2 ce reen 0 if D−0.2 −0.4 −0.6 −0.8 0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
R−Precision
Ad−Hoc Persian Bilingual Persian Task − Distribution of the Topics of the Experiment 8 tn7 e m ire6 xp E ftcsope45 h i o3 T f o r2 e b m u1 N 00% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
R−Precision
Overall statistics for 50 queries : Total number of documents over all queries Retrieved Relevant Relevant retrieved Priority
Query Construction 5,000 Source Language 5,161 Topic Fields 592 Pooled 9 AUTOMATIC English title, description true 0.0035 CrossLanguage Query Translation with Description 0.0897 Run Precision averages (%) for individual queries 1 0.8 0.6 0.4 0.2 ce reen 0 if D−0.2 −0.4 −0.6 −0.8 1 0.8 0.6 0.4 0.2 ce reen 0 if D−0.2 −0.4 −0.6 −0.8 40 ten35 m i rpe30 x E ftseo2250 h cp i foT15 o re10 b m u 5 N
Ad−Hoc Persian Bilingual Persian Task − Box plot of the Topics of the Experiment 0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Average Precision
Ad−Hoc Persian Bilingual Persian Task − Distribution of the Topics of the Experiment 00% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Average Precision
Docs Cutoff Levels Precision at DCL (%) 5 docs 18.00 10 docs 17.00 15 docs 16.53 20 docs 15.80 30 docs 15.53 100 docs 11.84 200 docs 5.92 500 docs 2.37 1000 docs 1.18 R-Precision (precision after R document retrieved, where R = Relevant retrieved) 11.12 100% 90% 80% 70% 60% 40% 30% 20% 10% 0%5 Ad−Hoc Persian Bilingual Persian Task − Box plot of the Topics of the Experiment R_PRECISION Precision averages (%) for individual queries 15.86 7.69 0.74 5.33 4.95 0.53 8.98 37.11 0.00 13.43 0.80 1.64 0.00 36.71 42.11 2.86 17.86 41.18 0.00 15.79 3.40 39.81 0.73 14.29 14.56 1 0.8 0.6 0.4 0.2 ce reen 0 if D−0.2 −0.4 −0.6 −0.8 1 0.8 0.6 0.4 0.2 ce reen 0 if D−0.2 −0.4 −0.6 −0.8 30 t n iem25 r e p xE20 e h ftso15 cp i o fT10 o r e b m 5 u N
0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
R−Precision
Ad−Hoc Persian Bilingual Persian Task − Distribution of the Topics of the Experiment 00% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
R−Precision Overall statistics for 50 queries : Total number of documents over all queries Retrieved Relevant Relevant retrieved Priority
Query Construction 5,000 Source Language 5,161 Topic Fields 677 Pooled 0.0035 CrossLanguage Query Translation Run 0.1152
100% 90% 80% 70% 60% 40% 30% 20% 10% Precision averages (%) for individual queries 27.37 11.47 0.05 3.65 0.29 0.00 0.37 35.44 0.00 2.63 0.01 0.26 0.00 12.28 32.96 0.22 2.15 28.95 0.07 0.72 0.99 24.00 0.00 3.70 6.38 1 0.8 0.6 0.4 0.2 ce reen 0 if D−0.2 −0.4 −0.6 −0.8 1 0.8 0.6 0.4 0.2 ce reen 0 if D−0.2 −0.4 −0.6 −0.8 40
Ad−Hoc Persian Bilingual Persian Task − Box plot of the Topics of the Experiment 0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Average Precision
Ad−Hoc Persian Bilingual Persian Task − Distribution of the Topics of the Experiment 00% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Average Precision
100% 90% 80% 70% 60% 500 1000 Ad−Hoc Persian Bilingual Persian Task − Box plot of the Topics of the Experiment Interquartile range Mean Mean With No Outliers Std With No Outliers 551-AH 552-AH 553-AH 554-AH 555-AH 556-AH 557-AH 558-AH 559-AH 560-AH 561-AH 562-AH 563-AH 564-AH 565-AH 566-AH 567-AH 568-AH 569-AH 570-AH 571-AH 572-AH 573-AH 574-AH 575-AH Precision averages (%) for individual queries 30 t n iem25 r e p xE20 e h ftso15 cp i o fT10 o r e b m 5 u N
Ad−Hoc Persian Bilingual Persian Task − Distribution of the Topics of the Experiment