How do Named Entities Contribute to Retrieval Effectiveness? Thomas Mandl, Christa Womser-Hacker University of Hildesheim, Information Science, Marienburger Platz 22 D-31141 Hildesheim, Germany {mandl,womser}@uni-hildesheim.de Abstract. Named entities in topics are a major factor contributing to the quality of retrieval results. In this paper, we report on an analysis on the correlation between the number of named entities present in a topic and the retrieval quality achieved for these topics by retrieval systems within CLEF. We found that a medium positive correlation exists for German, English and Spanish topics. Furthermore, we analyze the effect of the document or target language on the retrieval quality. 1 Introduction Within CLEF, many efforts are made to improve retrieval systems. This body of work allows the identification of successful approaches, algorithms and tools in CLIR (Braschler & Peters 2004). We believe, the knowledge and work dedicated to this effort can be exploited beyond the optimization of individual systems. The amount of data created by organizers and participants remains a valuable source of knowledge awaiting exploration. Many lessons can still be learned from evaluation initiatives such as CLEF, TREC (Voorhees & Buckland 2002), INEX (Fuhr 2003) or NTCIR (Oyama, Ishida & Kando 2003). Ultimately, further criteria and metrics for the evaluation of search and retrieval methods may be detected. This could lead to improved algorithms, quality criteria, resources and tools in CLIR (Schneider et al. 2004). This general research approach is illustrated in figure 1. The identification of patterns in the systems’ performance for topics with specific items may lead to improvements in system development. C001 Arquitectura C001en Berlín Encontrar documentos Arquitectura sobre la en Berlín C001 arquitectura en Berlín. Encontrar documentos sobre la Arquitectura Los documentos arquitectura en general, relevantes en Berlín en Berlín. sobre los rasgos Encontrar arquitectónicos tratan, documentos de Berlín Los documentos o, en relevantes sobre la tratan, Topic particular, sobre en general, arquitectónicos particular, la reconstrucción arquitectura sobre de en Berlín. los rasgos Berlín o, en sobre la relevantes Los documentos reconstrucción tratan, en general, sobre los Properties rasgos arquitectónicos de Berlín Multilingual Topic Set Topic Properties Stop Stem StemIndex Stop mer words Patterns StemIndex Stop mer words Index words mer Weights BRF Weights BRF Weights BRF Results per Systems Run and Topic Figure 1. Overview of the approach Our current analysis concentrates on named entities within the topics of CLEF. Named entities frequently occur in CLEF as part of the topic formulation. Table 1 gives an overview. Table 1. Name of named entities in the CLEF topics CLEF Number Total number of Average number of named Standard deviation of year of topics named entities entities in topics named entities in topics 2001 50 60 1.20 1.06 2002 50 86 1.72 1.54 2003 60 97 1.62 1.18 The large number of named entities in the topic set shows that they are a subject worth studying. The large number may be due to the fact that the document corpus for CLEF consists of newspaper texts. We can also observe an increase of named entities per topic in 2002 compared to 2001. Because of the effect of named entities on retrieval performance (Mandl & Womser-Hacker 2004c), the number of named entities needs to be carefully monitored. Table 2 shows how the named entities are distributed over groups with different numbers of named entities and shows the tasks analyzed in this paper. Table 2. Overview of named entities in CLEF tasks CLEF Task Topic Nr. Topics without Topics with one or Topics with more than year language runs named entities two named entities three named entities 2002 Mono German 21 12 21 17 2002 Mono Spanish 28 11 18 21 2002 Bi German 4 12 21 17 2002 Multi German 4 12 21 17 2002 Bi English 51 14 21 15 2002 Multi English 32 14 21 15 2003 Mono English 11 8 14 6 2003 Mono Spanish 38 6 33 21 2003 Multi Spanish 10 6 33 21 2003 Mono German 30 9 40 10 2003 Bi German 24 9 40 10 2003 Multi German 1 9 40 10 2003 Bi English 8 9 41 10 2003 Multi English 74 9 41 10 2 Named Entities in Topics and Retrieval Performance In a study presented at CLEF in 2003, we showed a correlation between the number of named entities present in topics and the systems’ performance for these topics (Mandl & Womser-Hacker 2004b). In this paper, we extend the analysis to Spanish and monolingual tasks. In our earlier analysis, the relation was shown for English and German. Including Spanish will show, whether this effect can be revealed for another topic language. By including monolingual tasks, we may be able to compare the strength of the effect between cross- and monolingual retrieval tasks. Named entities were intellectually assessed according to the schema of Sekine et al. 2002. The performance of the systems was extracted from the CLEF proceedings. The average precision for a topic is calculated as the average precision of all systems for a individual topic. From the average precision for a topic, we can calculate the average of all topics which contain n named entities. Figure 2 and 3 show the average precision for topics with n named entities for tasks in CLEF 3 and CLEF 4. 0,8 0,7 Monolingual English 0,6 Monolingual German Average Precision 0,5 Monolingual Spanisch 0,4 Bilingual Topic Language German 0,3 Multilingual Spanisch 0,2 Multilingual English 0,1 0 0 1 2 3 4 5 6 7 Number of named entities Figure 2. Average precision for topics with n named entities in CLEF 3 (in 2002) In figure 2 and 3 we can observe that monolingual tasks generally result in higher average precision than cross- lingual tasks. The precision tends to be better when more named entities are present. The relation previously observed for German and English can also be seen for Spanish. 0 .8 0 .7 M o n o lin g u a l G e rm a n 0 .6 M o n o lin g u a l S p a n is c h Average precision 0 .5 B ilin g ua l T o p ic L a n g u a g e E n g lis h 0 .4 M u ltilin g u a l E n g lis h 0 .3 M u ltilin g u a l G e rm a n 0 .2 0 .1 0 0 2 4 6 8 N u m b e r o f n a m e d e n titie s Figure 3. Average precision for topics with n named entities in CLEF 4 (in 2003) We also calculate the correlation between the number of named entities and the average precision per topic for each of the tasks. The results are presented in table 3 and 4. Table 3. Correlation between the number of named entities in topic and the average system performance per topic for tasks in CLEF 3 Monolingual Monolingual Bilingual Topic Multilingual Multilingual German Spanish Language English German English 0.449 0.207 0.399 0.428 0.294 Table 4. Correlation between the number of named entities in topic and the average system performance per topic for tasks in CLEF 4 Monolingual Monolingual Monolingual Bilingual Topic Multilingual Multilingual German Spanisch English Language German Spanisch English 0.372 0.385 0.158 0.213 0.213 0.305 We can observe that the correlation is in most cases higher for the monolingual task. That would mean, that named entities help systems more in monolingual retrieval than in cross-lingual retrieval. However, English seems to be an exception in CLEF 4, because the correlation is almost twice as strong in the multilingual task. 3 Potential for Optimization based on Named Entities The systems tested at CLEF perform differently well for topics with different numbers of named entities. Although proper names make topics easier in general and for almost all runs, the performance of systems varies within the three classes of topics based on the number of named entities. We distinguished three classes of topics, (a) the first class with no proper names called none, (b) the second class with one and two named entities called few and (c) one class with three or more named entities called lots. The patterns of the systems are strikingly different for the three classes. As a consequence, there seems to be potential to improve system by fusion based on the number of named entities in a topic. Many systems already apply fusion techniques. We propose a simple fusion rule. First, the number of named entities is determined for each topic. Subsequently, this topic is channeled to the system with the best performance for this named entity class. The best system is a combination of at most three runs. Each category of topics is answered by the optimal system within a group of systems for that number of named entities. The groups were selected from the original CLEF ranking of the runs in one task. We used a window of five runs. That means, five neighboring runs by systems which perform similarly well overall are grouped and fused by our approach. Table 5 shows the improvement by the fusion based on the optimal selection of a system for each category of topics. The highest levels of improvement are achieved for the topic language English. For 2002, we observe the highest improvement of 10% for the bilingual runs. Table 5. Improvement through named entity based fusion CLEF Run type Topic Average. precision Optimal average precision Improvement year language best run name fusion over best run 2001 Bilingual German 0.509 0.518 2% 2001 Multilingual English 0.405 0.406 0% 2002 Bilingual English 0.494 0.543 10% 2002 Multilingual English 0.378 0.403 6.5% 2003 Bilingual German 0.460 0.460 0% 2003 Bilingual English 0.348 0.369 6.1% 2003 Multilingual English 0.438 0.443 1.2% This approach regards the systems as black boxes and requires no knowledge about the treatment of named entities within the systems. Considering the linguistic processing within the systems might be even more rewarding. Potentially, further analysis might reveal which approaches, which components and which parameters are especially suited for topics with and without named entities. This analysis shows that the performance of retrieval systems can be optimized by channeling topics to the systems best appropriated for topics without, with one or two and with three and more names. Certainly, the application of this fusion on the past results approach is artificial and the number of topics in each subgroup is not sufficient for a statistically reliable result (Voorhees & Buckley 2002). Furthermore, in our study, the number of named entities was determined intellectually. However, this mechanism can be easily implemented by using an automatic named entity recognizer. We intend to apply this fusion technique in an upcoming CLEF task as one element of the fusion framework MIMOR (Mandl & Womser-Hacker 2004a, Hackl et al 2004). 4 Named Entities in Topics and Retrieval Performance for Target Languages So far, our studies have been focused to the language of the initial topic which participants used for their retrieval efforts. Additionally, we have analyzed the effect of the target or document language. In this case, we cannot consider the multilingual tasks where there are several target languages. The monolingual tasks have already been analyzed in section 2 and are also considered here. Therefore, this analysis is targeted at bilingual retrieval tasks. We grouped all bilingual runs with English, German and Spanish as document language. The correlation between the number of named entities in the topics and the average precision of all systems for that topic was calculated. The average precision may be interpreted as the difficulty of the topic. The following table shows the results of this analysis. Table 6. Correlation for target languages for CLEF 3 and 4 CLEF Task Target Number Correlation between number of named year type language of runs entities and average precision 2003 Mono English 11 0.158 2002 Bi English 16 0.577 2003 Bi English 15 0.187 2002 Mono German 21 0.372 2003 Mono German 30 0.449 2002 Bi German 13 0.443 2003 Bi German 3 0.379 2002 Mono Spanish 28 0.385 2003 Mono Spanish 38 0.207 2002 Bi Spanish 16 0.166 2003 Bi Spanish 25 0.427 First, we can see a positive correlation for all tasks considered. Named entities support the retrieval also from the perspective of the document language. This results for the year 2002 may be a hint, that retrieval in English or German document collections profits more from named entities in the topic than Spanish. However, in 2003, the opposite is the case and English and Spanish switch. For German, there are only 3 runs in 2003. As a consequence, we cannot yet detect any language dependency for the effect of named entities on retrieval performance. 5 Outlook In this paper a strong relation between named entities in topics and the performance of retrieval systems for these topics was confirmed. This finding allows us to formulate a hint for searchers and users of retrieval systems: Whenever you can think of a name related to your retrieval problem, consider including it in the query. In addition, our results encourage further analysis of other topic features. We are especially considering a part of speech (POS) analysis of the CLEF topics. Acknowledgements We would like to thank Martin Braschler for providing the crucial data for our study. Furthermore, we acknowledge the work of several students from the University of Hildesheim who contributed to this analysis as part of their course work. References Braschler, Martin; Peters, Carol (2004): Cross-Language Evaluation Forum: Objectives, Results, Achievements. In: Information Retrieval. no. 7. pp. 7-31. Fuhr, Norbert (2003) (ed.): Initiative for the Evaluation of XML Retrieval (INEX) : INEX 2003 Workshop Proceedings, Dagstuhl, Germany, December 15-17, 2003. http://purl.oclc.org/NET/duett-07012004-093151 Hackl, René; Kölle, Ralph; Mandl, Thomas; Ploedt, Alexandra; Scheufen, Jan-Hendrik; Womser-Hacker, Christa (2004): Multilingual Retrieval Experiments with MIMOR at the University of Hildesheim. To appear in: Peters, Carol; Braschler, Martin; Gonzalo, Julio; Kluck, Michael (eds.): Evaluation of Cross- Language Information Retrieval Systems. Proceedings of the CLEF 2003 Workshop. Berlin et al.: Springer [Lecture Notes in Computer Science] Mandl, Thomas; Womser-Hacker, Christa (2004a): A Framework for long-term Learning of Topical User Preferences in Information Retrieval. In: New Library World. vol. 105 (5/6). pp. 184-195. Mandl, Thomas; Womser-Hacker, Christa (2004b): Analysis of Topic Features in Cross-Language Information Retrieval Evaluation. In: 4th International Conference on Language Resources and Evaluation (LREC) Lisbon, Portugal, May 24-30. Workshop Lessons Learned from Evaluation: Towards Transparency and Integration in Cross-Lingual Information Retrieval (LECLIQ). pp. 17-19. Mandl, Thomas; Womser-Hacker, Christa (2004c): Proper Names in the Multilingual CLEF Topic Set. To appear in: Peters, Carol; Braschler, Martin; Gonzalo, Julio; Kluck, Michael (eds.): Evaluation of Cross- Language Information Retrieval Systems. Proceedings of the CLEF 2003 Workshop. Berlin et al.: Springer [Lecture Notes in Computer Science] Preprint: http://clef.iei.pi.cnr.it:2002/2003/WN_web/53.pdf Oyama, Keizo; Ishida, Emi; Kando, Noriko (2003) (eds.): NTCIR Workshop3 Proceedings of the Third NTCIR Workshop on research in information Retrieval, Automatic Text Summarization and Question Answering (September2001-October2002) http://research.nii.ac.jp/ntcir/workshop/OnlineProceedings3/index.html Schneider, René; Mandl, Thomas; Womser-Hacker, Christa (2004): Workshop LECLIQ: Lessons Learned from Evaluation: Towards Integration and Transparency in Cross-Lingual Information Retrieval with a special Focus on Quality Gates. In: 4th International Conference on Language Resources and Evaluation (LREC) Lisbon, Portugal, May 24-30. Workshop Lessons Learned from Evaluation: Towards Transparency and Integration in Cross-Lingual Information Retrieval (LECLIQ). pp. 1-4. Sekine, Satoshi; Sudo, Kiyoshi; Nobata, Chikashi (2002): Extended Named Entity Hierarchy. In: Proceedings of Third International Conference on Language Resources and Evaluation (LREC 2002); Las Palmas, Canary Islands, Spain. Voorhees, Ellen; Buckland, Lori (2002) (eds.): The Eleventh Text Retrieval Conference (TREC 2002). NIST Special Publication 500-251. National Institute of Standards and Technology. Gaithersburg, Maryland. November 2002. http://trec.nist.gov/pubs/trec11/t11_proceedings.html Voorhees, Ellen; Buckley, Chris (2002): The Effect of Topic Set Size on Retrieval Experiment Error. In: Proc Annual Intl. ACM Conference on Research and Development in Information Retrieval (SIGIR ‘02) Tampere, Finland. pp. 316-323. Womser-Hacker, Christa (2002): Multilingual Topic Generation within the CLEF 2001 Experiments. In: Peters, Carol; Braschler, Martin; Gonzalo, Julio; Kluck, Michael (eds.): Evaluation of Cross-Language Information Retrieval Systems. Springer [LNCS 2406] pp. 389-393.