Named entities in topics are a major factor contributing to the quality of retrieval results. In this paper, we report on an analysis on the correlation between the number of named entities present in a topic and the retrieval quality achieved for these topics by retrieval systems within CLEF. We found that a medium positive correlation exists for German, English and Spanish topics. Furthermore, we analyze the effect of the document or target language on the retrieval quality.
<top> <num> C001 <S-ti<tloep> Arqui<tneucmt>urCa00e1n Berlín <S-de<sSc->ti<tloep>> EnconAtrrqauri<tndeuocmct>umrCea0n0teon1sBseorblríen la ea<anrSrq-qgunueiai<LE<ntrtSonSeere-sc-rc>cnodattdne<EA<lóurotsSnrS,nrrcrc-q-ia>ua>doutscmreniiooeestttbsnndcrelrto>aceedBocrt>eesulrmuoBlrerdseínaornltcrl.eoueaívsmnsnaegnsnootots,ebsroeesntrlsaaotbarne, la Los daorcqumietnetcotsurraeleenvaBnetrelsínBt.erartlaínn, partiecnulgaern,eqrusaioltb,ercestolbuareareleconosnBsretarsuglcoícsni.ón arqui<tSe-cntaórnri>cos de Berlín o, en partiLcousladro,cusmobernetolsa rreelcoenvsatnrtuecsción tratan, en general, sobre los rasgos arquitectónicos de Berlín
worSdtsopmeSrtemIndex worSdtsopmeSrtemIndex
Systems
Patterns
Our current analysis concentrates on named entities within the topics of CLEF. Named entities frequently occur
in CLEF as part of the topic formulation. Table 1 gives an overview.
The large number of named entities in the topic set shows that they are a subject worth studying. The large
number may be due to the fact that the document corpus for CLEF consists of newspaper texts. We can also
observe an increase of named entities per topic in 2002 compared to 2001. Because of the effect of named
entities on retrieval performance
Named entities were intellectually assessed according to the schema of
3 4 Number of named entities 5 6
7 In figure 2 and 3 we can observe that monolingual tasks generally result in higher average precision than crosslingual tasks. The precision tends to be better when more named entities are present.
2
4 N um ber of n am ed en tities 6 8
M onolingual G erm an M onolingual S panisch B ilingual Topic Language E nglish M ultilingual E nglish M ultilingual G erm an We also calculate the correlation between the number of named entities and the average precision per topic for each of the tasks. The results are presented in table 3 and 4. The systems tested at CLEF perform differently well for topics with different numbers of named entities. Although proper names make topics easier in general and for almost all runs, the performance of systems varies within the three classes of topics based on the number of named entities. We distinguished three classes of topics, (a) the first class with no proper names called none, (b) the second class with one and two named entities called few and (c) one class with three or more named entities called lots. The patterns of the systems are strikingly different for the three classes. As a consequence, there seems to be potential to improve system by fusion based on the number of named entities in a topic. Many systems already apply fusion techniques. We propose a simple fusion rule. First, the number of named entities is determined for each topic. Subsequently, this topic is channeled to the system with the best performance for this named entity class. The best system is a combination of at most three runs. Each category of topics is answered by the optimal system within a group of systems for that number of named entities. The groups were selected from the original CLEF ranking of the runs in one task. We used a window of five runs. That means, five neighboring runs by systems which perform similarly well overall are grouped and fused by our approach. Table 5 shows the improvement by the fusion based on the optimal selection of a system for each category of topics.
The highest levels of improvement are achieved for the topic language English. For 2002, we observe the highest improvement of 10% for the bilingual runs.
CLEF year 2001 2001 2002 2002 2003 2003 2003
This approach regards the systems as black boxes and requires no knowledge about the treatment of named entities within the systems. Considering the linguistic processing within the systems might be even more rewarding. Potentially, further analysis might reveal which approaches, which components and which parameters are especially suited for topics with and without named entities.
This analysis shows that the performance of retrieval systems can be optimized by channeling topics to the
systems best appropriated for topics without, with one or two and with three and more names. Certainly, the
application of this fusion on the past results approach is artificial and the number of topics in each subgroup is
not sufficient for a statistically reliable result
Named Entities in Topics and Retrieval Performance for Target Languages So far, our studies have been focused to the language of the initial topic which participants used for their retrieval efforts. Additionally, we have analyzed the effect of the target or document language. In this case, we cannot consider the multilingual tasks where there are several target languages. The monolingual tasks have already been analyzed in section 2 and are also considered here. Therefore, this analysis is targeted at bilingual retrieval tasks. We grouped all bilingual runs with English, German and Spanish as document language. The correlation between the number of named entities in the topics and the average precision of all systems for that topic was calculated. The average precision may be interpreted as the difficulty of the topic. The following table shows the results of this analysis. In this paper a strong relation between named entities in topics and the performance of retrieval systems for these topics was confirmed. This finding allows us to formulate a hint for searchers and users of retrieval systems: Whenever you can think of a name related to your retrieval problem, consider including it in the query. In addition, our results encourage further analysis of other topic features. We are especially considering a part of speech (POS) analysis of the CLEF topics.
We would like to thank Martin Braschler for providing the crucial data for our study. Furthermore, we acknowledge the work of several students from the University of Hildesheim who contributed to this analysis as part of their course work.