In the last years, various tools for automatic semantic annotation of textual information have emerged. The main challenge of all approaches is to solve ambiguity of natural language and assign unique semantic entities according to the present context. To compare the di erent approaches a ground truth namely an annotated benchmark is essential. But, besides the actual disambiguation approach the achieved evaluation results are also dependent on the characteristics of the benchmark dataset and the expressiveness of the dictionary applied to determine entity candidates. This paper presents statistical analyses and mapping experiments on di erent benchmarks and dictionaries to identify characteristics and structure of the respective datasets.
One essential step in understanding textual information is the identi cation of semantic concepts within natural language texts. Therefore multiple Named Entity Recognition systems have been developed and become integrated in content management and information retrieval systems to handle the ood of information.
We have to distinguish between Named Entity Recognition (NER) systems that refer to nding meaningful entities within a given natural language text that are of a speci c predetermined type (as e. g., persons, locations, or organizations) and Named Entity Disambiguation (NED) systems (sometimes also referred to as Named Entity Mapping or Named Entity Linking ) that take the NER process one step further by interpreting named entities to assign a unique meaning (entity ) to a sequence of terms. In order to achieve this, rst all potential entity candidates for a phrase have to be determined with the help of a dictionary. The number of potential entity candidates corresponds to the level of ambiguity of the underlying text phrase. Taking into account the context of the phrase, as e. g. the sentence where the phrase occurs, a unique entity is selected according to the meaning of the phrase in a subsequent disambiguation step.
Multiple e orts compete in this discipline. But, the comparison of di erent NED systems is di cult, especially if they don't use a common dictionary for entity candidate determination. Therefore, it is highly desirable to provide common benchmarks for evaluation. On the other hand, benchmarks are applied to tune a NED system for its intended purpose and/or a speci c domain, i. e. context and pragmatics of the NED system are xed to a speci c task. To achieve this multiple benchmark datasets have been created to evaluate such systems. To evaluate a NED system and to compare its performance against already existing solutions the system's developer should be aware of the characteristics of the available benchmarks.
In this paper, prominent datasets { dictionary datasets as well as benchmark datasets { are analyzed to gain better insights about both their characteristics and on their capabilities while considering also potential drawbacks. The datasets are statistically analyzed for mapping coverage, level of ambiguity, maximum achievable recall, as well as di culty. All benchmarks and evaluation results are available online to achieve more target-oriented evaluations of NER and NED systems.
The paper is organized as follows: Section 2 gives an overview on NED tools and comparison approaches and introduces the benchmarks and dictionaries utilized in this paper. Statistical information about the benchmarks are presented in Section 3. Experiments using four di erent dictionaries on three di erent benchmarks are described and discussed in Section 4. Section 5 concludes the paper and summarizes the scienti c contribution. 2
Semantic annotation of textual information in web documents has become a key
technology for data mining and information retrieval and a key itself towards the Semantic
Web. Several tools for automatic semantic annotation have emerged for this task and
created a strong demand for evaluation benchmarks to enable comparison. Therefore,
a number of benchmarks containing natural language texts annotated with
semantic entities have been created. Cornolti et al. present a benchmarking framework for
entity-annotation tools and also compare the performances of various systems [
DBpedia Spotlight is an established NED application that applies an analytical
approach for the disambiguation process. Every entity candidate of a surface form
found in the text is represented by a vector composed of all terms that co-occurred
within the same paragraphs of the Wikipedia articles where this entity is linked [
AIDA is an online tool for disambiguation of named entities in natural language text
and tables [
The benchmark datasets under consideration contain annotated texts linking enclosed lexemes to entities. Based on these benchmarks the performance of NED systems can be evaluated. Within this work, we restrict our selection of benchmark datasets to those containing (a) english language texts (b) originating from authentic documents (e. g. newswire), (c) containing annotations to DBpedia entities or Wikipedia articles, and (d) involving context at least on sentence level.
The DBpedia Spotlight dataset [
KORE 50 (AIDA) [
The Wikilinks Corpus [
For further processing NIF representations of KORE 50 and DBpedia Spotlight have
been created, which are accessible at our website3. Further datasets not considered
in this paper are e. g. the complete AIDA/CoNLL corpus [
The three benchmark datasets under consideration cover di erent domains, e. g. though all datasets originate from authentic corporas varying portions have been selected and di erent types of entities have been annotated. Table 1 shows the distribution of DBpedia types within the benchmark dataset.
About 10% of the annotated entities in the DBpedia Spotlight dataset are locations and majority of about 80% of the annotated entities are not associated with any type information in DBpedia. Since the DBpedia Spotlight dataset originates from New York Times articles, the annotations are embedded in document contexts.
The KORE 50 dataset contains 144 annotations which mostly refer to agents (74 times dbo:Person and 28 times dbo:Organisation). Only a relatively small amount (18.5%) of annotated entities does not provide any type information in DBpedia. The context for the annotated entities in the KORE 50 dataset is limited to (relatively short) sentences.
The by far largest dataset is Wikilinks. Its sheer size allows to extract sub-benchmarks for speci c designated domains, e. g. there are about 281,000 mentions of 8,594 di erent diseases. However, a large amount (66%) of the annotated entities does not provide any type information in DBpedia and the largest amount of the typed entities refer to an agent (18.9%). 4
The benchmarks described in Section 3.1 are constructed to evaluate NED algorithms. The evaluation results of a NED method are not only dependent on the actual algorithm used to disambiguate ambiguous mentions but also on the structure of the benchmark and the underlying dictionary utilized to determine entity candidates for a mention. A mention mapping or mapped mention refers to a mention of a benchmark that is assigned to one or more entity candidates of the used dictionary. The following section introduces several dictionaries. 4.1
Dictionaries contain associations that map strings (surface forms) to entities represented by Wikipedia articles or DBpedia concepts. Typically, dictionaries are applied by NED systems in an early step to nd candidates for lexemes in natural language texts. In a further (disambiguation) step the actual correct entity has to be selected from all these candidates.
The DBpedia Lexicalizations dataset [
Google has released a similar, but far larger dataset: Crosswiki [
The AIDA Means dictionary is an extended version of the YAGO25 means
relation. The YAGO means relation is harvested from disambiguations pages, redirects,
and links in Wikipedia [
In addition to the three already existing dictionaries described above, we have constructed an own dictionary. Similar to the YAGO means relation this dictionary has been constructed by solving disambiguation pages and redirects and using these alternative labels additionally to the original labels of the DBpedia entities. Except the elimination of bracket terms (e. g. the label Berlin (2009 lm) is converted to Berlin by removing the brackets and the term within them) no further preprocessing has been performed on this dictionary. Thus, all labels are presented in original case sensitivity. Further evaluation on this issue is described in Section 4.3. This dictionary is subsequently referred to as RDM (Redirect D isambiguation M apping). 4.2
To identify several characteristics of the introduced dictionaries as well as consolidate assumptions about the structure of the benchmarks the experiments described in the following sections have been conducted. For performance issues only a subset of the Wikilinks benchmark has been used for the following experiments. For the subset the rst dump le containing 494,512 annotations and 192,008 distinct mentions and assigned entities has been used.
Mapping Coverage First, the coverage of mention mappings is calculated. All annotated entity mentions from the benchmarks are looked up in the four di erent dictionaries. If at least one entity candidate for the mention is found in the dictionary a counter is increased. This measure is an indicator for the expressiveness and versatility of the dictionary.
Entity Candidate Count For all mapped mentions the number of entity candidates found in the respective dictionary is added up. The number of entity candidates corresponds to the level of ambiguity of the mention and can be considered as an indicator for the level of di culty of the subsequent disambiguation process.
Maximum Recall The list of entity candidates for all mapped mentions is looked up whether the annotated entity (from the benchmark) is included. Only if it is contained in the list, a correct disambiguation is achievable at all. Thus, this measure predicts the maximum achievable recall using the respective dictionary on the benchmark. Recall and Precision achieved by Popularity For Word Sense Disambiguation (WSD) after determining entity candidates for the mentions a subsequent disambiguation process tries to detect the most relevant entity of all candidates according to the given context. For this experiment the disambiguation process is simpli ed: the most popular entity among the available candidates is chosen as correct disambiguation. To determine the popularity of the entity candidates three di erent measures are applied: { Incoming Page Links of entity candidates { Anchor-Link-Probability within web document corpus { Anchor-Link-Probability within Wikipedia corpus The rst measure is a simple entity-based popularity measure. The popularity is de ned according to the number of incoming Wikipedia page links. The more links point to an entity the more popular the entity is considered. The Anchor-Link-Probability de nes the probability of a linked entity for a given anchor text. Thus, the more often a mention is used to link to the same entity the higher is the Anchor-Link-Probability. This probability has been calculated on two di erent corpora. For the DBL dictionary this probability has been calculated based on the Wikipedia article corpus and for GCW dataset it has been calculated based on all web documents (cf. Section 4.1). The results of this experiment can be considered as an indicator for the degree of di culty of the applied benchmark in terms of WSD. A high recall and precision by simply using a popularity measure indicates a less di cult benchmark dataset. If a benchmark contains less popular entities the disambiguation process can be considered more di cult. 4.3
The experiments described above are discussed in the following paragraphs. For every experiment a table with the achieved results is given. The tables show the results for the four di erent dictionaries { represented by the columns { on the three di erent benchmarks { represented by the rows. For comparison issues, for all dictionaries the number of entries and for all benchmarks the number of distinct mentions and their annotated entities is given. For all results the total numbers as well as proportional respectively an averaged value is given. This facilitates the comparison of benchmarks and dictionaries that are signi cantly di ering in number of annotations and size.
The experiments mapping coverage, entity candidate count, maximum recall, and recall and precision based on page link popularity have been also performed using caseinsensitive mentions and labels in the four di erent dictionaries. For comparison, these results are presented in the same tables of the respective experiments as the results of the case-sensitive experiments. Recall and precision based on Anchor-Link-Probability have not been calculated as the probabilities for case-insensitive anchors are not available for the DBL and GCW datasets.
Mapping Coverage { GCW achieves highest coverage (between 94.67% and 100%) due to largest dictionary containing 378 m. entries and its construction method: anchor texts and linked Wikipedia articles in web documents. { RDM performs worst with only 25.19% on the Spotlight benchmark due to the lack of preprocessing { all labels are given with capital rst letters which is not common in English language except for persons, places, organizations. { Coverage for RDM increased by 69% (to 94%) when mentions in Spotlight benchmark are looked up in dictionary case-insensitive. Also, for the Wikilinks benchmark the coverage using the RDM dictionary is increased by 16% to 76%. The RDM dictionary consists of mainly case-sensitive labels (as no pre-processing has been performed). Persons, organizations, and places are written with a rst capital letter in English language texts. Mentions of entities of those types are found in a case-sensitive dictionary, such as RDM. In contrast, mentions of entities that are not of type person, organization or place, as e. g. internet are not found in the dictionary. If a benchmark contains mainly mentions of entities of type person, organization, or place the RDM dictionary achieves a high mapping coverage { as for the KORE 50 benchmark. Case-insensitive selection must increase the coverage, especially if the benchmark contains entity mentions that are not of type person, organization or place. This assumption is consolidated by the increased mapping coverage for the Spotlight and Wikilinks benchmark and the type information of the mentioned entities in the benchmarks presented in Table 1. { Overall, the dictionaries perform very well or even best on the benchmarks that have been constructed for the evaluation of their respective applications: DBL { Spotlight, AIDA { KORE 50, and GCW { Wikilinks.
{ DBL and RDM do not contain all rst names of persons as needed for benchmark
KORE 50. Thus, the maximum recall decreases compared to mapping coverage. { AIDA performs poorly on Spotlight benchmark due to the structure of dictionary.
The dictionary contains a large number of persons' rst names. Apparently, the dictionary does not re ect labels for entities in manually annotated texts. Recall and Precision achieved by Popularity { Incoming Wikipedia Page Links of Entity Candidates { Notably GCW performs poorly on all benchmarks compared to maximum achievable recall due to a high entity candidate count. Apparently entity candidate lists often contain more popular but incorrect entities. { In the KORE 50 benchmark, due to many annotated rst names, entity candidate lists contain many prospective entities and apparently the correct candidate is often not the most popular one compared to the other candidates. This explains the poor performance of all dictionaries on the KORE 50 using page link popularity. { Compared to the maximum achievable recall (of all dictionaries) on the KORE 50 the achieved recall is very low using a popularity measure as simpli ed disambiguation process. This con rms the intention of the benchmark to contain mentions that are hard to disambiguate.
Mention
Count 265 130 Evaluation results of NED approaches are dependent on the structure of the used benchmark dataset as well as on the dictionary used for entity candidate determination. The objective of this paper is to point out the di erences of several benchmarks and dictionaries for NED. For this purpose three di erent benchmarks have been analyzed. Two of them rst have been converted into NIF representations and made available online. The analyses included simple statistical information as well as type information of contained entities about the benchmarks. Additionally, four di erent dictionaries have been applied to determine entity candidates in the benchmarks. Based on our evaluation, important assumptions about the benchmarks have been consolidated and new insights into the characteristics of evaluated benchmarks as well as on the expressiveness of the dictionaries have been delivered. By making all benchmarks and evaluation results available online, evaluation of new NER or NED tools can be achieved more target-oriented with more meaningful results.