The tasks of entity extraction, recognition, and linking are largely affected by the nature of the textual documents being analyzed. In fact, a lot of research efforts have focused on improving each task for both formal text (such as newswire documents) and for informal text (such as tweets). In this work, we propose a so-called hybrid approach that aims to be agnostic of the document type. Two datasets, namely the #Micropost2014 NEEL corpus and the OKE2015 test dataset, are used to benchmark the performance of our approach. The experimental results show that the approach presented in this paper outperforms the state-of-the-art systems on OKE2015 dataset and provides good results for the #Micropost2014 dataset.
The Web of documents has significantly grown in the last decades, resulting in an ever increasing amount of unstructured data being published, such as newswire documents, encyclopedic content or scientific papers. On the other hand, with the advent of social media services, such as Twitter and Facebook, new unstructured data has contributed to make the Web larger and wider, taping a larger human audience and embracing a different style of writing. The content being shared on social media is generally short in length, dynamic in terms of topics and events being covered, and informal in the writing style (less curated syntax and grammar). In this paper, we refer to respectively formal and informal text when talking about those two categories of textual documents.
For years, the task of entity recognition, along with word sense disambiguation and entity linking, has been tested on well-formed text (e.g newswire). The advent of microposts has introduced a breakthrough: approaches being used in the past did not fit this new type of textual data, and tailored approaches have been proposed to cope with text characterized by: i) less than 140 characters (Twitter), ii) strong ambiguity, iii) spelling errors, iv) non standard and lexical items, v) non standard syntactic patterns.
The aim of this work is to propose a hybrid approach for performing the tasks of
entity extraction, recognition and entity linking1 that would perform fairly robustly on
1 We consider an entity, a DBpedia resource that corresponds to a Wikipedia article. We consider
a mention, one of the possible way to name an entity.
both formal and informal textual documents. The approach is hybrid since it makes use
of both linguistic and semantic algorithms working together and enclosed in a pipeline
of stages that can be turn on and off. In the experimental settings, we use DBpedia2014
as knowledge base to link the entities being extracted from text, and two standard
benchmark datasets: the #Microposts2014 NEEL [
The paper is organized as follows: Section 2 reviews the strengths and weaknesses of state-of-the-art approaches in processing formal and informal text. Section 3 describes our approach and its technical components. Section 4 reports the results of our evaluation. Section 5 highlights some frequent errors while Section 6 proposes some future work. 2
Numerous approaches have been proposed to tackle the task of extracting, typing and linking entities in formal texts and microposts. We have divided the state-of-the art in two parts: approaches that deal with formal texts and approaches dealing with microposts, generally tweets. 2.1
Amongst the recent and best performing systems, WAT [
DBpedia Spotlight [
AIDA [
Babelfy [
Those four methods (and particularly WAT and Babelfy) perform well on formal text, but rather poorly on informal text such as tweets. WAT is limited as they match any entry in the dictionary, including terms that are common words such as verbs or prepositions. For typing, WAT is limited to the kind of mentions that can be typed by OpenNLP NER. Similarly, AIDA depends on Stanford NER and the specific model used by the CRF algorithm. Those four systems are also limited to the fixed knowledge base being used, thus only extracting and linking entities that are referenced which prevents to recognize emerging entities.
E2E [
DataTXT [
AIDA [
These three methods are tailored to process tweets. Like for the four methods described in the Section 2.1, they do not recognize emerging entities as they are mostly based on dictionaries or trained model for formal text (e.g. AIDA). 2.3
We summarize this state-of-the art in the Tables 1 and 2, inspired from [
System E2E Babelfy Spotlight AIDA StanfordNER TagME and DataTXT WAT
NEE (Named Entity Extraction) External Tools Main Features Method Knowledge Base - N-Grams, stop words re- rule-based (candidate Wikipedia, moval, punctuation as token filter), dictionary Freebase
- NER dictionary N-Grams, overlap resolu- dictionary, link proba- Wikipedia tion, Wikipedia statistics bility OpenNLP N-Grams, Wikipedia statis- dictionary, SVM Wikipedia, tics NER dictionary - N-Grams, POS, superstring dictionary Babelnet
matching LingPipePOS string matching, POS
For the extraction, we observe that systems mainly use dictionaries based on a particular knowledge base (semantic-based approach). When POS tagging is being used, it is essentially a secondary feature which aims to enforce or to discard what has been extracted with the dictionary. Contrarily to the others, AIDA uses a pure NLP approach based on Stanford NER. TagME claims to make an overlap resolution between the extracted mentions at the end of this process. Our system tackles the problem using both a linguistic-based and a semantic-based approach of equal importance which demonstrates a higher performance at extraction level as detailed in the Section 4.
For the linking, we can see two main approaches: graph-based and arithmetic
combination. Contrarily to the others, E2E [
The architecture of our system is made for being modular. It is composed of different modules that can be switched on or off according to the nature of the text to be processed. A pipeline is a sequence of activated modules that will process an input text. Modules can have different goals: extraction, typing, recognition, linking or pruning. Those modules, according to their goal, have to respect some standards. Each module has to provide an output corresponding to a specific interface in order to be understood by the others modules. For example, in our approach, we have three extraction modules (POS, NER and gazetteer) that should provide a similar output for being understood by the candidate generator module. This system eases the overlap resolution process as it is easier to compare outputs that follow the same presentation. Furthermore, each module is run independently from the others.
The candidate generation process is based on an index. If an extracted mention does not have an entry in the index, we normally link it to NIL following the TAC KBP convention4. The index is created on top of the DBpedia 2014 Knowledge Base5 and a dump of the Wikipedia articles6 dated from October 2014. Each record of the index has a key which corresponds to a DBpedia resource, while the features are listed in Table 3. To create this index, we build an index using Lucene v5.2.1. The process requires 44 hours to be completed on a 64GB RAM, 20 core CPU at 2.5Ghz machine. The feature number 13 in Table 3 is made with an in-house library to parse the Wikipedia dump. We first tried several libraries that parse Wikipedia such as Sweble7, GWTWiki8 and wikipedia-parser9. However, these libraries are either too complex to use for the simple extraction we need or too greedy in terms of memory. We have therefore developed our own library in order to extract the pairs (Wikipedia article title, number of times the title appears in the article). For example, in the Wikipedia article of Paris we found 3 4 http://nlp.cs.rpi.edu/kbp/2014/ 5 http://wiki.dbpedia.org/services-resources/datasets/ datasets2014 6 https://dumps.wikimedia.org/enwiki/ 7 http://sweble.org/ 8 https://code.google.com/p/gwtwiki/ 9 https://github.com/Stratio/wikipedia-parser links to the Wikipedia article Eiffel Tower, resulting in the tuple (Eiffel Tower, 3), and this tuple will be associated to the corresponding entry of the DBpedia resource Paris in the index.
Our system uses three different modules to extract mentions: POS, NER and gazetteer. Our POS tagging system is the Stanford NLP POS Tagger. We use two different models depending if we process microposts (e.g. tweets) or formal texts (e.g. newswire). The model used for microposts is https://gate.ac.uk/wiki/twitter-postagger. html that is trained specifically in order to be case insensitive and to get independent tags for mentions and hashtags. The one used for formal text is english-bidirectionaldistsim that provides a better precision but for a higher computing time. We use the POS tagger to spot all the noun phrases and numbers. Our second module is the NER which relies on Stanford NER, trained with either the OKE2015 training set for formal text or trained with the #Micropost2014 training set for tweets. Moreover, we use NER to extract dates and other time related mentions. Our last module is the gazetteer that aims to reinforce the extraction stage bringing a robust spotting for well-known nouns such as abbreviations. Those three components are actually divided in five different modules: one POS module to extract only proper-nouns, one POS module to extract only numbers, one NER module without date spotting, one NER module with only date spotting and one gazetteer module. Finally, a sixth module has been developed to dereference mentions in microposts, for example, the mention @TheRealHowardW will be transformed into Howard Wolowitz.
Once all extractors have finished to extract mentions, we process their output with an overlap resolution module. This module takes two inputs and provides one output. It is run as many times as there are activated extractor modules minus one. For example, if a pipeline uses the extractor modules POS for proper nouns (POSNNP), the NER and gazetteer (GAZ) modules, then, the overlap resolution will first take the two inputs (POSNNP and NER) to provide one output (POSNNP-NER) and it will take once again two other inputs (POSNNP-NER and GAZ) to provide one final output (POSNNPNER-GAZ). Therefore, when three different extractor modules are used, the overlap resolution module is run twice. We have developed such a module since, sometimes, at least two extractors provide overlapping mentions. For example, given the two extracted mentions States of America from the NER module and United States from the POS for proper noun module, we detect that there is an overlap between both mentions according to their offset in the processed sentence. We take the union of both boundaries to create a new mention and we remove the two others. We obtain the mention United States of America and the type provided by the NER module is selected. The mechanism is the same if one mention is included in another one. For example, if United States and United States of America are extracted, the later will be kept. Finally, there are cases of ambiguity. For example, with the text Yesterday I went to Los Angeles, California, Los Angeles, California and Los Angeles, California are three different valid extractions. We have decided to systematically keep the longest mention, in this case Los Angeles, California.
Once we have independent mentions, the candidate generation module searches the
index to retrieve as many candidates as possible. This means that for each mention, we
have potentially numerous candidates, while many of them have to be filtered out
because they are most likely not related to the context. We use a filter module that creates
a graph with all the candidates of each mention and find the densest graph between all
of these candidates, similarly to [
Once we have reduced the number of candidates, we use a ranking module in order to score each of those candidates and to pick up the best one. This module is using a rank function r(l) to compute, for each candidate, a score based on a string similarity measure between the extracted mention and the title of the candidate, the set of redirect and the set of disambiguation pages associated to this candidate and its PageRank. r(l) = (a L(m; title) + b max(L(m; R)) + c max(L(m; D))) P R(l) (1) where the weights a, b and c have to follow the following hypothesis: a + b + c = 1 and a > b > c. We take the assumption that the string distance measure between a mention and a title is more important than the distance measure with a redirect page and is itself more important than the distance measure with a disambiguation page.
The last module is the pruning one, which is used to detect and remove the false positive annotations10 in order to improve the precision of the system. We use a machine learning approach, with the algorithm k-NN. We have tried four different algorithms (Random Forest, Naive Bayes, SVM and k-NN), and have empirically assessed that k-NN generally provides the best results. To train this algorithm and getting a model, we use ten features, most of them are listed in Table 3: the extracted mention and the title, type, PageRank, HITS, inLinks, outLinks, length, redirectsNumber and r(l) of the entity. The training method uses a four steps approach: 1) run our system on a training set; 2) classify entities as true or false according to the entities in the Gold Standard of the training set and the ones provided by the results of our system; 3) create a file with the features of each of these entities and their true / false classification; 4) train k-NN with this file that contains the features to get a model. Once the model has been created, we let k-NN classify each annotation provided by our system to true or false. 4
Our hybrid approach has been benchmarked against the test dataset of the #Micropost2014 NEEL challenge and the test dataset of the OKE2015 challenge. An example of #Micropost2014 and OKE2015 datasets are provided in Table 4. 10 We consider an annotation as the tuple (mention, entity) 4.1
We configured two different pipelines, one for each dataset. For OKE2015, we used the following modules: three extractors (POS for proper nouns, NER without date and gazetteer), the index lookup, the filtering and the scoring. For #Micropost2014, we used six extractors (POS for proper nouns, POS for numbers, NER with only dates, NER without dates, gazetteer and Twitter account dereferencing), the index lookup, the filtering and the scoring. We have run these two pipelines with and without the pruning module in order to assess its performance. This shows the advantages of our hybrid approach of being able to adapt an entity linking process by adding or removing modules (methods) and combining them.
dataset text links #Microposts2014 Murdoch unable to answer who the db:Rupert Murdoch top legal officer at News International db:News UK was. (via: http://mm4a.org/q3rx06) #p2 db:News of the World #notw #hackgate db:News International phone hacking scandal OKE2015 Breakdown figures for the #Micropost2014 NEEL challenge with and without the pruning reported in Table 5 have been computed with the official scorer of the challenge11. In the breakdown figures, the results at the recognition level is not presented since typing entities was not required by the challenge. Breakdown figures for the OKE2015 challenge with and without the pruning reported in Table 6 have been computed with the neleval scorer12.
The Table 7 shows the performance of our approach in comparison to other systems,
using the F1-measure at the final linking stage. Results for #Micropost2014 for TagME
(more precisely DataTXT), AIDA, E2E and UTwente are coming from the official
results of the challenge, while Babelfy and DBpedia Spotlight have not been tested as they
are not made for processing tweets. We refer the reader to [
Without Pruning With Pruning
Task P R F1 P R F1 Extraction 78.2 65.4 71.2 83.8 9.3 16.8 Recognition 65.8 54.8 59.8 75.7 8.4 15.1
Linking 49.4 46.6 48 57.9 6.2 11.1 Table 6. Breakdown figures on the OKE challenge training set with and without the pruning. all systems having participated in the 2014 NEEL challenge while we only reproduce in Table 7 the best performing systems. For the OKE2015 challenge, the results have been computed with the neleval scorer.
The results that are reported for DBpedia Spotlight, TagME, AIDA and Babelfy for OKE2015 have been obtained using their respective APIs with the best tested settings. For DBpedia Spotlight, those settings are: confidence=0.3 and support=20. For TagME, those settings are: include all spots=yes and epsilon=0.5. For AIDA, those settings are: technique=GRAPH, algorithm=COCKTAIL PARTY SIZE CONSTRAINED, alpha=0.6 and coherence=0.9. For Babelfy, those settings are: lang=en, annType=NAMED ENTITIES, annRes=WIKI, match=EXACT MATCHING, dens=true and th=0.4. WAT is not publicly available and could not be tested with the OKE challenge dataset.
Our approach outperforms those systems when analyzing formal texts. E2E and UTwente still perform significantly better than our approach when processing microposts while we achieve similar results than TagME (DataTXT). 5
For #Micropost2014, an error analysis shows that our approach misses 490 entities and finds 578 additional entities on a total of 1460 entities contained in the test dataset. Amongst the most common errors, our approach has problems with entities which do not contain tokens tagged as proper nouns but as noun (e.g. phone hacking). There is also a problem with the possessive endings (e.g Beyonce’s for Beyonce). Syntactically speaking, a noun phrase may contain a noun phrase, which provides duplicates. For example, in the sentence She gets into the action, overlapping Lloyd, the noun phrase the action, overlapping Llyod can be split in two noun phrases, which are the action and overlapping Lloyd. The extraction may lead to these three cases.
For OKE2015, an error analysis shows that our approach misses 207 entities and find 167 additional entities on a total of 594 entities contained in the test dataset. The most common error for this challenge is that our approach does not resolve the coreferences while the test dataset contained a lot of them. The training set was not big enough to properly train Stanford NER. This is why the correct mention is often extracted but is given a wrong type. The test dataset contained errors in the gold standard, some of them have been fixed by us, but some others need a closer work with the organizers to be resolved.
We observe that there is still a large margin of progress to reduce the performance
drop between the results at the recognition or extraction stage and the final results at
the linking stage. This drop can be explained by the score function and the candidates
filter used at the linking stage. For the pruning, we can see that it increases a bit the
precision but at the cost of significantly decreasing the recall. It overall performs poorly
as it removes too many (correct) mentions to get good results at the linking stage. This
idea has been inspired by the WAT system [
The results are encouraging and show that this hybrid approach which exploits both linguistic and semantic features can achieved the expected behavior. As future work, we plan to improve the linking stage by making more use of graph-based algorithms with more accurate ranking functions and filtering. Another future work is to further develop our pruning strategy by reviewing the list of feature used to show the full potential of our hybrid approach. We finally aim to optimize the creation of the index by parallelizing the process and to multiply the number of knowledge bases on which entities can be disambiguated against.
This work was partially supported by the innovation activity 3cixty (14523) of EIT Digital (https://www.eitdigital.eu).