English. This work presents the solution adopted by the sisinflab team to solve the task NEEL-IT (Named Entity rEcognition and Linking in Italian Tweets) at the Evalita 2016 challenge. The task consists in the annotation of each named entity mention in a Twitter message written in Italian, among characters, events, people, locations, organizations, products and things and the eventual linking when a corresponding entity is found in a knowledge base (e.g. DBpedia). We faced the challenge through an approach that combines unsupervised methods, such as DBpedia Spotlight and word embeddings, and supervised techniques such as a CRF classifier and a Deep learning classifier.
Italiano. Questo lavoro presenta la soluzione del team sisinflab al task NEELIT (Named Entity rEcognition and Linking in Italian Tweets) di Evalita 2016. Il task richiede il riconoscimento e l’annotazione del testo di un messaggio di Twitter in Italiano con entita` nominate quali personaggi, eventi, persone, luoghi, organizzazioni, prodotti e cose e eventualmente l’associazione di queste entita` con la corrispondente risorsa in una base di conoscenza quale, DBpedia. L’approccio proposto combina metodi non supervisionati quali DBpedia Spotlight e i word embeddings, e tecniche supervisionate basate su due classificatori di tipo CRF e Deep learning.
In the interconnected world we live in, the information encoded in Twitter streams represents a valuable source of knowledge to understand events, trends, sentiments as well as userbehaviors. While processing these small text messages a key role is played by the entities which are named within the Tweet. Indeed, whenever we have a clear understanding of the entities involved in a context, a further step can be done by semantically enriching them via side information available, e.g., in the Web. To this aim, pure NER techniques show their limits as they are able to identify the category an entity belongs to but they cannot be used to find further information that can be used to enrich the description of the identified entity and then of the overall Tweet. This is the point where Entity Linking starts to play its role. Dealing with Tweets, as we have very short messages and texts with little context, the challenge of Named Entity Linking is even more tricky as there is a lot of noise and very often text is semantically ambiguous. A number of popular challenges on the matter currently exists, as those included in the SemEval series on the evaluations of computational semantic analysis systems1 for English, the CLEF initiative2 that provides a crosslanguage evaluation forum or Evalita3 that aims to promote the development of language and speech technologies for the Italian language.
Several state of the art solutions have been
proposed for entity extraction and linking to a
knowledge base
In this paper we present the solution we
propose for the NEEL-IT task
The remainder of the paper is structured as follows: in Section 2 we introduce our strategy that combines DBpedia Spotlight-based and a machine learning-based solutions, detailed respectively in Section 2.1 and Section 2.2. Section 3 reports and discusses the challenge results. 2
The system proposed for entity boundary and type extraction and linking is an ensemble of two strategies: a DBpedia Spotligth5-based solution and a machine learning-based solution, that exploits Stanford CRF6 and DeepNL7 classifiers. Before 4https://github.com/wikilinks/neleval/wiki/Evaluation 5urlhttps://github.com/dbpedia-spotlight/dbpediaspotlight 6http://nlp.stanford.edu/software/CRF-NER.shtml 7https://github.com/attardi/deepnl applying both approaches we pre-processed the tweets used in the experiments, by doing: (1) data cleaning consisting of replacing URLs with the keyword URL as well emoticons with EMO; This has been implemented with ad hoc rules; (2) sentence splitter and tokenizer, implemented by the well known linguistic pipeline available for the Italian language: “openNLP”8, with its corresponding binary models9. 2.1
DBpedia Spotlight is a well known tool for entity linking. It allows a user to automatically annotate mentions of DBpedia resources in unstructured textual documents.
Spotting: recognizes in a sentence the phrases that may indicate a mention of a DBpedia resource.
Candidate selection: maps the spotted phrase to resources that are candidate disambiguations for that phrase.
Disambiguation: uses the context around the spotted phrase to decide for the best choice amongst the candidates.
In our approach we applied DBpedia Spotlight
As summarized in Figure 2, we propose an
ensemble approach that combines unsupervised and
supervised techniques by exploiting a large dataset
of unannotated tweets, Twita
The module NERs Enabler & Merger aims to enabling the usage of one or both classifiers. When them both are enabled there can be a mention overlap in the achieved results. In order to avoid overlaps we exploited regular expressions. In particular, we merged two or more mentions when they are consecutive, and we choose the largest span mention when there is a containment. While with Spotlight we are allowed to find linked entities only, with this approach we can detect both entities that matches well known DBpedia resources and those that have not been identified by Spotlight (NIL). In this case given an entity spot, for entity linking we exploited DBpedia Lookup and string matching between mention spot and the labels associated to DBpedia entities. In this way we were able to find both entities along with their URIs, plus several more NIL entities. At this point, for each retrieved entity we have the span, the type (multiple types if CRF and DeepNL disagree) and the URI (see Figure 2) so we use a type detection/validation module for assigning the correct type to an entity. This module uses ad hoc
Entity P R F1 TP FP FN LOC 0.6154 0.4000 0.4848 16 10 24 ORG 0.5238 0.2000 0.2895 11 10 44 PER 0.4935 0.4810 0.4872 38 39 41 PRO 0.2857 0.0488 0.0833 2 5 39 Totals 0.5115 0.2839 0.3651 67 64 169
Table 2: CRF NER over the validation set when mentions are correctly detected, types are wrongly identified. This is due of course to language ambiguity in a sentence. As an example, for a NER it is often hard to disambiguate between a person and an organization, or an event and a products are not. For this reason we applied a further type detection and validation module which allowed to combine, by ad hoc rules, the results obtained by the classifiers and the Spotlight-based approach previously described.
DeepNL NER. DeepNL is a Python library for Natural Language Processing tasks based on a Deep Learning neural network architecture. The Entity EVE LOC ORG PER PRO Totals
P
0
0.5385
0.4074
0.6458
0.4375
0.5333
responding link in DBpedia. A valid approach to
link the names in our datasets to entities in
DBpedia is represented by DBpedia Lookup11
As for the last step we used the Type Detection module introduced above, to select entities belonging only to those classes representative of the interest domain. We implemented other filters to reduce the number of false positives in the final mapping. As an example, we discard the results for the case of Person entity, unless the mention exactly matches the entity name. As a plus, for linking, we also used a dictionary made from the training set, where for a given surface form and a type it returns a correspondent URI, if already available in the labeled data.
sults obtained through Spotlight and Lookup or string match, refer to the Italian version of DBpedia. In order to canonicalized version as required by the task, we automatically found the corresponding canonicalized resource link for each Italian resource by means of the owl:sameAs property.
11https://github.com/dbpedia/lookup 12https://en.wikipedia.org/wiki/ Wikipedia:Redirect
As an example the triple dbpedia: Multiple_endocrine_neoplasia> owl:sameAs <http://it.dbpedia. org/resource/Neoplasia_endocrina_ multipla> maps the Italian version of Neoplasia endocrina multipla to its canonicalized version. In a few cases we were not able to perform the match. 3
In this section we report the results over the gold test set distibuted to the challenge participants, considering first 300 tweets only.
In order to evaluate the task results, the 2016 NEEL-it challenge uses the TAC KBP scorer13. TAC KBP scorer evaluates the results according to the following metrics: mention ceaf, strong typed mention match and strong linked match.
The overall score is a weighted average score computed as: score = 0.4 mention ceaf + 0.3 strong link match + +0.3 strong typed mention match
Our solution combines approaches presented in Section 2.1 and Section 2.2. For the 3 runs submitted for the challenge, we used the following configurations: run1 Spotlight with results coming from both CRF and DeepNL classifiers; run2 without CRF; run3 without DeepNL.
As for CRF and DeepNL classifiers, we used a model trained with the whole training set provided by the challenge organizers. In order to ensemble the systems output we applied again the NERs Enabler & Merger module, presented in Section 2.2 that aims to return the largest number of entity annotations identified by the different systems without overlap. If one mention has been identified with more then one approach, and they disagree about the type, that returned by the Spotlight approach is chosen. Results for the different runs are shown in Table 4 together with the results of 13https://github.com/wikilinks/neleval/wiki/Evaluatio System Spotlight-based run1 run2 run3 Best Team mention ceaf that has been addressed using simple heuristics. der to evaluate the contribution of the Spotlightbased approach to the final result, we evaluated the strong link match considering only the portion of link-annotation due to this approach over the challenge test set, see Table 5. We had a total of 140 links to Italian DBpedia, then following the approach described in Section 2.3 we obtained 120 links, 88 of which were unique. It was not possible to convert into DBpedia canonicalized version 20 links. Final results are summarized in Table 5. Looking at the Spotlight-based solution (row 1),
System
Spotlight-based
run1
However, in this task, we find that mention
detection often represents a bottleneck. In
mention ceaf detection, our submission results show
that CRF NER worked slightly better then Deep
NER, as already showed in the experiments over
the validation set in Section 2.2. Anyway
according to experiments in