This paper summarizes the participation of UNIMIB team in the Named Entity rEcognition and Linking (NEEL) Challenge in #Microposts2016. In this paper, we propose a knowledge-base approach for identifying and linking named entities from tweets. The named entities are, further, classified using evidence provided by our entity linking algorithm and type-casted into Microposts categories.
Microblogging platforms such as Twitter have become a rich source of real-time information. Today, information is being readily extracted from such platforms, in the form of named entities, relations and events. The tasks of this challenge comprise identification and classification of named entities from a set of tweets, and linking the identified entities to corresponding KB resources if a match is found, or to a NIL reference if no candidate resources can be retrieved [5].
In order to identify named entities, we use a pre-trained, state-of-the-art Named Entity Recognition (NER) system [4]. Using this system, we tokenize and segment the tweets to identify entities and non-entities. Further, our linking algorithm is based on a greedy approach which disambiguates and links all the identified entities with DBpedia resources. Finally, the entities are classified using evidence from the linking phase. 2.1
Copyright c 2016 held by author(s)/owner(s); copying permitted only for private and academic purposes.
Published as part of the #Microposts2016 Workshop proceedings, available online as CEUR Vol-1691 (http://ceur-ws.org/Vol-1691)
For the task of identifying named entities, we use a stateof-the-art NER system, T-NER [4] which is a supervised model based on Conditional Random Fields (CRF), pretrained on a state-of-the-art gold standard of tweets [4]. The CRF model of T-NER has been used to identify, given a tweet t as input, the candidate entities e1, e2, ..., en in t. In other words, the CRF model segments a tweet into entities and non-entities.
For performing entity recognition using T-NER, we remove the special characters (@, #,..) as a pre-processing step and process the tweets in UTF-8 format in order to deal with emoticons. T-NER is not trained to recognize @usernames as entities and the current version of our system does not resolve username references. This has a significant impact on the overall performance of our system. 2.2
For the task of selecting a candidate resource for an entity, we use DBpedia1 as our KB. We perform a pre-processing step here, wherein, for every identified entity which consists of a segment that begins with a capital letter, we segment that entity into a set of tokens based on the capital letter. For instance, the entity mention ‘StarWars’ is treated as ‘Star Wars’ during the candidate retrieval phase so as to obtain better candidate matches. To this end, we extract all the Titles of all Wikipedia articles2 from DBpedia using rdfs:label and index them using LuceneAPI3. For each identified entity, top-k candidate KB resources are retrieved using a high-recall approach. Here we set k = 500. We estimate a knowledge-base score, called KB(ck), for each candidate resource ck of an entity ej as follows:
KB(ej, ck) = (α · lex(ej, lck ) + (1 − α) · (cosk(ej∗, ack ))) + R(ck) (
More formally, lex(ej, lck ) is defined as follows: lex(ej , lck ) = lcs(ej , lck ) + WD JW (ej,lck )
WD+1
(
JW (ej, lck ) = Jaro(ej, lck ) + 10 · (1 − Jaro(ej, lck )) (
JW (ej, lck ) = max(JW (ej, P1lck ), ..., JW (ej, Pnlck )) (
To evaluate the second component cosk(ej∗, ack ) of the KB
score in equation (
Finally, the last contribution provided in equation (
R(ck) = β · P R(ck)
(
In order to determine the optimal configuration of our
system, the parameters have been experimentally evaluated.
The top-k candidates are ranked using equation (
We followed an unsupervised, greedy approach to link an
entity with a DBpedia resource. In this way, we link
every identified entity with a corresponding candidate resource
with the highest candidate score achieved using equation
(
Moreover, we established a mapping between the DBpedia Ontology and Microposts categories (Thing, Person, Location, Organization, Event, Character and Product ) by following the description of the Microposts categories [5] by the challenge organizers. Every DBpedia Ontology class that can not be mapped intuitively following this description, such as the Ontology class Species, has been mapped to the Microposts category Thing. We adopted only one exception to this rule, where we mapped the DBpedia Ontology class Name, with its subclasses, GivenName, Surname to the Microposts category Person. GivenNames and Surnames are used in tweets mostly to refer to a person in the real world, i.e., they are mentions of entities that would be re-classified under the Microposts category Person. This interpretation of mapping for names and surnames is inspired by previous work on mapping semantics [1]. 2.4
We performed an additional post-processing step, where an identified entity’s boundary is re-scoped based on the label of the resource linked to the entity in the previous phase. We apply this step when the resource label is a substring of 5http://mappings.dbpedia.org/server/ontology/classes/ 6http://dbpedia.org/Downloads2015-04
STMM 0.297 0.300 0.139 0.134
Mention Ceaf 0.380 0.378 0.237 0.250 the entity mention. In this way, we are able to filter out noisy tokens in entities that were identified in the first step by the entity recognition system. For instance, in the tweet, Day 9: Wearing a StarWars T-Shirt each day until ‘The Force Awakens’. We’re half way there! https://t.co/QoAOxoSCJk the entity recognition system identifies ‘StarWars T-Shirt’ as an entity, due to a capitalization issue, however, our linking algorithm is able to link this entity correctly with the KB resource Star Wars, based on contextual and KB evidence. As a result, we re-scope the boundary of the identified entity ‘StarWars T-Shirt’ to ‘StarWars’ to improve the identification performance of the system. We evaluate our system using two configurations, viz. without entity boundary rescoping and with entity boundary re-scoping, as reported in Section 3 below.
We use the training and dev datasets to test the performance of the pre-trained NER system (supervised approach) and, use the identified entities for testing the performance of our linking algorithm (unsupervised approach). The training and dev gold standards consist of ≈6000 and 100 tweets, annotated with a total of 8665 and 338 entities, respectively.
Table 1 shows the performance of our entity linking and classification approach for Strong Link Match (SLM), Strong Typed Mention Match (STMM) and Mention Ceaf. As evident, the performance of the linking approach (SLM) improves when entity boundary re-scoping is applied, for both the datasets. An overall low performance of the entity linking system could be attributed to poor performance of the entity recognition system, as illustrated in Table 2. On the other hand, the performance for type classification approach (STMM) improves for the training dataset with entity rescoping, however, the improvement is not significant.
As shown in table 2, significant precision values are obtained on both the datasets, however, recall as well as F1 scores on the dev dataset are poor. A possible reason could be attributed to the presence of a lot of #hashtags and @usernames recognized as entities in the ground truth, which leads to a poor performance of the entity recognition system, even if @ and # are removed. An important observation is that by applying entity boundary re-scoping, precision and recall fall for the training dataset, however, its the opposite for the dev dataset. This can again be attributed to the presence of lot of #hashtags and @usernames in the dev dataset, due to which the entity recognition system exhibits entity segmentation errors.
Finally, table 3 summarizes the performance of our entity linking algorithm in terms of precision, recall and F1 scores assuming a NER Oracle. To this end, we use a modified version of the Training and Dev gold standards, denoted as Training* and Dev* which comprise of linkable entities only, i.e., void of NIL mentions. They are annotated with 6371 and 253 linkable entities, respectively. Our linking approach is able to link correctly ≈ 50% of the entities in the modified ground truth. When a NER Oracle is used, the performance of the system obviously falls for entity boundary re-scoping. Hence, we report the results without entity boundary rescoping for the Training* and Dev* datasets. For the test set evaluation, we provide 2 runs of our system on the test dataset for both configurations.
In previous work we defined a more sophisticated entity classification method, which combines evidence from the LabeledLDA component of T-NER and from the types of candidate entities [3]. In this challenge we could not apply this method due to problems in integrating the LabeledLDA component in our current pipeline, but we plan to use this method again in the near future. 4.