English. In this paper we present the FBK-NLP system which participated to the NEEL-IT task at Evalita 2016. We concentrated our work on domain adaptation of an existed Named Entity Recognition tool. Particularly, we created a new annotated corpus for the NEEL-IT task using an Active Learning method. Our system obtained the best results for the task of Named Entity Recognition, with an F1 of 0.516.
This paper describes the FBK-NLP system which
participated to the NEEL-IT task at EVALITA
2016
The system follows 3 steps: entity recognition
and classification, entity linking to DBpedia and
clustering. Entity recognition and classification is
performed by the EntityPro module
The paper is organized as follows. In Section 2
we present the domain adaptation of the Named
Entity Recognition tool using Active Learning.
Then in Section 3 we describe the system with
which we participated to the task and in Section
4 the results we obtained as well as some further
experiments. Finally we conclude the paper with
a discussion in Section 5.
We have at our disposal a system for Named
Entity Recognition and Classification, a module of
the TextPro pipeline
As we were interested in applying Active Learning (AL) methods to the production of training data, we decided to annotate manually our own set of domain specific training data using AL method.2 Active Learning is used in order to select the most informative examples to be annotated, instead of selecting random examples.
We exploited TextPro-AL
The architecture of the TextPro-AL platform is represented in Figure 1. The AL cycle starts with an annotator providing supervision on a tweet automatically tagged by the system (step 1): the annotator is asked to revise the annotation in case the system made a wrong classification. At step 2a the annotated tweet is stored in a batch, where it is accumulated with other tweets for re-training, and, as a result, a new model (step 3) is produced. This model is then used to automatically annotate a set of unlabeled tweets (step 4) and to assign a confidence score3 to each annotated tweet. At step 2b the manually annotated tweet is stored in the 2The annotated data made available by the organizers of the task were used partly as test data and partly as a reference for the annotators (see Section 2.2).
3The confidence score is computed as the average of the margin estimated by the SVM classifier for each entity. Global Memory of the system with the information about the manual revision. At step 5 a single tweet is selected from the unlabeled dataset through a specific selection strategy (see Algorithm 1). The selected tweet is removed from the unlabeled set and is given for revision to the annotator.
The Global Memory contains the revision done by the annotator for each tweet. In particular we are interested in the entities wrongly annotated by the system, which are used to select new tweets to be annotated. Each entity (or error) saved in the memory is used up to 6 times in order to select new tweets. From the unlabeled dataset, the system selects the most informative instance (i.e. with the lowest confidence score) that contains one of the errors saved in the Global Memory (GM). The selection strategy is detailed in Algorithm 1. In a first step the system annotates the tweets of the unlabeled dataset. Then the tweets are sorted from the most informative to the less informative and browsed. The first tweet in the list that contains an error saved in the GM is selected to be revised by the annotator. If no tweets are selected through this process, the system picks one tweet randomly.
Algorithm 1: Algorithm of the selection strategy
Data: NESet = fNE1 ... NEng begin
NESortedList
getMostInformativeInstances(NESet); repeat instance, sample
NESortedList.next(); if inMemory(instance) and revised(instance) then
return sample; until NESortedList.hasNext(); return getRandomSample(NESet); 2.2
As unlabeled database of tweets in the AL process
we used around 8,000 tweets taken from the
development set of Sentipolc 20144
4http://www.di.unito.it/˜tutreeb/ sentipolc-evalita14/tweet.html
5http://valeriobasile.github.io/twita/ about.html class # sent/tweets # tokens Person Location Organization Product Event Thing Character
AL tweets
The development data provided by the NEELIT organizers is composed by 1000 annotated tweets. We split it in two parts: 30% for development (used mainly as a reference for the annotators) and 70% for evaluation (referred to as test 70%).
We decided to retrain EntityPro using a smaller training set to be able to change the behavior of the model more quickly. In particular we used a sub-part of the training data used by EntityPro, i.e. 6.25% of the training set of the NER task at Evalita 2007,6 for a total of 8,304 tokens (referred to as news corpus in the remainder of the paper).
In order to determine the portion to be used, we tested the performance of EntityPro using as training data different portions of the corpus (50%, 25%, 12.5% and 6.25%) on test 70%. The best results were obtained using 6.25% of the corpus (statistics about this corpus is given in Table 1). 2.3
In our experimentation with TextPro-AL for domain adaptation we built the first model using the news corpus only. Evaluated on test 70%, it reached an F1 of 41.62 with a precision of 54.91 and a recall of 33.51. It has to be noted that with this model only 3 categories of entities can be recognized: person, location and organization. Then every time that 50 new tweets were annotated, the system was retrained and evaluated on the test 70% corpus. The learning curves of the system are presented in Figure 2. In total we were able to manually annotate 2,654 tweets for a total of 3,370 6http://www.evalita.it/2007/tasks/ner entities (we will refer to this corpus as AL tweets), which allowed us to obtain an F1 of 53.22 on test 70%. Statistics about the corpus are presented in Table 1. 3 3.1
The preprocessing of the tweets is done using the
TextPro tool suite7
In order to avoid some encoding problems we replaced all the emoji by their Emoji codes (e.g. :confused face:) using the python package emoji 0.3.9.8
The task of entity recognition and classification
is performed using an adapted version of the
EntityPro module
The classifier is used in a one-vs-rest multiclassification strategy. The format used for the 7http://textpro.fbk.eu/ 8http://pypi.python.org/pypi/emoji/ annotation is the classic IOB2 format. Each token is labeled either as B- followed by the entity class (person, location, organization, product, event, thing or character) for the first token of an entity, I- followed by the entity class for the tokens inside an entity or O if the token is not part of an entity. 3.2
Entity Linking is performed using the Named Entity Disambiguation (NED) module developed within the NewsReader Project9 supplemented with the use of a resource for Twitter profiles linking. The NED module is a wrapper around DBpedia spotlight developed within NewsReader and part of the ixa-pipeline.10 Each entity recognized by the NER module is sent to DBpedia Spotlight which returns the most probable URI if the entity exists in DBpedia.
The tweets often contain aliases, i.e. user profile
names, which enable the author of the tweet to
refer to other Twitter users. For example
@edoardofasoli and @senatoremonti in the following tweet:
@edoardofasoli @senatoremonti Tutti e due. In
order to identify the DBpedia links of the aliases
in the tweets we used the Alignments dataset
A procedure is built to query Twitter to get the Twitter profile id from the alias of a user, then query the Alignments dataset to get the corresponding DBpedia link if it exists. 3.3
The clustering task aims at gathering the entities referring to the same instance and at assigning to them an identifier, either a DBpedia link or a corpus based identifier. We performed this task applying a basic string matched method, i.e. we consider that two entities are part of the same cluster if their strings are the same. 4
We submitted 3 runs to the NEEL-IT task; they differ from the data included in the training dataset of EntityPro:
Run 1: news corpus and AL tweets Run 2: news corpus, AL tweets and NEEL-IT devset
The official results are presented in the first part of Table 2. Our best performance is obtained with the run 3, with a final score of 0.49.
After the evaluation period, we have run further experiments, which are marked with an asterisk in Table 2. The run 4 is a version of run 3 in which we have removed the wrong links to the Italian DBpedia (URIs of type http://it. dbpedia.org/). For runs 5, 6 and 7, EntityPro is trained using the news corpus alone, the NEELIT devset, and both respectively.
In Table 3, we present the performances of our systems in terms of precision, recall and F1 for the subtask of named entity recognition and classification. We observed that using the NEEL-IT devset the precision of our system increased, instead using the news corpus the recall increased. run 1 run 2 run 3 precision recall 0.571 0.459 0.581 0.451 0.598 0.454 We have described our participation to the NEELIT task at Evalita 2016. Our work focused on the task of named entity recognition, for which we get the best results. We were interested in the topic of domain adaptation. The domain adaptation includes two aspects: the type of the documents and the named entity classes of interest. Using EntityPro, an existing NER tool, and the TextPro-AL platform, we created a training dataset for NER in tweets, for the 7 classes identified in the task.11 With this new resource our system obtained an F1 11We will soon make available the new training set from the website of the HLT-NLP group at FBK (http:// hlt-nlp.fbk.eu/). of 0.516 for named entity recognition.
Our work has been concentrated on the use of
Active Learning for the domain adaptation of a
NER system. On the other hand, the Micro-NEEL
team
One of the main difficulty in identifying named entities in tweets is the problem of the splitting of hashtags and aliases (e.g. the identification of Monti in @senatoremonti). We adapted the TextPro tokenizer to split in small units those sequences of characters, but it works only if the different words are capitalized or separated by some punctuation signs (e.g. or -). A more complex approach should be used, using a dictionary to improve the splitting.
Named entity categories covered in this task are seven: person, location, organization, product, event, thing and character. The first three categories are the classical ones and cover the highest number of named entities in several corpora. Table 1 gives us an evidence of the prominence of these three classes. With the AL method we used, we were able to annotate new tweets containing entities of the less represented classes, in particular for product, event and character. However the class thing is still not well represented in our corpus and the classes unbalanced. In the future we plan to add in the TextPro-AL platform the possibility for the annotators to monitor the Global Memory used in the AL process in order to give precedence to examples containing entities of not well represented classes.
This work has been partially supported by the EUCLIP (EUregio Cross LInguistic Project) project, under a collaboration between FBK and Euregio.12