English. The paper describes our submissions to the task on Named Entity rEcognition and Linking in Italian Tweets (NEEL-IT) at Evalita 2016. Our approach relies on a technique of Named Entity tagging that exploits both character-level and word-level embeddings. Character-based embeddings allow learning the idiosyncrasies of the language used in tweets. Using a full-blown Named Entity tagger allows recognizing a wider range of entities than those well known by their presence in a Knowledge Base or gazetteer. Our submissions achieved first, second and fourth top official scores.
Italiano. L’articolo descrive la nostra partecipazione al task di Named Entity rEcognition and Linking in Italian Tweets (NEEL-IT) a Evalita 2016. Il nostro approccio si basa sull’utilizzo di un Named Entity tagger che sfrutta embeddings sia character-level che word-level. I primi consentono di apprendere le idiosincrasie della scrittura nei tweet. L’uso di un tagger completo consente di riconoscere uno spettro più ampio di entità rispetto a quelle conosciute per la loro presenza in Knowledge Base o gazetteer. Le prove sottomesse hanno ottenuto il primo, secondo e quarto dei punteggi ufficiali. correspond to entities in the domain of interest; entity disambiguation is the task of linking an extracted mention to a specific instance of an entity in a knowledge base.
Most approaches to mention detection rely on
some sort of fuzzy matching between n-grams in
the source text and a list of known entities
Detecting the correct entity mention is however crucial: Ritter et al. (2011) for example report a 0.67 F1 score on named entity segmentation, but an 85% accuracy, once the correct entity mention is detected, just by a trivial disambiguation that maps to the most popular entity.
We explored an innovative approach to mention detection, which relies on a technique of Named Entity tagging that exploits both character-level and word-level embeddings. Characterlevel embeddings allow learning the idiosyncrasies of the language used in tweets. Using a fullblown Named Entity tagger allows recognizing a wider range of entities than those well known by their presence in a Knowledge Base or gazetteer.
Another advantage of the approach is that no pre-built resource is required in order to perform the task, minimal preprocessing is required on the input text and no manual feature extraction nor feature engineering is required.
We exploit embeddings also for disambiguation and entity linking, proposing the first approach that, to the best of our knowledge, uses only embeddings for both entity recognition and linking.
We report the results of our experiments with this approach on the task Evalita 2016 NEEL-IT. Our submissions achieved first, second and fourth top official scores.
The NEEL-IT task consists of annotating named entity mentions in tweets and disambiguating them by linking them to their corresponding entry in a knowledge base (DBpedia).
According to the task Annotation Guidelines
The Knowledge Base onto which to link entities is the Italian DBpedia 2015-10, however the concepts must be annotated with the canonicalized dataset of DBpedia 2015, which is an English one. Therefore, despite the tweets are in Italian, for unexplained reasons the links must refer to English entities. 3
The training set provided by the organizers consists of just 1629 tweets, which are insufficient for properly training a NER on the 7 given categories.
We thus decided to exploit also the training set of the Evalita 2016 PoSTWITA task, which consists of 6439 Italian tweets tokenized and gold annotated with PoS tags. This allowed us to concentrate on proper nouns and well defined entity boundaries in the manual annotation process of named entities.
We used the combination of these two sets to train a first version of the NER.
We then performed a sort of active learning step, applying the trained NER tagger to a set of over 10 thousands tweets and manually correcting 7100 of these by a team of two annotators.
These tweets were then added to the training set of the task and to the PoSTWITA annotated training set, obtaining our final training corpus of 13,945 tweets. 4
Our approach to Named Entity Extraction and Linking consists of the following steps:
Train word embeddings on a large corpus of Italian tweets
Train a bidirectional LSTM characterlevel Named Entity tagger, using the pretrained word embeddings Build a dictionary mapping titles of the Italian DBpedia to pairs consisting of the corresponding title in the English DBpedia 2011 release and its NEEL-IT category. This helps translating the Italian titles into the requested English titles. An example of the entries in this dictionary are: Cristoforo_Colombo (http://dbpedia.org/resource/Christopher_Colu mbus, Person) Milano (http://dbpedia.org/resource/Milan, Location) From all the anchor texts from articles of the Italian Wikipedia, select those that link to a page that is present in the above dictionary. For example, this dictionary contains: Person Cristoforo_Colombo Colombo Create word embeddings from the Italian Wikipedia For each page whose title is present in the above dictionary, we extract its abstract and compute the average of the word embeddings of its tokens and store it into a table that associates it to the URL of the same dictionary Perform Named Entity tagging on the test set For each extracted entity, compute the average of the word embeddings for a context of words of size c before and after the entity.
Annotate the mention with the DBpedia entity whose lc2 distance is smallest among those of the abstracts computed before.
For the Twitter mentions, invoke the Twitter API to obtain the real name from the screen name, and set the category to Person if the real name is present in a gazetteer of names.
The last step is somewhat in contrast with the
task guidelines
We decided instead that it would have been more useful, for practical uses of the tool, to produce a more general tagger, capable of detecting mentions recognizable not only from syntactic features. Since this has affected our final score, we will present a comparison with results obtained by skipping this last step. 4.1
The word embeddings for tweets have been created using the fastText utility1 by Bojanowski et al. (2016) on a collection of 141 million Italian tweets retrieved over the period from May to September 2016 using the Twitter API. Selection of Italian tweets was achieved by using a query containing a list of the 200 most common Italian words.
The text of tweets was split into sentences and
tokenized using the sentence splitter and the
tweet tokenizer from the linguistic pipeline Tanl
We produced two versions of the embeddings, one with dimension 100 and a second of dimension 200. Both used a window of 5 and retained words with a minimum count of 100, for a total of 245 thousands words.
Word embeddings for the Italian Wikipedia were created from text extracted from the Wikipedia dump of August 2016, using the WikiExtractor utility by Attardi (2009). The vectors were produced by means of the word2vec utility2, using the skipgram model, a dimension of 100, a window of 5, and a minimum occurrence of 50, retaining a total of 214,000 words. 4.2
Bi-LSTM Character-level NER Lample et al. (2016) propose a Named Entity Recognizer that obtains state-of-the-art performance in NER on the 4 CoNLL 2003 datasets 1 https://github.com/facebookresearch/fastText.git 2 https://storage.googleapis.com/google-code-archivesource/v2/code.google.com/word2vec/source-archive.zip without resorting to any language-specific knowledge or resources such as gazetteers.
In order to take into account the fact that named entities often consist of multiple tokens, the algorithms exploits a bidirectional LSTM with a sequential conditional random layer above it.
Character-level features are learned while
training, instead of hand-engineering prefix and suffix
information about words. Learning character-level
embeddings has the advantage of learning
representations specific to the task and domain at hand.
They have been found useful for morphologically
rich languages and to handle the
out-ofvocabulary problem for tasks like POS tagging
and language modeling
The character-level embedding are given to bidirectional LSTMs and then concatenated with the embedding of the whole word to obtain the final word representation as described in Figure 1: The architecture of the NER tagger is described in Figure 2.
rMar lM M rMario rMalMa a eMario rMar lMar r CRF Layer Bi LSTM Encoder Word Embeddings Mario
B-PER
I-PER c1 r1 l1 c2 r2 l2 Monti rMa lMari i O c3 r3 l3 a lMario rM lMari o B-LOC c4 r4 l4 Roma
Since Named Entity tagging is the first step of our technique and hence its accuracy affects the overall results, we present separately the evaluation of the NER tagger.
Here are the results of the NER tagger on a development set of 1523 tweets, randomly extracted from the full training set.
Category
Precision
Recall
On the subset of the test set used in the evaluation, which consists of 301 tweets, the NER performs as follows:
In the disambiguation and linking process, we experimented with several values of the context size c of words around the mentions (4, 8 and 10) and eventually settled for a value of 8 in the submitted runs. 6
We submitted three runs. The three runs have in common the following parameters for training the NER:
Character embeddings dimension dropout Learning rate Training set size 25 0.5 0.001 12,188
Specific parameters of the individual runs are:
UniPI.1: twitter embeddings with dimension 100, disambiguation by frequency of mention in Wikipedia anchors UniPI.2: twitter embeddings with dimension 100, disambiguation with Wikipedia embeddings UniPI.3: twitter embeddings with dimension 200, disambiguation with Wikipedia embeddings, training set with geographical entities more properly annotated as Location (e.g. Italy).
The runs achieved the scores listed in the following table:
Run UniPI.3 UniPI.1 Team2.base UniPI.2 Team3.3
Mention ceaf 0.561 0.561 0.530 0.561 0.585
Strong typed mention match 0.474 0.466 0.472 0.463 0.516
Strong link match 0.456 0.443 0.477 0.443 0.348
Final score
The final score is computed as follows:
0.4 mention_ceaf + 0.3 strong_typed_mention_match +
0.3 strong_link_match As mentioned, our tagger performs an extra effort in trying to determine whether Twitter mentions represent indeed Person or Organization entities. In order to check how this influences our result we evaluate also a version of the UniPI.3 run without the extra step of mention type identification. The results are reported in the following table:
Run UniPI.3 without mention check
Mention ceaf On the other hand, if we manually correct the test set annotating the Twitter mentions that indeed refer to Twitter users or organizations, the score for strong typed mentions match increases to 0.645.
The effectiveness of the use of embeddings in disambiguation can be seen in the improvement in the strong link match score between run UniPI.2 and UniPI.3. Examples where embeddings lead to better disambiguation are:
Italy_national_football_team vs Italy S.S._Lazio vs Lazio Diego_Della_Valle vs Pietro_Della_Valle Nobel_Prize vs Alfred_Nobel There are many cases where the NER recognizes a Person, but the linker associates the name to a famous character, for example:
The approach of using embeddings for disambiguation looks promising: the abstract of articles sometimes does not provide appropriate evidence, since the style of Wikipedia involves providing typically meta-level information, such as the category of the concept. For example disambiguation for “Coppa Italia” leads to “Italian_Basketball_Cup” rather than to “Italian_Football_Cup”, since both are described as sport competitions. Selecting or collecting phrases that mention the concept, rather than define it, might lead to improved accuracy.
Using character-based embeddings and a large training corpus requires significant computational resources. We exploited a server equipped with nVidia Tesla K 80 GPU accelerators.
Nevertheless training the LSTM NER tagger still required about 19 hours: without the GPU accelerator the training would have been impossible. 8
Several papers discuss approaches to end-to-end
entity linking
Most approaches to mention detection rely on
some sort of fuzzy matching between n-grams in
the source and the list of known entities
Yamada et al. (2015) propose an end-to-end approach to entity linking that exploits word embeddings as features in a random-forest algorithm used for assigning a score to mention candidates, which are however identified by either exact or fuzzy matching on a mention-entity dictionary built from Wikipedia titles and anchor texts.
Guo et al. (2016) propose a structural SVM algorithm for entity linking that jointly optimizes mention detection and entity disambiguation as a single end-to-end task. 9
We presented an innovative approach to mention extraction and entity linking of tweets, that relies on Deep Learning techniques. In particular we use a Named Entity tagger for mention detection that exploits character-level embeddings in order to deal with the noise in the writing of tweet posts. We also exploit word embeddings as a measure of semantic relatedness for the task of entity linking.
As a side product we produced a new gold resource of 13,609 tweets (242,453 tokens) annotated with NE categories, leveraging on the resource distributed for the Evalita PoSTWITA task.
The approach achieved top score in the Evalita 2016 NEEL-IT Challenge and looks promising for further future enhancements.
We gratefully acknowledge the support by the University of Pisa through project PRA 2016 and by NVIDIA Corporation through the donation of a Tesla K40 GPU accelerator used in the experiments.
We thank Guillaume Lample for making available his implementation of Named Entity Recognizer. Information.