In this paper, we explore the problem of recognizing named entities in microposts, a genre with notoriously little context surrounding each named entity and inconsistent use of grammar, punctuation, capitalization, and spelling conventions by authors. In spite of the challenges associated with information extraction from microposts, it remains an increasingly important genre. This paper presents the MIT Information Extraction Toolkit (MITIE) and explores its adaptability to the micropost genre.
*This work was sponsored by the Defense Advanced Research Projects Agency under Air Force Contract FA8721-05-C-0002. Opinions, interpretations, conclusions, and recommendations are those of the authors and are not necessarily endorsed by the United States Government.
Copyright c 2016 held by author(s)/owner(s); copying permitted only for private and academic purposes.
Published as part of the #Microposts2016 Workshop proceedings,
available online as CEUR Vol-1691 (http://ceur-ws.org/Vol-1691)
2. MITIE
The MIT Information Extraction Toolkit (MITIE) [3] is a free,
open-source software library of state-of-the-art NLP tools
developed at MIT Lincoln Laboratory. MITIE enables the
automated extraction of named entities and of binary relations (for
example, a person’s place of birth) from unstructured text in
English and Spanish. MITIE utilizes distributional word
embeddings [
One of the goals in developing MITIE was to enable fast named
entity recognition. To this end, MITIE is capable of processing
53,600 words per second when run single-threaded on a 2.4GHz
Intel Xeon processor. Even with this speed, accuracy was not
compromised and MITIE is able to achieve an F1 score of 88.1 on
the CoNLL 203 NER task [3] [
We utilized the training data from the NEEL 2016 Challenge Data
Set [
Our experiments consisted of varying the number of training documents and testing on the remainder of the documents, utilizing 5-fold cross validation. No out of domain training data was used to supplement the in-domain data. Each document corresponded to a single tweet. The documents were not guaranteed to contain any mentions of named entities. For each experiment, we trained a single MITIE model to simultaneously classify all seven of the entity types under consideration.
Across all of the entity types other than character and thing, training with in-domain data began to show diminishing (but still positive) returns with 500 training documents. This was seen in measuring F1 and precision and recall independently, as shown in Figures 1, 2, and 3. Also of note was that increasing the number of in-domain training documents benefited performance in precision significantly more than recall for all entity types.
We considered the hypothesis that the difference in performance in correctly recognizing different types of entity mentions was due to the number of times that that entity type appeared in the training data. This hypothesis however, proved to be false. Of particular interest is the performance in recognizing event mentions. Despite the fact that this was a particularly rare entity in this corpus, MITIE excelled at recognizing event mentions, particularly with regard to precision. While not a sufficient condition, there is a threshold quantity of mentions of a given entity type which is necessary for NER accuracy to be significantly above chance performance. As seen in Figure 4, the character entity type is extremely rare and correspondingly begins showing large beginning with 3000 training documents. performance gains Also of note was the consistently poor performance in recognizing mentions of thing entities. We hypothesize that this is due to thing being a poorly defined entity type, but have not yet tested that hypothesis.
In this paper, we presented exploratory analysis comparing the number of in-domain training documents used with named entity recognition performance in the micropost genre. In this analysis, we also compared performance on recognizing different entity types. Additionally, we presented the MIT Information Extraction Toolkit, an open-source structural SVM approach to named entity recognition and binary relation extraction.
In future work we would like to explore other dimensions of retraining NER systems. Some particular questions of interest are examining whether the patterns seen in the number of in-domain training micropost documents required are mirrored in other domains and identifying a causal factor that explains the varying performance in recognizing entities of different types.
We would like to thank Davis King, Arjun Majumdar, and Michael Yee for their work on developing MITIE.
[3] D. King, "MITLL/MITIE," https://github.com/mit-nlp/MITIE.
Available: [Online].