In this paper, we describe our approach taken in the MSM2013 IE Challenge, which was aimed at concept extraction from microposts. The goal of the approach was to combine several existing NER tools which use different classification methods and benefit from their combination. Several NER tools have been chosen and individually evaluated on the challenge training set. We observed that some of these tools performed better on different entity types than other tools. In addition, different tools produced diverse results which brought a higher recall when combined than that of the best individual tool. As expected, the precision significantly decreased. The main challenge was in combining annotations extracted by diverse tools. Our approach was to exploit machine-learning methods. We have constructed feature vectors from the annotations yielded by different extraction tools and various text characteristics, and we have used several supervised classifiers to train the classification models. The results showed that several classification models have achieved better results than the best individual extractor.
Most of the current Named Entity Recognition (NER) methods have been designed for concept extraction from relatively long and grammatically correct texts, such as newswire texts or biomedical texts. More and more user-generated content on the Web consists of a relatively short text which is often grammatically incorrect (e.g., microposts, on which these methods perform worse). The goal of the approach proposed in this paper is to combine several different information extraction methods in order to reach a more precise concept extraction on relatively short texts. We hypothesized that if these methods were combined properly, they would perform better than the best individual method from the pool. This assumption was partially proven through the evaluation of several available and well-known NER tools that use different entity extraction methods. The merged results of these tools showed a higher recall than that of the best tool but with a very low precision. The goal was to reduce or eliminate this tradeoff. Higher recall indicates that different methods complement each other and that there is room for improvement. We have tried various machinelearning algorithms and built several models capable of producing results based on concepts extracted by yielded tools. The goal was to produce a model with the highest possible precision approximating the recall measured for unified extracted concepts. In the following sections, we describe the NER tools that have been used and how they individually performed on the MSM2013 IE Challenge (from here on referenced as “challenge”) training set (version 1.5). We briefly describe the methodology of our investigation (i.e., how our solution was built). 2
Our solution incorporates several available well-known NER tools: Annie Named
Entity Recognizer [
All of the tools used were evaluated on the challenge training set. There were three ways of computing the Precision, Recall, and F1 metrics used. The first method was strict (PS , RS and F1S ), which considered partially correct responses as incorrect; however, the second, lenient, considered them as correct (PL, RL
and F1L). The third method was an average of the previous two (PA, RA, and F1A). The evaluation results are shown in Fig. 1. We also evaluated the unified responses of all of the tools. Results showed that the recall was much higher (RS = 90%) than the best individual tool (Illinois NER got RS = 60%), but the precision was very poor (PS = 18%), hence the F1 score (F1S = 30%). The best performing tool on microposts was OpenCalais, which scored PS = 70%, RS = 58% and F1S = 64%.
RA
PA
F1A
RS 1.00 0.75 0.50 0.25 0.00
PS
PL
F1S
Annie Apache OpenNLP Illinois NER Illinois Wikifier LingPipe Open Calais Stanford NER WikipediaMiner F1L
RL Our goal was to create a model that would take the most relevant results detected by each tool and perform better than the best tool did individually. We have used statistical classifiers to achieve this goal. as a training vector, this description was an input for training a classification model. A vector of input training features was generated for each annotation found by integrated NER tools. We called this annotation a reference annotation. The vector of each reference annotation consisted of several sub-vectors. The first sub-vector of the training vector was an annotation vector. The annotation vector described the reference annotation – whether it was uppercase or lowercase, used a capital first letter or capitalized all of its words, the word count, and the type of the detected annotation (LOC, MISC, ORG, PER, NP noun phrase, VP verb phrase, OTHER). The second sub-vector described microposts as a whole. It contained features describing whether all words longer than four characters were capitalized, uppercase, or lowercase. The rest of the sub-vectors were computed according to the overlap of the reference annotation with annotations produced by other NER tools. Such sub-vector (termed a method vector by us) was computed for each extractor and contained four other vectors (average scores per named entity type) for each target entity type (LOC, MISC, ORG, PER). The average score vector consisted of five components – ail: the average intersection length of a reference annotation with annotations produced by other extractors (from here on referenced as other’s annotations), aiia: the average percentage intersection of other’s annotations with reference annotation, aiir: the average percentage intersection of a reference annotation with other’s annotations, average confidence (if the underlying extractors return such value), and variance of the average confidence. The last component in the training vector was the correct answer (i.e., the correct annotation type taken from manual annotation). 4.2
Several types of classification models were considered, especially tree-models
which allow the use of numerical and discrete attributes. Due to its large number
of trees, Random Forests looked very advisable and reliable during the first
round of testing. However, the increasing number of input attributes caused the
performance of Random Forests to degrade. Therefore, we used a single decision
tree generated by the C4.5 algorithm [
4.3
To get an idea of our model performance, we have trained the model on an 80% split of the challenge training set cleaned from duplicate records and have evaluated it on the remaining 20% split. The evaluation results are displayed in Table 1. We included the results from the best individually performing tools for each entity type. Three runs were submitted for evaluation in the challenge. The first run was generated by the C4.5 algorithm trained model with parameter M denoting the minimum number of instances per leaf set to 2. The second run was generated by the model trained with parameter M set to 3. The third run was based on the first run and involved specific post-processing. If a micropost identical to one in the training set was annotated, we extended the detected concepts by those from manually annotated training data (affecting three microposts). A gazetteer built from a list of organizations found in the training set has been used to extend the ORG annotations of the model (affecting 69 microposts). The models producing the submission results were trained on a full challenge training set. Acknowledgments. This work is supported by projects VEGA 2/0185/13, VENIS FP7-284984, CLAN APVV-0809-11 and ITMS: 26240220072.