Supervised machine learning methods have been widely used in relation extraction that finds the relation between two named entities in a sentence. However, their disadvantages are that constructing training data is a cost and time consuming job, and the machine learning system is dependent on the domain of the training data. To overcome these disadvantages, we construct a weakly labeled data set using distant supervision and propose a relation extraction system using a transformation-based learning (TBL) method. This model showed a high F1-measure (86.57%) for the test data collected using distant supervision but a low F1-measure (81.93%) for gold label, due to errors in the training data collected by the distant supervision method.
In natural language documents, there are a huge number of relations between named
entities. Automatic extraction of these relations from documents would be highly
beneficial in the data analysis fields such as question answering and social network
analysis. Previous studies on relation extraction [
As shown in Figure 1, the proposed system consists of three parts: (1) a distant supervision part that generates weakly labeled data from a relation knowledge base and articles, (2) a training part that generates a TBL model by using the weakly labeled data as training data, and (3) an applying part that extracts relations from new articles using the TBL model. For distant supervision, we use DBpedia ontology as a knowledge base. The DBpedia ontology is populated using a rule-based semi-automatic approach that relies on Wikipedia infoboxes, a set of subject-attribute-value triples that represents a summary of some unifying aspect that the Wikipedia articles share. As shown in Figure 2, the proposed system extracts sentences from Wikipedia articles by using the triple information of Wikipedia infobox. For example, the second sentence in Figure 2 is extracted from the weakly labeled data having a “Born” relation between “Madonna” and “Bay City, Michigan” and a “Residence” relation between “Madonna” and “New York City.” 2.2
To extract features from the weakly labeled data, the proposed system performs
natural language processing using Apache OpenNLP [
For TBL, the proposed system uses two kinds of templates; a morpheme-level template and a syntax-level template. As shown in Figure 3, the morpheme-level template is constituted by the combination of words and POSs that are present in the left and right three words of the target named entities based on the POS tagging results. In Figure 3, the numbers described as “-n” and “+n” mean a left n-th word and right n-th word, respectively, from the target named entities. Then, “NEC” means the class name of the target named entity. As shown in Figure 4, the syntax-level template is constituted by the combination of two target named entities, common head word of the two named entities, head word (parent node in a dependency tree) of the two named entities, and dependent word (child node in a dependency tree) of the two named entities based on dependency parsing results.
We used DBpedia ontology 3.9 in order to generate weakly labeled data. Then, we selected the ‘person’ class as a relation extraction domain. Next, we selected five attributes highly occurred in the infobox templates of the ‘person’ class. Finally, we constructed a weakly labeled data set by using distant supervision. Table 1 shows the distribution of the weakly labeled data set. For the experiment, 90% of the weakly labeled data were used as training data, and the remaining 10% were used as test data (hereinafter “weak-label test data”). In addition, to measure the reliability of the weak-label test data, a gold-label test data set, which was manually annotated with correct answers after randomly selecting 822 items, was constructed. During manual annotation, 395 noise sentences out of 822 sentences were removed. 3.2
As shown in Table 2, the proposed system showed high performance with regard to the weak-label test data but low performance with regard to the gold-label test data. This is because of noise sentences (i.e., sentences that do not describe the relation between two named entities) that are included in the training data collected through the distant supervision method. For example, “Christopher Plummer” and “Academy Award” have an “Award” relation, and based on this, the extracted sentence (“The film also features an extensive supporting cast including Amanda Peet, Tim Blake Nelson, Alexander Siddig, Amr Waked and Christopher Plummer, as well as Academy Award winners Chris Cooper, William Hurt”) describes the “Award” relation between “Chris Cooper and William Hurt” and “Academy Award.” The results of analysis of the gold-label test data showed that there were 395 noise sentences out of 822 sentences. The proposed system showed high performance for the weak-label test data. The fact reveals that the proposed system can show better performance for the gold-label test data only if good quality training data is ensured. In order to solve this problem, a method that can reduce the noise of the training data collected through distant supervision is required. 4
We proposed a relation extraction system using TBL based on distant supervision. In the proposed system, transformation rules were extracted based on a morpheme-level template that reflects the linguistic characteristics around named entities. They were also extracted based on a syntax-level template that reflects the syntactic dependency between two named entities. The experiment results indicated that higher performance could be expected only when the quality of the training data, which were extracted based on the distant supervision method, is improved. In the future, we will study on a method to increase the quality of training data collected by distant supervision.
This research was supported by Basic Science Research Program through the National Research Foundation of Korea(NRF) funded by the Ministry of Education, Science and Technology(2013R1A1A4A01005074), and was supported by ATC(Advanced Technology Center) Program “Development of Conversational Q&A Search Framework Based On Linked Data: Project No. 10048448”. It was also supported by 2014 Research Grant from Kangwon National University(No. C1010876-01-01). 6