Machine learning approaches based on supervised classification have emerged as effective methods for Biomedical relation extraction such as the Chemical-InducedDisease (CID) task. These approaches owe their success to a rich set of features crafted from the lexical and syntactic regularities in the text. Kernel methods are an effective alternative to manual feature engineering and have been successfully used in similar tasks such as text classification. In this paper, we study the effectiveness of tree kernels for Chemical-Disease relation extraction. Our experiments demonstrate that subset tree kernels increase the F-score to 61:7% as compared to 57:9% achieved with simple feature engineering. We also describe the strengths and shortcomings of tree kernel approaches for the CID relation extraction task.
Scientific publications in the fields of biomedical
and life sciences are vast and growing fast (Haas
et al., 2014). Prior research has shown that
Chemicals and Diseases and their relationships are
among the most searched topics by PubMed users
(Dogan et al., 2009), due to their importance in
applications such as toxicology, drug discovery and
safety surveillance. Efforts to manually curate and
extract these important concepts such as
Chemicals and Diseases and their relationships have
led to the creation of structured databases such
as the Comparative Toxicogenomics Database
(CTD) (Davis et al., 2012). However, manual
curation is unlikely to scale
The recent shared task for
Chemical-InducedDisease relation extraction (CID) organized by
BioCreative-V
Kernel methods in NLP (Collins and Duffy, 2001) have been designed precisely to address this problem of manual feature engineering. These methods enable an efficient and comprehensive exploration of a very high dimensional feature space and to automatically adapt to the dominant patterns expressed in the training set.
In our work, we show that kernel methods can
be used for boosting relation extraction
performance without having to manually engineer
additional features. We demonstrate through
experiments that combining tree kernels over constituent
parses with simple lexical and syntactic features
can substantially enhance the performance of the
CID task. We also discuss the strengths and
weaknesses of these methods which can assist in the
design of better methods in the future.
Our system is developed in the context of the CID
subtask described in BioCreative-V
Our goal is to minimize task specific and domain specific feature engineering. We therefore explore the power of domain independent techniques such as kernel methods for effective relation extraction. Kernel methods automatically explore a large feature space and can reduce the need for rich hand crafted features. In our system, we do not employ any preprocessing or custom filtering techniques. We use simple text based features and a knowledge base (CTD) look up as our primary feature set. Further knowledge extraction from text is accomplished through tree kernels.
We cast the CID relation extraction as a binary classification problem. The input to the classifier is a pair of chemical and disease mentions. From the set of predicted relation mentions, we extract their normalized entity ids (MeSH Ids) and add it to the final list of Chemical-Disease relations. We built and tested two types of classifiers, namely linear classifier and tree kernel classifier. The linear classifier uses a flat list of simple features. The tree kernel classifier uses kernel methods over constituent parse trees of input sentences. The detailed steps are described below: 1. Every chemical mention (C) that appears in the article is paired with every other disease mention (D) to generate an entity pair (C-D) for classification. An entity pair in which both the entity mentions are within a sentence are referred to as intrasentence pairs, and those that cross a sentence boundary are referred to as intersentence pairs. The full test data is the union of intrasentence and intersentence entity pairs. 2. Intersentence and intrasentence pairs are grouped separately for training and testing with two separate classifiers. No further filtering or post-processing of (C-D) pairs is performed. 3. At training time, we label a (C-D) pair as positive if there exists a valid CID relation between these entities, using the relation annotations. At test time, the label is inferred from the classifier output. 4. Features for intrasentence pairs include verbs, bag of words, POS tags, dependency parse and the token distance between entity mentions in the sentence. 5. Features for intersentence pairs include the POS tags and bag of words of the two sentences containing entity mentions, distance (number of sentences) between them, statistical features of the entity mentions in the document (frequency of mentions), entity frequencies and zonal information (document zone containing the mentions). 6. We use the Chemical Toxicology Database (Davis et al., 2012) to generate a binary feature Ictd(C; D) that evaluates to 1 if the (C-D) pair is known to be related in the CTD database and 0 otherwise.
We used a Support Vector Machine (SVM) with linear kernel from Scikit (Pedregosa et al., 2011) to classify candidate entity pairs. The predicted (C-D) pairs from the sentence level and document level classifiers are combined to form the final list of document level CID relations. We refer to this system as “Linear Classifier”. 3.2
Kernel methods have gained wide spread acceptance, because they allow direct computation of similarity (dot product) between two examples in an implicitly mapped high dimensional NP
JJ
space
Given two trees T1 and T2 and the set of all possible tree fragments F = ff1; f2; : : :g, an indicator function Ii(n) is defined which evaluates to 1 if the fragment fi is rooted at node n and 0 otherwise. The unnormalized kernel score is given by where NT1 and NT2 are the sets of nodes of T1 and T2 respectively and (n1; n2) = PjiF=j1 Ii(n1)Ii(n2). The normalized kernel score is given by
K(T1; T2) =
k0(T1; T2) pk0(T1; T1) k0(T2; T2)
We experimented with subtree and subset tree kernels over constituent parse trees and found subset tree kernels to be superior for our task. In the rest of the paper, we mean subset tree kernel when we refer to tree kernels. We used Stanford CoreNLP (Manning et al., 2014) to generate the constituent parse trees. For tree kernels we use the SVM-LIGHT-TK toolkit1 (Moschitti, 2006a) that offers kernel implementation within SVM2 (Joachims, 1999). For each intrasentence (C-D) pair we get a single parse tree and for each intersentence pair we get a forest of two constituent parse trees, from each sentence containing one of the two entity mentions.
The contribution from flat features as defined in the section 3.1 can be combined with tree kernels by linearly combining the dot products of the flat feature vectors and the tree kernel. That is, the kernel for the new classifier (linear + tree kernel) is computed as the sum of the linear kernel over flat features and the tree kernel over the constituent parse trees. We report results for these classifiers, namely Linear, Tree Kernel and Linear + Tree Kernel classifier in Section 4.
Tree kernels attempt to classify a sentence in its entirety and in its default form are unaware of the entity mentions in the sentence. This approach is suitable if our goal is to simply detect if a sentence expresses a relation or not. However, to render greater focus on the entity mentions, we can preprocess the sentence to highlight the location of a word with reference to entity mentions. We prefixed all words in the sentence with “pre”, “mid”, and “post” tags, based on whether they are located prior to, in between, or post entity mentions, before generating the constituent parse trees.
4.1
We work with the dataset provided by
BioCreative-V
We measure the effectiveness of our relation
extraction system over the provided test data set, as
set out in the CID task. We use the standard
entity annotations provided with the data set. Given
the limited annotated data, we decided to use both
the training and development data set for
classifier training with default settings and no
custom parameter tuning. Results for intersentence,
intrasentence and the full set of (C-D) pairs are
presented for linear classifier, tree kernel
classifier and their combination, in Table 1. We also
present the results for the Linear classifier without
the CTD feature. Finally, the table also contains
the results reported in a prior work by
To summarize, our final system (linear + tree kernel) achieves an F-score of 61:7% over the CID test data. Note that the combination of linear and tree kernels outperforms the linear and tree kernel classifiers individually. The Table also reveals the substantial contribution of the CTD look up feature towards the linear classifier’s performance. 5
lished results in the CID BioCreative-V task used
custom entity recognition tools. Therefore, their
CID performance is not directly comparable
without replicating their entity annotation process.
A more accurate comparison can be made with
Intrasentence Intrasentence Intrasentence Intrasentence Intersentence Intersentence Intersentence Intersentence Full test Full test Full test
Lin + TK
Lin - CTD
Lin
TK
Lin + TK
Lin - CTD
Lin
TK
Lin + TK
Effectiveness of Tree Kernels. We note that tree kernels can significantly improve the performance of CID relation extraction as illustrated in the results. Also, this additional performance is obtained using PubMed abstracts and not external information sources. These results suggest that a greater amount of information exists in annotated text that is easier to extract with tree kernels as compared to manual feature mining for richer patterns. Further, tree kernels have an effect of increasing the precision of the classifiers, specially for intersentence cases. This is likely due to the fact that tree kernels enable stringent comparison of sentence structures (constituent parse trees) as compared to the lenient approach of bag of words matching with linear kernels.
Further enhancements. Incorporating entity focus to tree kernels (section 3.2.1) produced a slight improvement (to 61:7% from 61:0%). This approach is likely to be beneficial for sentences that express multiple relations (> 1) between multiple entity pairs. In the CID dataset, we found that sentences expressing multiple relations constitute around 14%, 15% and 14% of training, development and test datasets respectively. Alternate approaches that discriminate parts of the sentence based on the relation expressed are likely to further improve the performance.
In the context of intersentence (C-D) pairs, we are currently using only the two sentences si; sj that contain the entity mentions. However, the actual relationship might be collectively expressed by any subset of the sentences in the document. We attempted to model the whole of the document as a forest of parse trees of all its sentences, but did not observe any improvement in performance. For the CID task where more than 30% of the (CD) pairs cross sentence boundaries, effective intersentence relation extraction remains a challenge. 6
In this work, we show that tree kernels were very effective in CID relation extraction and boosted F1 score to 61:7% as compared to 57:9% achieved with a linear classifier using simple handcrafted features alone. In future work, we seek to improve intersentence relation extraction from documents. of Protein-Protein Interaction. (2011):124–133.
BioNLP 2011, Michael Collins and Nigel Duffy. 2001. Convolution kernels for natural language. In Advances in neural information processing systems, pages 625–632. Allan Peter Davis, Cynthia Grondin Murphy, Robin Johnson, Jean M Lay, Kelley Lennon-Hopkins, Cynthia Saraceni-Richards, Daniela Sciaky, Benjamin L King, Michael C Rosenstein, Thomas C Wiegers, et al. 2012. The comparative toxicogenomics database: update 2013. Nucleic acids research, page gks994.
Rezarta Islamaj Dogan, G Craig Murray, Aure´lie Ne´ve´ol, and Zhiyong Lu. 2009. Understanding pubmed R user search behavior through log analysis. Database, 2009:bap018.
Laura Haas, Melissa Cefkin, Cheryl Kieliszewski, Wil Plouffe, and Mary Roth. 2014. The ibm research accelerated discovery lab. SIGMOD Rec., 43(2):41– 48, December.
Thorsten Joachims. 1999. Advances in kernel methods. chapter Making Large-scale Support Vector Machine Learning Practical, pages 169–184. MIT Press, Cambridge, MA, USA.
Man Lan, Chew Lim Tan, and Jian Su. 2009. Feature generation and representations for protein-protein interaction classification. Journal of Biomedical Informatics, 42:866–872.
Christopher D Manning, Mihai Surdeanu, John Bauer, Jenny Finkel, Steven J Bethard, and David McClosky. 2014. The stanford corenlp natural language processing toolkit. In Proceedings of 52nd Annual Meeting of the Association for Computational Linguistics: System Demonstrations, pages 55–60.
Alessandro Moschitti. 2006a. Efficient convolution kernels for dependency and constituent syntactic trees. In Machine Learning: ECML 2006, pages 318–329. Springer.
Alessandro Moschitti. 2006b. Making tree kernels practical for natural language learning. In EACL.