The 1st DDIExtraction-2011 challenge task: Extraction of Drug-Drug Interactions from biomedical texts

Abstract

We present an evaluation task designed to provide a framework for comparing different approaches to extracting drug-drug interactions from biomedical texts. We define the task, describe the training/test data, list the participating systems and discuss their results. There were 10 teams who submitted a total of 40 runs.

Keywords

• Biomedical Text Mining
• Drug-Drug Interaction Extraction

Work

A drug-drug interaction (DDI) occurs when one drug influences the level or activity of another drug. Since negative DDIs can be very dangerous, DDI detection is the subject of an important field of research that is crucial for both patient safety and health care cost control. Although health care professionals are supported in DDI detection by different databases, those being used currently are rarely complete, since their update periods can be as long as three years [ 12 ]. Drug interactions are frequently reported in journals of clinical pharmacology and technical reports, making medical literature the most effective source for the detection of DDIs. The management of DDIs is a critical issue, therefore, due to the overwhelming amount of information available [ 8 ].

Information extraction (IE) can be of great benefit for both the pharmaceutical industry by facilitating the identification and extraction of relevant information on DDIs, as well as health care professionals by reducing the time spent reviewing the relevant literature. Moreover, the development of tools for automatically extracting DDIs is essential for improving and updating the drug knowledge databases.

Different systems have been developed for the extraction of biomedical relations, particularly PPIs, from texts. Nevertheless, few approaches have been proposed to the problem of extracting DDIs in biomedical texts. We developed two different approaches for DDI extraction. Since no benchmark corpus was available to evaluate our approaches to DDI extraction, we created the DrugDDI corpus annotated with 3,160 DDIs. Our first approach is a hybrid linguistic approach [ 13 ] that combines shallow parsing and syntactic simplification with pattern matching. This system yielded a precision of 48.69%, a recall of 25.70% and an F-measure of 33.64%. Our second approach [ 14 ] is based on a supervised machine learning technique, more specifically, the shallow linguistic kernel proposed in Giuliano et al. (2006) [ 7 ]. It achieved a precision of 51.03%, a recall of 72.82% and an F-measure of 60.01%.

In order to stimulate research in this direction, we have organized the challenge task DDIExtraction2011. Likewise the BioCreAtIvE (Critical Assessment of Information Extraction systems in Biology) challenge evaluation has devoted to provide a common frameworks for evaluation of text mining driving progress in text mining techniques applied to the biological domain, our purpose is to create a benchmark dataset and evaluation task that will enable researchers to compare their algorithms applied to the extraction of drug-drug interactions. 2

The DrugDDI corpus

While Natural Language Processing(NLP) techniques are relatively domainportable, corpora are not. For this reason, we created the first annotated corpus, the DrugDDI corpus, studying the phenomenon of interactions among drugs. We hope that the corpus serves to encourage the NLP community to conduct further research in the field of pharmacology.

As source of unstructured textual information on drugs and their interactions, we used the DrugBank database[ 17 ]. This database is a rich resource combining chemical and pharmaceutical information of approximately 4,900 pharmacological substances. For each drug, DrugBank contains more than 100 data fields including drug synonyms, brand names, chemical formula and structure, drug categories, ATC and AHFS codes (i.e., codes of standard drug families), mechanism of action, indication, dosage forms, toxicity, etc. Of particular interest to this study, DrugBank offers the field ’Interactions’ (it is no longer available) that contained a link to a document describing DDIs in unstructured texts. DrugBank provides a file with the names of approved drugs1, approximately 1,450. We randomly chose 1,000 drug names and used the RobotMaker2, a screen-scrapper application, to download the interaction documents for these drugs. We only retrieved a total of 930 documents since some drugs did not have any linked document. Due to the cost-intensive and time consuming nature of the annotation process, we decided to reduce the number of documents to be annotated and only considered 579 documents. We believe that these texts are a reliable and representative source of data for expressing DDI since the language used is mostly devoted to descriptions of DDIs. Additionally, the highly specialized pharmacological language is very similar to that found in the Medline pharmacology abstracts.

These documents were then analyzed by the UMLS MetaMap Transfer (MMTx) [ 2 ] tool performing sentence splitting, tokenization, POS-tagging, shal1 http://www.drugbank.ca/downloads 2 http://openkapow.com/

( low syntactic parsing (see Figure 1) and linking of phrases with UMLS Metathesaurus concepts. Drugs are automatically identified by MMTx since the tool allows for the recognition and annotation of biomedical entities occurring in texts according to the UMLS semantic types. An experienced pharmacist reviewed the UMLS Semantic Network as well as the semantic annotation provided by MMTx and recommended us the inclusion of the following UMLS semantic types as possible types of interacting drugs: Clinical Drug (clnd), Pharmacological Substance (phsu), Antibiotic (antb), Biologically Active Substance (bacs), Chemical Viewed Structurally (chvs) and Amino Acid, Peptide, or Protein (aapp).

The principal value of the DrugDDI corpus undoubtedly comes from its DDIs annotations. To obtain these annotations, all documents were marked-up by a researcher with pharmaceutical background. DDIs were annotated at the sentence level and, thus, any interactions spanning over several sentences were not annotated here. Only sentences with two or more drugs were considered and the annotation was made sentence by sentence. Figure 1 shows an example of an annotated sentence that contains three interactions. Each interaction is represented as a DDI node in which the names of the interacting drugs are registered in its NAME DRUG 1 and NAME DRUG 2 attributes. The identifiers of the phrases containing these interacting drugs are also annotated, providing an easily access to the related concepts provided by MMTx. As mentioned, Figure 1 shows three DDIs: the first DDI represents an interaction between Aspirin and probenecid, the second one an interaction between aspirin and sulfinpyrazone, and the last one a DDI between aspirin and phenylbutazone.

The DrugDDI corpus is also provided in the unified format for PPI corpora proposed in Pyysalo et al. [ 11 ] (see Figure 2). This shared format could attract attention of groups studying PPI extraction because they could easily adapt their systems to the problem of DDI extraction. The unified XML format does not contain any linguistic information provided by MMTx. The unified format only provides the sentences, their drugs and their interactions. Each entity (drug) includes reference (origId) to its id phrase in the MMTX format corpus text in which the corresponding drug appears. For each sentence from the DrugDDI corpus represented in the unified XML format, its DDI candidate pairs should be generated from the different drugs appearing therein. Each DDI candidate pair is represented as a pair node in which the ids of the interacting drugs are registered in its e1 and e2 attributes. If the pair is a DDI, the interaction attribute must be set to true, and false value otherwise.

Table 1 shows basic statistics of the DrugDDI corpus. In general, the size of biomedical corpora is quite small and usually does not exceed 1,000 sentences. The average number of sentences per MedLine abstract was estimated at 7.2 ± 1.9 [ 18 ]. Our corpus contains 5,806 sentences with 10.3 sentences per document on average. MMTx identified a total of 66,021 phrases of which 12.5% (8,260) are drugs. The average number of drug mentions per document was 24.9, and the average number of drug mentions per sentence was 2.4. The corpus contains a total of 3,775 sentences with two or more drug mentions, although only 2,044 sentences contain at least one interaction. With the assistance of a pharmacist, a total of 3,160 DDIs were with an average of 5.46 DDIs per document and 0.54 per sentence.

DDI extraction can be formulated as a supervised learning problem, more particularly, as a drug pair classification task. Therefore, a crucial step is to /

( generate suitable datasets to train and test a classifier from the DrugDDI corpus. The simplest way to generate examples to train a classifier for a specific relation R is to enumerate all possible ordered pairs of sentence entities. We proceeded in a similar way. Given a sentence S with at least two drugs, we defined D as the set of drugs in S and N as the number of drugs. The set of examples generated for S, therefore, was defined as follows: {(Di, Dj ) : Di, Dj D, 1 <= i, j <= N, i = j, i < j}. If the interaction existed between the two DDI candidate drugs, then the example was labeled 1. Otherwise, it was labeled 0. Although some DDIs may be asymmetrical, the roles of the interacting drugs were not included in the corpus annotation and are not specifically addressed in this task. As a result, we enumerate candidate pairs here without taking their order into account, such that (Di, Dj ) and (Dj , Di) are considered as a single candidate pair. Since the order of the drugs in the sentence was not taken into account, each example is the copy of the original sentence S where the candidates were assigned the tag, ’DRUG’, and remaining drugs were assigned the tag, ’OTHER’. The set of possible candidate pairs was the set of 2−combinations from the whole set of drugs appearing in S. Thus, the number of examples was CN,2 = N2 .

Table 2 shows the total number of relation examples or instances generated from the DrugDDI corpus. Among the 30,757 candidate drug pairs, only 3,160 (10.27%) were marked as positive interactions (i.e., DDIs) while 27,597 (89.73%) were marked as negative interactions (i.e., non-DDIs).

Once we generated the set of relation instances from the DrugDDI corpus, the set was then split in order to build the datasets for the training and evaluation of the different DDI extraction systems. In order to build the training dataset used for development tests, 75% of the DrugDDI corpus files (435 files) were randomly selected for the training dataset and the remaining 25% (144 files) is used in the final evaluation to determine which model was superior. Table 3 shows the distribution of the documents, sentences, drugs and DDIs in each set. Approximately 90% of the instances in the training dataset were negative examples (i.e., non-DDIs). The distribution between positive and negative examples in the final test dataset was also quite similar (see Table 2). 3

The participants

The task of extracting drug-drug interactions from biomedical texts has attracted the participation of 10 teams who submitted 40 runs. Table 4 lists the teams,

Set Training Final Test Total their affiliations, the number of runs submitted and the description of their systems.

The runs’ performance information in terms of precision, recall, F-measure and accuracy, appears in Table 5. The best performance is achieved by the team WBI [ 15 ]. Its system combines several kernels (APG [ 1 ], SL [ 7 ], kBSPS [ 16 ]) and a case-based reasoning (CBR) (called MOARA [ 10 ]) using a voting approach. In particular, the combination of the kernels APG, SL and the MOARA system yields the best F-measure (0.6574).

The team FBK-HLT [ 5 ] proposes new composite kernels using well-known kernels such as MEDT [ 6 ], PST [ 9 ] and SL [ 7 ]. Similarly, the team LIMSI-FBK [ 4 ] combines the same kernels (MEDT, PST and SL) and a feature-based method using SVM. This system achieves an F-measure of 0.6398.

The team Uturku [ 3 ] proposes a feature-based method using the classifiers SVM and RLS. Features used by the classifiers include syntactic information (tokens, dependency types, POS tags, text, stems, etc) and semantic knowledge from DrugBank and MetaMap. This system achieves an F-measure of 0.6299.

In general, approaches based on kernels methods achieved better results than the classical feature-based methods. Most systems have used primarily syntactic information, however semantic information has been poorly used. 5

Conclusion

This paper describes a new semantic evaluation task, Extraction of drug-drug interactions from biomedical texts. We have accomplished our goal of providing a framework and a benchmark data set to allow for comparisons of methods for this task. The results that the participating systems have reported show successful approaches to this difficult task, and the advantages of kernel-based methods over classical machine learning classifiers.

The success of the task shows that the framework and the data are useful resources. By making this collection freely accessible, we encourage further research into this domain. Moreover, next SemEval-3 (6th International Workshop on Semantic Evaluations3) to be held in summer 2013 has scheduled the ”Extraction of drug-drug interactions from biomedical Texts” task 4. In order to accomplish this new task, the current corpus is being extended to collect new data test.

Acknowledgements

This study was funded by the projects MA2VICMR (S2009/TIC-1542) and MULTIMEDICA (TIN2010-20644-C03-01). The organizers are particularly grateful to all participants who contributed to detect annotation errors in the corpus. 3 http://www.cs.york.ac.uk/semeval/ 4 http://www.cs.york.ac.uk/semeval/proposal-16.html

References

1. Airola , A. , Pyysalo , S. , Bjorne , J. , Pahikkala , T. , Ginter , F. , Salakoski , T. : Allpaths graph kernel for protein-protein interaction extraction with evaluation of cross-corpus learning . BMC bioinformatics 9(Suppl 11 ), S2 ( 2008 )

2. Aronson , A.R. : Effective mapping of biomedical text to the UMLS Metathesaurus: the MetaMap program . Annual AMIA Symposium pp. 17 - 21 ( Jan 2001 )

3. Bjo¨rne, J., Airola , A. , Pahikkala , T. , Salakoski , T. : Drug-drug interaction extraction with rls and svm classiffers . In: Proceedings of the First Challenge task on Drug-Drug Interaction Extraction (DDIExtraction 2011 ) ( 2011 )

4. Chowdhury , M. , Abacha , A. , Lavelli , A. , P. , Z. : Two different machine learning techniques for drug-drug interaction extraction . In: Proceedings of the First Challenge task on Drug-Drug Interaction Extraction (DDIExtraction 2011 ) ( 2011 )

5. Chowdhury , M. , Lavelli , A. : Drug-drug interaction extraction using composite kernels . In: Proceedings of the First Challenge task on Drug-Drug Interaction Extraction (DDIExtraction 2011 ) ( 2011 )

6. Chowdhury , M. , Lavelli , A. , Moschitti , A. : A study on dependency tree kernels for automatic extraction of protein-protein interaction . ACL HLT 2011 p. 124

7. Giuliano , C. , Lavelli , A. , Romano , L. : Exploiting shallow linguistic information for relation extraction from biomedical literature . In: Proceedings of the Eleventh Conference of the European Chapter of the Association for Computational Linguistics (EACL-2006) . pp. 401 - 408 ( 2006 )

8. Hansten , P.D.: Drug interaction management . Pharmacy World & Science 25 ( 3 ), 94 - 97 ( 2003 )

9. Moschitti , A. : A study on convolution kernels for shallow semantic parsing . In: Proceedings of the 42nd Annual Meeting on Association for Computational Linguistics . pp. 335 - es . Association for Computational Linguistics ( 2004 )

10. Neves , M. , Carazo , J. , Pascual-Montano , A. : Extraction of biomedical events using case-based reasoning . In: Proceedings of the Workshop on BioNLP: Shared Task . pp. 68 - 76 . Association for Computational Linguistics ( 2009 )

11. Pyysalo , S. , Airola , A. , Heimonen , J. , Bjorne , J. , Ginter , F. , Salakoski , T. : Comparative analysis of five protein-protein interaction corpora . BMC bioinformatics 9(Suppl 3) , S6 ( 2008 )

12. Rodr ´ıguez- Terol , A. , Camacho , C. , Others : Calidad estructural de las bases de datos de interacciones. Farmacia Hospitalaria 33 ( 03 ), 134 ( 2009 )

13. Segura-Bedmar , I. , Mart´ınez, P., de Pablo-S´anchez, C.: A linguistic rule-based approach to extract drug-drug interactions from pharmacological documents . BMC Bioinformatics 12 ( Suppl 2 ), S1 ( 2011 )

14. Segura-Bedmar , I. , Mart´ınez, P., de Pablo-Sa´nchez, C.: Using a shallow linguistic kernel for drug-drug interaction extraction . Journal of Biomedical Informatics In Press, Corrected Proof ( 2011 )

15. Thomas , P. , Neves , M. , Solt , I. , Tikk , D. , Leser , U. : Relation extraction for drugdrug interactions using ensemble learning . In: Proceedings of the First Challenge task on Drug-Drug Interaction Extraction (DDIExtraction 2011 ) ( 2011 )

16. Tikk , D. , Thomas, P. , Palaga , P. , Hakenberg , J. , Leser , U. : A comprehensive benchmark of kernel methods to extract protein-protein interactions from literature . PLoS Computational Biology 6 ( 7 ), e1000837 ( 2010 )

17. Wishart , D.S. , Knox , C. , Guo , A.C., Cheng, D., Shrivastava , S. , Tzur , D. , Gautam , B. , Hassanali , M.: DrugBank: a knowledgebase for drugs, drug actions and drug targets . Nucleic acids research 36(Database issue), D901-6 (Jan 2008 )

18. Yu , H.: Towards answering biological questions with experimental evidence: automatically identifying text that summarize image content in full-text articles . Annual AMIA Symposium proceedings pp. 834 - 8 ( Jan 2006 )