PubMed, the main literature repository for the life sciences, contains more than 23 million publication references. In average nearly two publications per minute are added. There is a wealth of knowledge hidden in unstructed format in these publications that needs to be structured, linked, and semantically annotated so that it becomes actionable knowledge. We present an approach towards large-scale processing of biomedical literature in order to extract domain entities and semantic relationships among them. We describe some practical applications of the resulting knowledge base.
Text mining technologies are increasingly providing an e ective response to the
growing demand for faster access to the vast amounts of information hidden in
the literature. Recent comprehensive reviews of the eld are [
Several tools are becoming available which o er the capability to mine the
literature for speci c information, such as for example protein-protein
interactions or drug-disease relationships. Examples of well known biomedical text
mining tools are MetaMap [
The biomedical text mining community regularly veri es the progress of the
eld through competitive evaluations, such as BioCreative [
Evidence in support of relationships among biomedical entities, such as proteinprotein interactions, can be gathered from a multiplicity of sources. The larger the pool of evidence, the more likely a given interaction can be considered to be. In the context of biomedical text mining, this elementary observation can be translated into an approach that seeks to nd in the literature all available evidence for a given interaction, and thus provides a reliable means to assign it a likelihood score before delivering the results to an end user.
In this paper we present the results of an on-going collaborative project between a major pharmaceutical company and an academic group with extensive expertise in biomedical text mining, with the initial goal of extracting proteinprotein interactions from a large pool of supporting papers, later to be extended to di erent entity relationships.
The OntoGene group3 at the University of Zurich (UZH) specializes in mining
the scienti c literature for evidence of interactions among entities of relevance
for biomedical research (genes, proteins, drugs, diseases, chemicals). The quality
of the text mining tools developed by the group is demonstrated by top-ranked
results achieved at several community-organized text mining competitions [
In the rest of this paper we describe the methods that were used in the development of the system (sec. 2), then we brie y report the results of an evaluation (sec. 3), and nally we focus speci cally on some applications (sec. 4). 2
The OntoGene group developed in recent projects an advanced text mining
pipeline which is used to provide all the basic text mining capabilities that are
3 http://www.ontogene.org/
needed for the successful realization of the activities described in this paper. Our
text mining system has been evaluated in several community-organized
competitive evaluation tasks and always shown to perform at state-of-the-art levels, for
example obtaining best results in the annotation of experimental methods [
The OntoGene system uses terminology derived from life science databases,
storing internally the several possible names for a given term and its unique
identi er. All potential terms of relevance in the target collection are automatically
annotated with an approximate matching approach. The annotation process not
only identi es a string as a potential term of a given type (e.g. a protein), but
also associates to it the unique identi er of the term. In case of ambiguos terms,
several potential identi ers will be associated to the term. This high-recall
approach is then followed by a machine-learning based ltering step which removes
false positives and disambiguates ambiguos terms. In the case of the application
considered in this paper, the reference database was BioGrid4 [
In order to produce candidate interactions the system rst generates all pairwise combinations of the pairs (term,identifier) seen in the document, and scores them using information from the original database, as described below. A threshold on the score can then be used to select the best candidate interactions. For each of these interactions, evidence snippets from the text are provided.
BioGrid provides for each protein-protein interaction at least one reference, i.e. a paper where that interaction is mentioned. Our initial collection for the experiment described in this paper consisted of 20,000 PubMed abtracts, selected from the references of BioGrid, in order to insure that at least one protein-protein interaction would be detectable. In addition to the abstract itself, we use also the MeSH keywords and list of Chemical substances which are associated to the abstract in PubMed.
For every annotated term, and for every possible identi er for that term, we compute a concept score, which is based on the likelihood of that (term, identifier) pair to participate in a curated interaction in the reference database. Additional factors, such as the position in the document (title vs abstracts) are used as weights.
For every pairwise combination of concepts (term identi ers) we also compute a sentence score, based on all the sentences which contain annotated terms with the identi ers of the two concepts (excluding cases where the two identi ers happen to refer to the same term). Such sentences (using a distant-learning approach) are assumed to be positives if the pair of identi ers is provided in the reference database as a curated interaction. This probability, estimated with a Nave Bayes (NB) model having a bag of words as its features, tries to capture
the linguistic context in which the interaction can occur. For more technical
details about the approach described in this section, please consult [
The evaluation of the extracted interactions was performed using a 10-fold cross validation. The results are compared against the reference database (BioGrid in this case) taken as a gold standard. Care must be taken in the interpretation of these results, since the reference database does not provide the position of possible interactions, but only the fact that two interacting entities have been identi ed in a given paper. Therefore any results provided by the system in the form of a triple (article=id0; entity=id1; entity=id2) is considered positive if and only if the interaction (id1; id2) can be found in the database associated with paper id0.
Extending this approach to the evaluation of the detection of domain entities, we can consider a detected term as a true positive if it is associated to an identi er which is part of a curated relation for the analyzed article. This approach has the advantage that we can use the manually curated database as a gold standard for evaluation, but it has of course the disadvantage that a correctly annotated term might nonetheless be considered as a false positive simply because it is not part of one of the curated interactions.
Using the approach mentioned above, we evaluated the performance of the entity recognition component using the conventional evaluation measures. The basic system, as described above, reaches a recall of 70.6%, with however only a precision of 11.6% (F-score: 19.9%). After inspecting the results, we introduced a small number of exclusion rules for very frequent false positives, which lead to an improvement of precision (14.6%) at a small cost for recall (70.0%), leading to a clearly better F-score (24.2%).
There are several reasons for the relatively low level of these results. The low level of precision can be partly explained by the indirect evaluation mechanism, i.e. an annotated term is only considered a positive if it is part of an interaction, which results in several false positives which could actually be perfectly correct entity annotations. The recall is a ected by the fact that while database curators have access to the full articles, the system analyzes only abstracts.
In any case, our main concern in this experiment is the retrieval of
interactions, therefore we aim for for a relatively high recall at the entity level, even at
the cost of a low precision. The metric used for the evaluation of the detected
interactions is Threshold Average Precision (TAP-k) [
Since relations are pairwise combinations of detected entities, an upper boundary for the recall of relation extraction can be estimated from the entity recognition recall. Given the current recall value of 0.7 for entity recognition, relation extraction is not expected to exceed 0.5 (0:7 0:7). Precision in relation extraction, just like in entity extractions, is limited by the type of evaluation methodology. Relations which have not been curated in the reference databases will be considered as false positives, even though they might be mentioned in the text as legitimate interactions. Using di erent variants of the parameter which combines the concept score and the relation score we were able to obtain a best value of 0.229 for the TAP-10 score of extracted relations. 4
In the previous section we described an approach towards semi-automated semantic annotation of PubMed abstracts (or full papers, when available), using unique identi ers from reference databases. A web-based user interface allows interaction of the expert user with the text mining system in order to achieve an e cient and accurate annotation. The automated annotations are also used in a large-scale application which enables a semantic search for interactions among domain entities. 4.1
Large-scale interaction extraction and interaction validation As an application of the methods described in the previous section, we have analyzed the whole of PubMed, and produced a database of the extracted proteinprotein interactions. Each interaction is characterized by a con dence score (derived from the relation score) which summarizes in a compact form the reliability of the interaction based on the evidence spread across the entire literature.
There are several potential applications for this database. As a demonstration we implemented an interface (using Apache Solr) which allows examination of the results. The user can enter an arbitrary protein name, and the system will provide a list of candidate interactors, ranked according to the con dence score (see gure 1). Once the user selects one of these interactions, the system will deliver the textual snippets which are considered to be most relevant for that particular interaction. Figure 2 shows precisely this nal step (best evidence for a given interaction) from the current version of the interface.
In another application we have been given by a domain expert a list of several
hundred proteins of interest in a particular biological study (see gure 3). The
researcher was interested in what are the potential interactions among those
proteins. Since the number of potential interactions is quadratic to the number
of input proteins, it is useful, before planning an experimental validation, to have
some pre- ltering technique that allows to narrow down the space of interactions
to be investigated. Using the database described above we were able to reduce
considerably this set, removing a huge number of potential interactions for which
there is no evidence whatsoever in the literature. Additionally, the remaining
set of candidate interactions is ranked according to our con dence score, thus
providing a potential way to further narrow down the scope of the experimental
investigation (see gure 4), by selecting the highest ranked interaction interaction
candidates which are not yet known to the domain expert.
Biomedical curators are professionals with a strong background in the life
sciences who read the literature in search of particular items of information (e.g.
newly detected protein interactions), and store such information in public
databases, which can in turn be accessed later by the biologists. For example, UniProt
[31] collects information on all known proteins. IntAct [
Despite the signi cant improvements in the last couple of years, most experts agree that, at least for the time being, it is unrealistic to expect fully automated text mining systems to perform at a level acceptable for tasks that require high accuracy, such as automated database curation. However, existing systems can already achieve results which are su ciently good to be used in a semi-automated context, where a human expert validates the output of the system. One application where this support is badly needed is biomedical literature curation.
In order to satisfy this need, we have implemented a user-friendly web based interface which interfaces our text mining system and allows a domain expert to inspect the results of the automated annotation process (see Figure 5). The purpose of the system is to enable a human annotator/curator to leverage upon the result of a advanced text mining system in order to enhance the speed and e ectiveness of the annotation process.
In case of ambiguity, the curator is o ered the opportunity to correct the choices made by the system, at any of the di erent levels of processing: entity identi cation and disambiguation, organism selection, interaction candidates. The curator can access all the possible readings given by the system and select the most accurate. Candidate interactions are presented in a ranked order, according to the score assigned by the system. The curator can, for each of them, con rm, reject, or leave undecided. The results of the curation process can be fed back into the system, thus allowing incremental learning.
The documents and the annotations are represented consistently within a single XML le, which also contains a record of the user interaction, thus allowing advanced logging support. The annotations are selectively presented, in a ergonomic way through CSS formatting, according to di erent view modalities, While the XML annotations are transparent to the annotator (who therefore does not need to have any specialized knowledge beyond his biological expertise), his/her veri cation activities result in changes at the DOM of the XML document through client-side JavaScript. The use of modern AJAX methodology allows for online integration of background information, e.g. information from di erent term and knowledge bases, or further integration of foreign text mining services. The advantage of a client-side presentation logic is the exibility for the end user and the data transparency. For text mining applications, it is important to be able to link back curated metainformation to its textual evidence.
In a recently approved NIH-funded project (\High Throughput Literature Curation of Genetic Regulation in Bacterial Models") we intend to leverage the capabilities of the OntoGene/ODIN system in order to improve the e ciency of the curation process of the RegulonDB database. RegulonDB5 is the primary database on transcriptional regulation in Escherichia coli K-12 containing knowledge manually curated from original scienti c publications, complemented with high throughput datasets and comprehensive computational predictions. 5
We have presented an advanced text mining architecture, which is capable of automatically annotating the biomedical literature with domain entities of relevance for speci c applications, and to detect interactions among those entities. In particular, we have discussed and evaluated a speci c scenario for proteinprotein interactions.
Additionally, we discussed an application in assisted curation, and an application for the ltering of potential interactions among a given set of proteins. In order to support a process of assisted curation we provide a user-friendly webbased interface, which is currently being used by life science databases within the scope of large curation projects.
The OntoGene group at the University of Zurich is partially supported by the Swiss National Science Foundation (grants 100014 118396=1 and 105315 130558=1) and by F. Ho mann-La Roche Ltd, Basel, Switzerland. The collaboration with the RegulonDB group at the Mexican National University is sponsored by the NIH grant 1R01GM110597-01A1. 26. Sangkuhl, K., Berlin, D.S., Altman, R.B., Klein, T.E.: PharmGKB: Understanding the e ects of individual genetic variants. Drug Metabolism Reviews 40(4), 539{551 (2008), http://informahealthcare.com/doi/abs/10.1080/03602530802413338, pMID: 18949600 27. Segura-Bedmar, I., Mart nez, P., Sanchez-Cisneros, D.: The 1st ddi extraction2011 challenge task: Extraction of drug-drug interactions from biomedical texts.
In: Proc DDI Extraction-2011 challenge task. pp. 1{9. Huelva, Spain (2011) 28. Stark, C., Breitkreutz, B.J., Reguly, T., Boucher, L., Breitkreutz, A., Tyers, M.: Biogrid: A general repository for interaction datasets. Nucleic Acids Research 34, D535{9 (2006) 29. Sun, W., Rumshisky, A., Uzuner, O.: Evaluating temporal relations in clinical text: 2012 i2b2 Challenge. J Am Med Inform Assoc 20(5), 806{813 (2013) 30. Thomas, P., Starlinger, J., Vowinkel, A., Arzt, S., Leser, U.: GeneView: a comprehensive semantic search engine for PubMed. Nucleic Acids Res. 40(Web Server issue), W585{591 (Jul 2012) 31. UniProt Consortium: The universal protein resource (uniprot). Nucleic Acids Research 35, D193{7 (2007)