This paper presents an approach towards high performance extraction of biomedical entities from the literature, which is built by combining a high recall dictionarybased technique with a high-precision machine learning filtering step. The technique is then evaluated on the CRAFT corpus. We present the performance we obtained, analyze the errors and propose a possible follow-up of this work.
The problem of technical term extraction (herein
TTE) is the problem of extracting relevant
technical terms from a scientific paper. It can be seen
as related to Named Entity Recognition (NER),
where the entities one wants to extract are
technical terms belonging to a given field. For
example, while in traditional NER the entities that one
is looking for are of the types “Person”, “Date”,
“Location”, etc., in TTE we look for terms
belonging to a particular domain, e.g. “Gene”, “Protein”,
“Disease”, and so on
NER (and then TTE) can be solved using very different techniques: technique may require deep domain and linguistic knowledge. A simple example may be the task of recognizing US phone numbers, which can be solved by a simple regular expression.
Machine learning-based approach: a statistical classifier is used to recognize an entity, such as Naive Bayes, Conditional Random Fields, and so on. Several different types of features can be used by such systems, for example prefixes and suffixes of the entity candidates, the number of capital letters, etc. A major drawback of this approach is that it typically requires a large, manually annotated corpus for algorithm training and testing. Dictionary-based approach: candidate entities are matched against a dictionary of known entities. The obvious drawback of this approach is that it is not able to recognize new entities, making this technique ineffective e.g. in documents which present new discoveries.
Hybrid approaches: two or more of the previous techniques are used together. For example, Sasaki et al. (2008) as well as Akhondi et al. (2016) combine the dictionary and MLbased approaches to combine the strengths of both.
Rule-based approach: a group of manually written rules is used to identify entities. This The aim of this work is to propose a hybrid approach based on two stages. First, we have a dictionary phase, where a list of all the possible terms is generated by looking for matches in a database. This aims to build a low precision, high recall set with all the candidate TTs. Then, this set is filtered using a machine learning algorithm that ideally is able to discriminate between “good” and “bad” terms selected in the dictionary matching phase to augment the precision.
This approach is realized by using two software
modules. The first phase is performed by the
OntoGene pipeline
The field of technical term extraction has about 20
years of history, with early works focusing on
extracting a single category of terms, such as
protein names, from scientific papers
While the most recent state-of-the-art
performance is obtained by using machine learning
based systems
The CRAFT Corpus
The CRAFT corpus is a set of 671 manually annotated journal articles from the biomedical field. These articles are taken from the PubMed Central Open Access Subset,2 a part of the PubMed Central archive licensed under Creative Commons licenses.
The corpus contains about 100,000 concept annotations which point to seven ontologies/terminologies:
Chemical entities of Biological Interest (ChEBI) (Degtyarenko et al., 2008)
Entrez Gene
Sequence Ontology
To make references to documents in the CRAFT corpus easily retrievable for the reader, when we 1The full CRAFT corpus comprises another 30 annotated articles, which are reserved for future competitions and have to date not been released.
2http://www.ncbi.nlm.nih.gov/pmc/ tools/openftlist/
3https://github.com/obophenotype/ cell-ontology/ 4http://www.ncbi.nlm.nih.gov/taxonomy 5http://pir.georgetown.edu/pirwww/ index.shtml 6http://knowtator.sourceforge.net/ will refer to an article contained in the corpus we will list the name of its corresponding XML file as contained in the corpus distribution, its PubMed Central ID (PMCID), and its PubMed ID (PMID).7 3.2
The OntoGene group has developed an approach
for biomedical entity recognition based on
dictionary lookup and flexible matching. Their
approach has been used in several competitive
evaluations of biomedical text mining technologies,
often obtaining top-ranked results
OntoGene’s term annotation pipeline accepts a range of input formats, e.g. PubMed Central fulltext XML, gzipped chunks of Medline abstracts, BioC,8 or simply plain text. It provides the annotated terms along with the corresponding identifiers either in a simple tab-separated text file, in brat’s standoff format,9 or – again – in BioC. It allows for easily plugging in additional components, such as alternative NLP preprocessing methods or postfiltering routines.
In the present work, the pipeline was configured as follows: After sentence splitting, the input documents were tokenized with a simple method based on character class: Any contiguous sequence of either alphabetical or numerical characters was considered a token, whereas any other characters (punctuation and whitespace) were considered token boundaries and were ignored during the dictionary look-up. This lossy tokenization already has a normalizing effect, in that it collapses spelling variants which arise from inconsistent use of punctuation symbols, e.g. “SRC 1” vs. “SRC-1” vs. “SRC1”. (A similar approach is described by Verspoor et al. (2010), which refer to it as “regularization”.) All tokens are then converted to lowercase, except for acronyms that collide with a word from general language (e.g. “WAS”). We enforced a case-sensitive match in these cases by 7We will not include articles from the CRAFT corpus in the references as they are not actual bibliography for the purposes of this work.
8http://bioc.sourceforge.net/ 9http://brat.nlplab.org/standoff.html using a list of the most frequent English words. As a further normalization step, Greek letters were expanded to their letter name in Latin spelling, e.g. ! alpha, since this is a common alternation.
For term matching, we compiled a dictionary
resource using the Bio Term Hub
Selecting the seven ontologies used in CRAFT resulted in a term dictionary with 20.2 million entries. Based on preliminary tests, we removed all entries with terms shorter than two characters or terms consisting of digits only; this reduced the number of entries by less than 0.3%. In the OntoGene system, the entries of the term dictionary were then preprocessed in the same way as the documents. Finally, the input documents were compared to the dictionary with an exact-match strategy. 3.3
Distiller11 is an open source framework written in Java and R for machine learning, introduced in Basaldella et al. (2015). While the framework has its roots in the work of Pudota et al. (2010), thus focusing on the task of automatic keyphrase extraction (herein AKE), Distiller’s framework design allows us to adapt its pipeline to various purposes.
AKE is the problem of extracting relevant
phrases from a document
While AKE can be performed using
unsupervised techniques, the most successful results have
been obtained using a supervised machine
learning approach
To adapt Distiller to perform TTE effectively, we substituted the candidate generation phase with the output of OntoGene, i.e. candidate technical terms become the potential “key phrases”. This configuration is then evaluated as our baseline. Next, we gradually add new features into the system to train a machine learning model specialized in the actual TTE task, and assess the improvements in the performance of the system. 4 4.1
First, we evaluated the performance of the OntoGene/Distiller system using the same feature set used in the original keyphrase extraction model presented by Basaldella et al. (2015), which contains: Frequency The frequency of the candidate in the document, also known as TF.
Height The relative position of the first appearance of the candidate in the document.
Depth The relative position of the last appearance of the candidate in the document.
Lifespan The distance between the first and the last appearance of the candidate.
TF-IDF The peculiarity of the candidate with respect to the current document and the CRAFT corpus. This is a very common feature both in the AKE and TTE fields.
Abstract Presence A flag set to 1 if the candidate appears in the abstract, 0 otherwise. This is motivated by the fact that often keyphrases are found to appear in the abstract. 4.2
To improve the performance of the TTE task we start to augment our feature set by introducing features that should be able to catch some more finegrained information about the candidate terms. Title Presence A flag which is set to 1 if the term appears in the title of the document and 0 otherwise, much like the Abstract Presence feature.
Symbols Count A counter for the number of punctuation symbols, i.e. not whitespaces and not alpha-numeric characters, appearing in the candidate term.
Uppercase Count A counter for the number of uppercase characters in the candidate term. Lowercase Count A counter for the number of lowercase characters in the candidate term. Digits Count A counter for the number of digits in the candidate term.
Space Count A counter for the number of spaces in the candidate term.
Greek Flag A flag that is set to 1 if the candidate contains a Greek letter in spelled-out form, like “alpha”, “beta”, and so on.
These features offer a good improvement for detecting the particular shape that a technical term could have. For example, from the document PLoS Biol-2-1-314463.nxml (PMC: PMC314463, PMID: 14737183) we have the term “5-bromo-4-chloro-3-indolyl beta-D-galactoside”. This term contains: A spelled-out Greek letter, beta; An uppercase letter; Seven symbols (dashes);
A whitespace.
Without the new features this information would have been lost, so it may have been much harder to recognize the term as a technical one. 4.3
This small feature set is the baseline of the experimental evaluation performed on the proposed approach.
In this step we add even more features aimed at detecting more fine-grained information about candidate terms. The new features are: Dash flag Dashes are one of the most (if not the most) common symbols found in technical terms. This flag is set to 1 if the term contains a dash, 0 otherwise.
Ending number flag This flag is set to 1 if the term ends with a number, 0 otherwise.
Inside capitalization This flag is set to 1 if the term contains an uppercase letter which is not at the beginning of a token.
All uppercase This flag is set to 1 if the term contains only uppercase letters, 0 otherwise. All lowercase This flag is set to 1 if the term contains only lowercase letters, 0 otherwise. 4.4
This feature set adds information about the affixes (i.e. prefixes and suffixes) of the words. This information is particularly useful in the biomedical field, since affixes in this field convey often a particular meaning: for example, words ending with “ism” are typically diseases, words starting with “zoo” refer to something from the animal life, and so on. Another example is the naming of chemical compounds: for example, many ionic compounds have the suffix “ide”, such as Sodium Chloride (the common table salt).
Using the Bio Term Hub resource, we compiled a list of all the prefixes and suffixes of two or three letters from the following databases:
Cellosaurus,12 from the Swiss Institute of
Bioinformatics;
Chemical compounds found in the
Toxicogenomics Database (CTD),13 from the North
Carolina State University Comparative;
Diseases found in the CTD;
EntrezGene
Since not all affixes are equally important, the affixes list needs to be cut at some point. While a trivial decision could have been to pick the top 100 or 10% ranked prefixes and suffixes, our choice was to let the machine learning algorithm decide by itself where to apply the cut.
To obtain this goal, each affix a from a database D is assigned a normalized score s 2 [0; 1] computed this way: s(a) =
freq(a; D) max(ffreq(a1; D) : : : freq(ajDj; D)g) where freq(a; D) is the frequency of an affix a in D. This way we obtain a simple yet effective mechanism to let a ML algorithm learn which of affixes are the most important.
It is also worth noting that since we generate scores for prefixes and affixes of two and three letters from six databases, we have a total of 2 2 6 = 24 features generated with this approach. 4.5
Now that we have many features that are more specific for the technical term extraction field, we remove the baseline feature set, which was tailored on keyphrase extraction, to use only the features aimed at recognizing technical terms.
These features (depth, height, lifespan, frequency, abstract presence, title presence, TF-IDF) are specific for the AKE field and supposedly bring little value on knowing if a term is technical or not. In fact, a term may appear just once in a random position of the text, and still be technical; the same does not hold for a keyphrase, which is assumed to appear many times in specific positions (introduction, conclusions. . . ) in the text. 4.6
Both OntoGene and Distiller have been tested on a laptop computer with an Intel i7 4720HQ processor running at 2,6GHz, 16 GB RAM and a Crucial M.2 M550 SSD. The operating system was Ubuntu 15.10.
The speed was of 16275 words/second for OntoGene and 4745 words/second for Distiller. OntoGene requires an additional time of about 25 second to load the dictionary at start up, but since this operation is run only once we do not consider it for the average.
Recall 0.40 0.12 0.34 0.43 0.4 0.43 0.55 0.37
F1 0.42 0.19 0.40 0.47 0.41 0.43 0.42 0.51
MMTx MGrep Concept Mapper cTakes Dictionary Lookup cTakes Fast Lookup NOBLE Coder OntoGene OntoGene+Distiller Using the feature sets defined above, we trained a neural network to classify technical terms. The network used is a simple multi-layer perceptron, with one hidden layer containing twice the number of neurons of the input layer and configured to use maximum conditional likelihood. The network is trained using 47 documents of the CRAFT corpus as training set and its performance is evaluated on the remaining 20, which in turn form the test set.
We also experimented using a C5.0 decision tree, but with unsatisfactory results (the performance decreases with the number of features) so we do not include its analysis in this paper.
The metrics used are simple Precision, Recall and F1-Score. Table 1 presents the performance of the different iterations of the proposed system. Plain OntoGene obtains 55.0% recall and 34.2% precision, while the baseline AKE feature set improves the precision score with 69.2% score in precision but shows a dramatic drop in recall to 18.7%.
It can be seen that the introduction of TTEspecific features brings an important improvement in recall, with a 6% improvement between the baseline and Feature Set 1. Together with a small drop in precision by 1%, it augments the F1-score by 7 points.
Feature Set 2 performs slightly better than Feature Set 1, with a general improvement between 2% and 3%. Then Feature Set 3, adding the affixes, brings a great improvement of 7% F1-Score, thanks to a general improvement of precision and recall of the same order.
Finally, it is clear that the Feature Set 4 (i.e. all the TTE-focused features, without the AKEfocused ones) is the best performing one. The obtained precision of 85.3% is a large improvement from the baseline of 69% and more than twice the precision of the raw OntoGene output, which is just 34.2%.
More importantly, recall rises from 18.7% to 36.8% (over a theoretical maximum of 55.0% of the raw OntoGene output). Feature Sets 3 and 4 also obtain a better F1-score than OntoGene, with 45.7% and 51.5%, respectively, while the score obtained by the OntoGene system is just 42.1%.
To compare our pipeline with similar TTE/CR
software, we use the results by Tseytlin et al.
(2016), which compared the NOBLE coder with
MMTx,16 Concept Mapper,17 cTAKES18 and
MGrep
We must stress that our results are not directly comparable to the ones in Tseytlin et al. (2016), for three reasons. Firstly, we evaluate the combined pipeline only on a portion of the dataset, since a training set is needed for the Distiller system. Secondly, we do not do concept disambiguation, but rather we consider a true positive whenever our pipeline marks a term that spans the same text region as a CRAFT annotation, regardless of what entity is associated with this term, which is an easier task than concept recognition. On the other hand, Tseytlin et al. (2016) count also partial positives, i.e. if the software annotation does not exactly overlap with the gold annotation, they allocate one-half match in both precision and recall. Instead, while evaluating our system, we count only exact matches, giving a disadvantage to our system.
Still, the more than doubling of precision from the dictionary-only approach is noteworthy, especially because it compensates the loss in recall well enough to have a general improvement in F1-score. The comparison, while not completely fair, shows that the high precision of our system is hardly matched by other approaches.
The biggest drawback of our approach is the relatively low recall still obtained by the OntoGene pipeline, which puts an upper bound to the recall obtainable by the complete pipeline. The 55% recall score obtained on the CRAFT corpus is not a bad result per se, as it is better to the best performance obtained in Tseytlin et al. (2016) by NOBLE Coder and cTAKES Dictionary Lookup. Nevertheless, we believe that recall can be improved by addressing some specific issues we analyze in greater detail in Section 6.2.
6.1
Looking at the errors performed by our system, we believe that some outcomes that seem to be false positive should actually be marked as true positives. Take as an example document PLoS Genet-1-6-1342629.nxml (PMCID: PMC1315279, PMID: 16362077). In the Discussion section, we have (emphasis ours): Serum levels of estrogen decreased in aging Sam68 = females as expected; however, the leptin levels decreased in aged Sam68 = females.
The term estrogen is not annotated in the CRAFT corpus, even though it is found in the ChEBI resource. OntoGene, on the other hand, recognizes this as a relevant term. The same holds for the two other occurrences of this term in the same article.
In the Results section of the same document, we have
Given the apparent enhancement of mineralized nodule formation by Sam68 = bone marrow stromal cells ex vivo and the phenotype observed with short hairpin RNA (shRNA)treated C3H10T1/2, we stained sections of bone from 4- and 12-month-old mice for evidence of changes in marrow adiposity.
Here, OntoGene annotates the Sequence-Ontology term shRNA both in its full and abbreviated form. Nevertheless, they are missing from the CRAFT annotations (along with 6 more occurrences of shRNA); however, CRAFT provides annotations to parts of the term (hairpin and RNA).
Then, in the Materials and Methods section, we have
Briefly, cells were plated on glass coverslips or on dentin slices in 24-well cluster plates for assessment of cell number and pit number, respectively.
Again, the term dentin, which is present in the Cell Ontology, is found by OntoGene but absent from the CRAFT corpus, together with 5 more occurrences of the same term.
Looking at this example document, we can see that the annotation of the CRAFT corpus seems to be somewhat inconsistent. While the reasons may be various and perfectly reasonable (e.g. the guidelines might explicitly exclude the mentioned terms in that context), this fact may affect the training and evaluation of our system. 6.2
Many terms annotated in the CRAFT corpus are missed by the OntoGene pipeline. As a general observation, the OntoGene pipeline – originally geared towards matching gene and protein names – is not optimally adapted to the broad range of term types to be annotated. A small number of the misses (less than 1%) is caused by the enforced case-sensitive match for words from the general vocabulary (such as “Animal” at the beginning of a sentence). Another portion (around 5%) are due to the matching strategy, in that the aggressive tokenization method removed relevant information, such as trailing punctuation symbols or terms consisting entirely of punctuation (e.g. “+”). Approximately 9% are short terms of one or two characters’ length, which had been excluded from the dictionary a priori, as described above. A major portion, though, are inflectional and derviational variants, such as plural forms or derived adjectives (e.g. missed “mammalian” besides matched “mammal”). Some CRAFT annotations include modifiers that are missing from the dictionary, e.g. the protein name “TACC1” is matched on its own, but not when disambiguated with a species modifier such as “mouse TACC1”/“human TACC1”. Other occasional misses include paraphrase (“piece of sequence”) or spelling errors (“phophatase” instead of “phosphatase”). 7
In this paper we have presented and evaluated an approach towards efficient recognition of biomedical entities in the scientific literature. Although some limitations are still present in our system, we believe that this approach has the potential to deliver high quality entity recognition, not only for the scientific literature, but on any related form of textual document. We have analyzed the limitations of our approach, clearly discussing the causes of the low recall when evaluated over the CRAFT corpus. The results show that the postannotation filtering step can significantly increase precision at the cost of a small loss of recall. Additionally, the approach provides a good ranking of the candidate entities, thus enabling a manual selection of the best terms in the context of an assisted curation environment.
As for future work, we intend to improve coverage of the OntoGene pipeline with respect to the CRAFT annotations. Based on the false-negative analysis, the next steps include: (1) use a stemmer or lemmatizer, (2) optimize the punctuation handling, (3) revise the case-sensitive strategy.
We also plan to improve Distiller’s machine
learning phase, adding more features to the
neural network classifier or switching to other
approaches used in literature, such as conditional
random fields