Text mining approaches for automated ontology-based curation of biological and biomedical literature have largely focused on syntactic and lexical analysis along with machine learning. Recent advances in deep learning have shown increased accuracy for textual data annotation. However, the application of deep learning for ontology-based curation is a relatively new area and prior work has focused on a limited set of models. Here, we introduce a new deep learning model/architecture based on combining multiple Gated Recurrent Units (GRU) with a character+word based input. We use data from five ontologies in the CRAFT corpus as a Gold Standard to evaluate our model's performance. We also compare our model to seven models from prior work. We use four metrics Precision, Recall, F1 score, and a semantic similarity metric (Jaccard similarity) to compare our model's output to the Gold Standard. Our model resulted in a 84% Precision, 84% Recall, 83% F1, and a 84% Jaccard similarity. Results show that our GRU-based model outperforms prior models across all five ontologies. We also observed that character+word inputs result in a higher performance across models as compared to word only inputs. These findings indicate that deep learning algorithms are a promising avenue to be explored for automated ontology-based curation of data. This study also serves as a formal comparison and guideline for building and selecting deep learning models and architectures for ontology-based curation.
II. INTRODUCTION
Ontology-based data representation has been widely adopted
in data intensive fields such as biology and biomedicine due
to the need for large scale computationally amenable data [
One of the most important tasks for annotating scientific
literature with ontology concepts is Named Entity Recognition
(NER). In the context of ontology-based annotation, NER can
be described as recognizing ontology concepts from text [
The large majority of ontology driven NER techniques rely
on lexical and syntactic analysis of text in addition to machine
learning for recognizing and tagging ontology concepts [
Here, we present a new deep learning architecture that utilizes Gated Recurrent Units (GRU) while taking advantage of word and character encodings from the annotation training data to recognize ontology concepts from text. We evaluate our model in comparison to 7 deep learning models used in prior work to show that our model outperforms the state of art at the task of ontology-based NER.
We use the Colorado Richly Annotated Full-Text (CRAFT)
corpus [
Precision and Recall are traditionally used to assess the
performance of information retrieval systems. However, these
metrics do not take into account the notion of partial
information retrieval which is important for ontology-based annotation
retrieval. Sometimes, an NLP system might not retrieve the
same ontology concept as the gold standard but a related
concept (sub-class or super-class). To assess the performance
of the NLP system accurately, we need semantic
similarity metrics that can measure different degrees of semantic
relatedness between ontology concepts [
III.
RELATED WORK
The application of deep learning for ontology-based Named
Entity Recognition is a nascent area with relatively little prior
work. Habibi et al. [
The striking difference between these prior studies and our work here is that the majority of prior literature focuses on LSTM based methods along with CRF and word embeddings. The potential of other deep learning models such as Recurrent Neural Networks, Gated Recurrent Units, etc., at the task of ontology-based NER remains unexplored presenting a unique need and opportunity. Our study aims to fill this knowledge gap. In addition, all the above studies focus on non-ontology based NER for entities such as genes, disease names, etc. In contrast, our study’s focus is on recognizing ontology concepts within text.
IV.
METHODS
Annotation files for the 67 papers in CRAFT were cleaned to remove punctuation symbols (except for period at the end of sentences), special symbols, and non-ASCII characters. Annotations for GO, CHEBI, Cell, Protein, and Sequence ontologies were converted from the cleaned files to separate ontology-specific text files that represent the presence or absence of ontology terms. For each ontology, every sentence containing at least one annotation from that ontology was represented using two lines in the ontology-specific text file. The first of these two lines contained an array with each word in the sentence. The second contained an ordered encoding corresponding to words in the first line. These encodings could be an ontology concept ID if the corresponding word was annotated in CRAFT or an 0O0 if the corresponding word was not annotated.
For example, the sentence “Rod and cone photoreceptors subserve vision under dim and bright light conditions respectively” where the word “vision” was annotated to GO ID “GO:0007601 (perception of sight)” would be represented using the two lines below: [’Rod’, ’and’, ’cone’, ’photoreceptors’, ’subserve’, ’vision’, ’under’, ’dim’, ’and’, ’bright’, ’light’, ’conditions’, ’respectively’] [’O’, ’O’, ’O’, ’O’, ’O’, ’GO:0007601’, ’O’, ’O’, ’O’, ’O’, ’O’, ’O’, ’O’]
Annotations to single words (unigrams) were only included in these preprocessed files. So, if an annotation was made in CRAFT to a phrase containing more than one word, it was ignored in the preprocessed data.
Precision, Recall, F1-score, and Jaccard similarity were used
to evaluate the performance of the models. The Jaccard
similarity (J ) of two ontology concepts (in this case, annotations)
(A, B) in an ontology is defined as the ratio of the number of
classes in the intersection of their subsumers over the number
of classes in their union of their subsumers [
J (A; B) = jS(A) \ S(B)j
jS(A) [ S(B)j where S(A) is the set of classes that subsume A. Jaccard similarity ranges from 0 (no similarity) to 1 (exact match).
Below, we describe four deep learning models - Multilayer perceptrons, Recurrent Neural Networks, Long ShortTerm Memory, and Gated Recurrent Units. Next, we describe three architectures - window based, word based, and characterword based that can be used in conjunction with the above models. Finally, we describe our new model that combines character-word based architecture with Gated Recurrent Units and six models used in prior work.
1) Multi-Layer Perceptron (MLP): A Multi-Layer
Perceptron (MLP) [
2) Recurrent Neural Network: A Recurrent Neural
Network (RNN) [
0; sof tmax(W sht)
t = 0 (W hhht 1 + W hxxt); t > 0 (1) where, xt is the input provided to the hidden state ht at time t which is updated using a sigmoid function . is calculated over the previous time state of the network given by ht 1 and current input xt. W hh, W hx, and W s are the weights computed over training. The network can then produce an output prediction < y0; y1; y2; yj > using a sof tmax function on the hidden state ht.
A bidirectional Recurrent Neural Network (BiRNN) is an
RNN where the input data is fed to the neural network two
times - once in forward and again in reverse order.
3) Long-Short Term Memory: While RNNs are effective
in learning temporal patterns, they suffer from a vanishing
gradient problem where long term dependencies are lost. A
solution to the problem was proposed by Hochreiter et al. [
Whht 1) ~ ht where, zt and rt are update and reset gates respectively, h˜t is the candidate activation/hidden state.
Similar to the LSTM architecture, GRUs benefit from the additive properties in their network to remember long term dependencies, and solve the vanishing gradient problem. Since GRUs do not utilize an an output gate, they are able to write the entire contents of their memory cell to the network. The lack of a memory cell also makes GRUs more efficient in comparison to LSTMs.
Below, we describe three architectures - window-based, word-based, and word+character based to be used in conjunction with the different models described above. where, it, ft, and ot are the input, forget, and output gates respectively. Each gate uses a sigmoid ( ) function applied over the sum of input xt and previous hidden state ht 1 (multiplied with their weight matrices W ). gt denotes the candidate state computed over a tanh function on the input and previous hidden state. W ix, W fx, W ox, W gx are weight matrices used with input xt, while W ih, W fh, W oh, and W gh are used with hidden states for each gate and candidate state. The memory cell ct utilizes the forget gate (ft) and multiplies ( - element-wise) it old memory cell ct 1 and adds to the state of candidate (gt) multiplied with the input gate (it). The hidden state is given by a tanh function applied to the memory cell ct multiplied with output gate (ot).
4) Gated Recurrent Unit: A variation on LSTM, was
introduced by Cho et al. [
1) Window-based: In this architecture, the window-based input (iv) consists of feature vectors (fv) for each word/term (t) within an encoded sentence. Each fv consisted of the following attributes: fv =< t; nt; t1; t 1; tC ; taC ; tla; t0p; t0p 1; t0p 2; ts0; ts0 1; ts0 2; tN ; tP > (4) where, t is the term, nt is the number of terms in the sentence, t1 is a boolean value indicating if the term is the first term in the sentence, t 1 is 1 if term is the last term in the sentence 0 otherwise, tC is 1 if first letter in t is uppercase, taC is 1 if all letter in t are uppercase, tla is 1 if all in t are lower case, t0p; t0p 1; t0p 2 record character prefixes of t at various window size,
ts0; ts0 1; ts0 2 record character suffixes of t at various window sizes, tN and tP are the next and previous terms respectively. 2) Word-based: Each word and its corresponding annotation labels (tags) are encoded with integer values, derived from unique words and annotations present in the corpus. The dataset was based on unigram annotations that only use ontology annotations where a single word in text maps to an ontology concept.
In word-based architectures (Figure 1), the input (XtWr ) is provided to an Embedding layer which converts the input into dense vectors of 100 dimensions. The output vectors are then fed to a bidirectional model (RNN/GRU/LSTM) consisting of 150 hidden units. The output from the model goes to a dense perceptron layer using ReLU activation which also employs a 0.6 Dropout. The output is further fed into a CRF layer which looks for correlations between annotations in close sequences to generate the predictions (ypr).
3) Character+Word Based: A Character+Word based architecture is similar to the word based architecture described above. In addition to word-based inputs (XtWr ) is also takes advantage of characters (XtCr) within words to make predictions.
We developed a new deep learning architecture that uses a Character+word based architecture coupled with two bidirectional Gated Recurrent Units. Our architecture (Figure 2) consists of character level input (XtCr) provided to an Embedding layer (E1) which compresses the dimensions of characters to the number of unique annotations in the corpus (N T ags).
The output of Embedding layer E1 is fed to a bidirectional GRU (BiGRU1) layer with 150 units followed by a 60% output drop in a Dropout layer (D1). Simultaneously, the word-level input (XtWr ) was provided to a second Embedding layer (E2) with 30 dimensions. The output from E2 was concatenated with the output from the first Dropout layer D1 and fed through a second Dropout layer (D2) with a 30% drop. Output from D2 was fed into a second bidirectional GRU layer (BiGRU2) consisting of 150 units.
The above model was tested with and without a final CRF layer leading to two new configurations - CW BiGRU CRF and CW BiGRU . The models were run for 15 epochs with a batchsize of 32 instances.
We compared the performance of our new Character+word based GRU architecture and the two models developed therein (CW BiGRU CRF , CW BiGRU ) (Section IV-E) to six state of the art models that have been used in prior work. Below, we specify the component details of each of the six prior models that have been evaluated.
1) MLP: Multi layer perceptrons were used with a window based architecture to create a three layered (input, hidden, output) M LP model. The input and the hidden layer consisted of 512 perceptrons with a Rectified Linear Unit (ReLU) activation function while the output layer consisted of perceptrons equal to the number of unique annotations in the corpus (N T ags). 20% Dropout was used for the hidden and output layers to prevent overfitting of the data. Categorical cross-entropy was used for calculating the loss function and NAdam (Adam RMSprop with Nesterov momentum) was used as the optimizer function. Each of the feature vectors (from the training data), were fed into the MLP architecture for 15 epochs with a batch size of 256.
2) BiRNN-CRF: The BiRNN-CRF model uses a wordbased input coupled with a BiRNN model and ending with a CRF model. Similar to the BiRNN architecture (Figure 1), the BiRNN-CRF model consists of a 100 dimension Embedding layer followed by a BiRNN with 150 units followed by a 0.6 Dropout layer. The output of the 0.6 Dropout layer is fed to a CRF which generated the predicted output.
3) BiLSTM-CRF: The BiLSTM-CRF model is identical to the BiRNN-CRF except that it uses a LSTM in place of the RNN.
4) BiGRU-CRF: The BiGRU-CRF model is identical to BiRNN-CRF and BiLSTM-CRF except that it uses a Gated Recurrent Unit in place of the RNN or LSTM.
5) CW-BiLSTM: The CW-BiLSTM model is similar to the CW-BiGRU model described above (see Section IV-E) except that the BiGRU is replaced with a BiLSTM.
6) CW-BiLSTM-CRF: The CW-BiLSTM-CRF model is developed by adding a CRF layer at the end of the CWBiLSTM model pipeline indicating that the output of the CWBiLSTM model would be fed to a CRF layer to generate the final predictions.
The GO annotation data was split into training and test sets using a 70:30 ratio. The training set was used to tune the following parameters for all models. Multiple architecture parameters such as - 1) Number of layers in MLP (along with number of perceptrons), 2) Number of units in RNN/GRU/LSTM, 3) Embedding Dimensions for Characters and Words, and 4) Optimization functions, were evaluated for model performance. A grid-search model was explored, where each architecture was evaluated for different combinations of the parameter. In each case, model performance metrics were recorded in form of Precision, Recall, F1-score, and Jaccard similarity. H. Experiments to predict ontology annotations
The largest number of annotations in the CRAFT corpus came from the Gene Ontology. So, we first used the GO annotations to train and test the suite of 8 models described above. Subsequently, we applied the best model from these experiments to annotate the CRAFT corpus with the other four ontologies (Chebi, Cell, Protein, and Sequence corpora).
Root-Mean-Square propagation (RMSProp) optimizer was used to test the performance of the different models. A batch size of 32 along with 15 epochs was used for model training. Performance characteristics in terms of train-test loss (calculated using the CRF function), prediction precision, recall, F1score along with mean semantic similarity score was recorded for each model.
V.
RESULTS AND DISCUSSION
The CRAFT corpus contains 67 full length papers with annotations from five ontologies (GO, CHEBI, Cell, Protein, and Sequence). For each of these ontologies, we extracted all sentences across the 67 papers with at least one annotation for the ontology. The largest number of annotations came from the GO (Table I) while the Cell ontology accounted for the lowest number of annotations.
Figure 3 shows the loss and accuracy trends for each model on the GO annotation data. The goal of the models is to minimize loss while increasing accuracy as the number of epochs increase.
First, we see that our CW-BiGRU model shows improvement in both training and validation accuracy as the number of epochs increase. Correspondingly, we observe a decrease in training and validation loss indicating that the model is able to self-improve with each subsequent epoch.
The CW-BiGRU-CRF model initially shows the same accuracy improvement like the CW-BiGRU model but later increases in epochs result in a divergence in the training and validation accuracy indicating that the model might be prone to overfitting. While there is a substantial decrease in training loss, a similar decrease is not observed in validation loss.
CW-BiLSTM shows similar trends to CW-BiGRU. CWBiLSTM-CRF training and validation accuracy increase similarly until a certain point after which the validation accuracy drops and diverges sharply from the training curve indicating a case of overfitting.
BiGRU-CRF and BiRNN-CRF models show substantial improvement in accuracy with increasing epochs. However, BiRNN-CRF shows divergence in the loss patterns. Similar to CW-BiLSTM-CRF, BiLSTM-CRF also shows signs of overfitting in the accuracy patterns. MLP is the worst performing model with very minor improvements in validation accuracy as the number of epochs increase indicating that the model is unable to improve itself with each subsequent epoch.
It is clear that the CW-BiGRU models are able to outperform the other models by improving accuracy and reducing loss with each epoch without overfitting.
A large proportion of input data is not annotated to GO terms but to a tag 0O0 indicating the absence of an annotation. In addition to accurately predicting GO annotations, the model also needs to accurately predict the absence of an annotation. However, given the disproportionate amount of data pertaining to the absence of annotations, the models were observed to predict the absence of annotations remarkably accurately in comparison to predicting presence.
To provide a more conservative view of the models’ performance, we report Precision, Recall, F-1 Score, and Jaccard similarity (Table II) only on data indicating presence of ontology terms, i.e. text annotated with an ontology term. Unlike the accuracy measurements above, the metrics below do not take into account the models’ performance at identifying the absence of annotations, but rather focus on ability to identify annotations when they’re present in the Gold Standard.
These results (Table II and Figure 3) show that our model (CW-BiGRU) outperforms the other 7 models in all four metrics. Our model outperforms the best among the other 7 models (CW-BiLSTM) by 4% (Precision), 2% (Recall), 3% (F1 score), 1% (Jaccard similarity).
Additionally, we observe that character-word based models (CW-BiGRU, CW-BiLSTM, CW-BiLSTM-CRF, CW-BiGRUCRF,) outperform models that use only word embeddings.
Among the character-word based models, surprisingly, the addition of an extra CRF layer (CW-BiLSTM-CRF, CWBiGRU-CRF) either fails to improve performance (e.g CWBiLSTM vs. CW-BiLSTM-CRF) or leads to a decline in performance (e.g CW-BiGRU vs. CW-BiGRU-CRF) as compared to not using a CRF end layer (CW-BiLSTM, CW-BiGRU). The MLP model shows substantially lower performance as compared to the other models across all four metrics. The Accuracy and Loss plots (Figure 3) suggest that the decline in performance when adding a CRF layer is due to potential overfitting.
We explored how predictions from our best model, CWBiGRU, diverge from the Gold Standard. We found that the majority of predictions (89.25%) are an exact match for the CRAFT annotations. Surprisingly, only a small proportion of predictions are partial matches (2.45%). 8.26% of the model’s predictions are false negatives while 6.38% are false positives. We hypothesize that one of the primary reasons for false negatives might be lack of enough training instances for those particular GO annotations.
Finally, we applied the best performing model from the above evaluation (CW-BiGRU) and tested it on data from four other ontologies. Interestingly, the model shows better prediction performance on the other ontologies as compared to GO despite the substantially smaller training datasets (Table III).
The data used in this study was limited to single words annotated to ontology concepts (unigrams). Next, we will explore more robust models including n-grams to account for sequences of words tagged with an annotation. Future work will also include models that can be trained to weight the prediction of some target classes higher than others. These models would be able to prioritize presence prediction of annotations as compared to the absence of an annotation.
This study demonstrates the utility of deep learning approaches for automated ontology-based curation of scientific literature. Specifically, we show that models based on Gated Recurrent Units are more powerful and accurate at annotation prediction as compared to the LSTM based models in prior work. Our findings indicate that deep learning is a promising new direction for ontology-based text mining, and can be used for more sophisticated annotation tasks (such as phenotype curation) that build upon Named Entity Recognition. Model 1.00 0.99 0.98 0.97 cy0.96 a ru0.95 cc0.94 A0.93 0.92 0.91 0.90 0.70 0.60 0.50 ss0.40 o L0.30 0.20 0.10 1.00 0.99 0.98 0.97 cy0.96 a ru0.95 cc0.94 A0.93 0.92 0.91 0.90 17.40 17.30 17.20 s so17.10 L 17.00 16.90 16.80 CW-BiGRU
CW-BiGRU-CRF CW-BiLSTM
CW-BiLSTM-CRF 0 2 4 6 8 10 12 14
BiGRU-CRF 0 2 4 6 8 10 12 14
BIRNN-CRF 0 2 4 6 8 10 12 14
BiLSTM-CRF