Natural language processing methods powered by deep learning have been well-studied over the past years for the task of automated ontology-based annotation of scientific literature. Many of these approaches focus solely on learning associations between text and ontology concepts and use that to annotate new text. However, a great deal of information is embedded in the ontology structure and semantics. Here, we present deep learning architectures that learn not only associations between text and ontology concepts but also the structure of the ontology. Our experiments show that creating architectures that are capable of learning the structure of the ontology result in enhanced annotation performance.
Biological ontologies are widely used for representing biological knowledge across a wide range
of sub-domains ranging from gene function to clinical diagnoses to evolutionary phenotypes
[
Natural Language Processing (NLP) techniques beginning with lexical analysis, standard
machine learning approaches, and of late, powered by deep learning models have made big
strides in this area [
ceur-ws.org ISSN1613-0073 semantics of an ontology whereas generic entities can be independent objects. Knowledge of the ontological structure and relationships is a crucial part of biological annotation when performed by a human curator. It is therefore imperative to develop NLP models that are cognizant of the ontological hierarchy and can efectively incorporate it into the prediction mechanism for improved ontology concept recognition.
The automated annotation models previously developed by this team [
Our goal in this study is to develop deep learning architectures that learn not only patterns in text but also the ontology structure. Our hypothesis is that the process of learning the ontology structure would in turn improve prediction of annotations. Deep learning models learn patterns in text and annotations from a gold standard corpus and similarly, we need to provide a gold standard representation of the ontology structure so the models can learn to predict the ontology structure.
In this study, we use graph embeddings for representing the ontology structure. These graph
embeddings are used as a reference and reinforcement tool for the model as it learns to predicts
the ontology structure. Semantic embedding of large knowledge graphs has been long used
successfully for predictive tasks including natural language processing [
There are several approaches for learning ontology embeddings [
We use the Colorado Richly Annotated Full Text Corpus (CRAFT) as a gold standard for
training and testing the performance of our architectures [
We hypothesize that the added information gained from ontology embeddings can improve model performance in recognizing ontology concepts from scientific literature. We persent two deep learning architectures and explore how the diferent architectures combined with the inclusion of ontology embeddings impacts annotation performance.
The rise of deep learning in the areas of image and speech recognition has translated into
text-based problems as well. Preliminary research has shown that deep learning methods
result in greater accuracy for text-based tasks including identifying ontology concepts in
text [
Our initial foray into this area involved a feasibility study of using deep learning for the task
of recognizing ontology concepts [
In 2020, we presented new architectures based on GRUs and LSTMs combined with diferent
input encoding formats for automated annotation of ontology concepts [
Continuing this work, a 2022 study [
Our next work explored a diferent approach to providing the ontology as input to the deep
learning model [
Our most recent contribution presented a method called Ontology Boosting [
We used the Node2Vec approach for generating ontology embeddings from the Gene Ontology.
The Node2Vec algorithm consists of two steps:
1. Conduct random walks from a graph or ontology to generate sentences which are a list of
ontology concepts. Once all random walks are conducted, the set of all sentences makes
the corpus which is a representation of the ontology.
2. The Word2Vec [
These embeddings or feature vectors can be used in downstream tasks such as classification or natural language processing. in a downstream task such as node classification.
We set the weight of all edges to 1 for weighted random walks indicating that all edges are weighted equally. The length of random walk was set to 5 and the walk number set to 100. Dimensionality of embeddings was set to 128 and batch size is set to 50 and the model was trained for 2 epochs to learn the embeddings.
Here, we present and test three sets of architectures: 1. Baseline • Tag only ( ) • Ontology Embedding only (
)
2. Cross-connected:
3. Multi-connected:
• Tag to Ontology Embedding ( − >
• Ontology Embedding to Tag (− >
)
)
• Embedding to Tag to Embedding
3.2.1. Baseline Architectures
We created two baseline architectures (Figure 1): Tag only ( ) and Ontology Embedding only
( ). The architecture predicts tags/ontology IDs while the architecture predicts
ontology embeddings. The architecture has previously been presented in our prior work
[
The baseline architectures use three inputs. Each word in a sentence from the CRAFT corpus is represented by three inputs - 1) token ( ), 2) character sequence ( ℎ ), 3) Parts-OfSpeech (POS) ( ).The token ( ) input, is a sequential tensor consisting tokens, each ℎ ) is represented with a high dimensional one hot encoded vector. The character sequence ( also a sequential tensor consisting of character sequences present in a word/token. POS tags ( ) indicate the type of words in a sentence.
Embeddings are used to provide a compressed latent space representation for very high dimensional input components. For example, the one hot vectorization of an individual word has a dimensionality of 34,166 (vocabulary size). In order to represent them succinctly and with contextual representation, we use supervised embeddings created from the CRAFT corpus. Note that these embeddings are diferent from the ontology embeddings discussed above. These embeddings provide low dimensional representations of words in the training corpus and do not use the ontology in any way.
Both baseline architectures use a bi-directional gated recurrent model (Bi-GRU). The choice of
Bi-GRU for the architectures was informed by several of our prior works where this model has
consistently outperformed other models such as CNNs, RNNS, and LSTMs [
The two baseline architectures difer primarily because of what they produce as output and what they use during the propagation stages. In , the output is a tag/ontology ID where each word in the input data is mapped to either a GO annotation or a non-annotation. takes the hidden/learned representations of the input from the preceding layers of the network and applies softmax activation to produce a probability distribution over all possible ontology ids. The predicted vector output values and ground truth values are compared to compute sparse categorical cross entropy as loss, followed by backpropagation which involves computing the gradients of the loss with respect to the model’s weights. The ontology ID with the highest probability is regarded as the prediction.
In contrast, uses ground truth ontology embeddings generated using Node2Vec during back propagation and for computing the loss functions. The intuition is that providing ontology embeddings to the architecture during the propagation stages will enable it to get an understanding of the ontology structure and eventually enable it to make more accurate and intelligent predictions. The output of is an ontology embedding. The predicted ontology embedding is compared to all ground truth ontology embeddings using cosine similarity calculation. The ground truth ontology embedding that is most similar to the predicted embedding is identified and the ontology ID associated with it is treated as the architecture’s prediction. Accuracy metrics are then computed by comparing the predicted ontology ID to that in the CRAFT corpus.
We developed two cross connected architectures: 1) Tag to Ontology Embedding ( − > ) and 2) Ontology Embedding to Tag (− > ). Here we test if connecting the tag and ontology embedding architectures causing one to inform the prediction of the other would result in improved accuracy and if the direction of the connection matters. The − > architecture (Figure 2) has two diferent outputs, tags/ontology ids and ontology embeddings. The tag output ( in Figure 2) is concatenated with the output of the main Bi-GRU layer to give a higher dimensional vector output. The concatenation is then passed through dense layers to further learn the hierarchical representations of the ontology before generating an ontology embedding for each input token. This predicted ontology embedding is compared with the ground truth ontology embeddings learned using Node2Vec. Using cosine similarity as the loss function, loss is calculated and the gradients are backpropagated to adjust the model’s weight for convergence.
In − > , the ontology embedding output ( is concatenated with the output of the main Bi-GRU layer to give a higher dimensional vector output. The concatenation is then passed through dense layers before generating a tag for each input token. This predicted tag is compared with the ground truth tag in CRAFT. The loss is calculated and the gradients are backpropagated to adjust the model’s weight for convergence. The − > architecture can be depicted by switching the and blocks as well as the two outputs in Figure 2.
Figure 3 presents an explanation of the − > cross-connected architecture on three example tokens. Cross connected architectures difer from the baseline architectures by producing both tags and ontology embeddings instead of one or the other. Here, we show that the training/ inference is done on a sequence of tokens “vesicle”, “formation”, and “in” (which are parts of a sentence in the CRAFT corpus) as it is evaluated by the network. Each token is preprocessed to obtain the representative tensors – , ℎ , which are passed through embedding layers learned from CRAFT. The embedding of ℎ is also passed via a Bi-GRU layer. All of the resulting values are concatenated to be processed via the main Bi-GRU layer. The output from ‘Tag Dense Layer’ is concatenated with the output of main ‘Bi-GRU layer’ and passed as input to the ‘Ontology Embedding Dense Layer’ where the model generates ontology embeddings for each of the input tokens.
The final architecture ( − > − > ) explores if ontology embeddings can be improved iteratively by connecting a preliminary ontology embedding output to the tag output enabling improvements to the tag prediction. This predicted tag block is connected back to the ontology embedding block to urge further learning.
We evaluate our architectures using a modified F1 score and semantic similarity [
Since the majority of tags in the training corpus are non-annotations, the model predicts them with great accuracy. In order to avoid biasing the F1 score, we omit accurate predictions of non-annotations and focus instead on annotations only report a relatively conservative modified F1 score.
The CRAFT v4.0.1 dataset contains 18689 annotations pertaining to 974 concepts from the three GO sub-ontologies across 97 articles. Table 1 provides further information of the coverage of GO terms in CRAFT.
The baseline tag-only architecture ( ) resulted in a 0.80 F1 and a 0.83 semantic similarity score. The baseline ontology embeddings only architecture ( ) resulted in a 0.65 F1 and a 0.74 semantic similarity.
Among the two cross-connected architectures, we found that the Tag to Ontology Embedding architecture ( − > ) substantially outperformed the − > architecture according to F1 and was able to achieve similar performance as measured by semantic similarity. This indicates that − > is better at generating exactly matching predictions resulting in high F1 and semantic similarity. In contrast, − > performs better are generating semantically similar matches rather than exact matches leading to lower F1 than semantic similarity scores.
The − > architecture was able to improve upon ’s prediction of ontology embeddings by 23% (F1) and 9.4% (semantic similarity). We observed relatively modest improvements to ’s tag prediction with 3.8% (F1) and 1.2% (semantic similarity).
Connecting ontology embedding output to the tag output (− > ) either did not improve on the embedding prediction (F1) or resulted in a slight improvement (semantic similarity). − > did produce improvements for tag prediction over the model by 3.7% (F1) and 1.2% (semantic similarity). The multi-connected architecture did poorly in comparison to the cross-connected architectures.
Overall, the results suggest that architectures that use ontology embeddings only without learning associations between text and annotations perform poorly. The other takeaway is that connecting tag predictions to the ontology embedding block ( − > ) and letting embedding prediction learn from the predicted tag iteratively results in more robust architectures. The − > cross-connected architecture results in improved performance in predicting both tags and ontology embeddings across both metrics.
This work is funded by a CAREER grant to Manda from the Division of Biological Infrastructure at the National Science Foundation of United States of America (#1942727).