Knowledge acquisition is the essential first step of any Knowledge Graph (KG) application. This knowledge can be extracted from a given corpus (KG generation process) or specified from an existing KG (KG specification process). Focusing on domain specific solutions, knowledge acquisition is a labor intensive task usually orchestrated and supervised by subject matter experts. Specifically, the domain of interest is usually manually defined and then the needed generation or extraction tools are utilized to produce the KG. Herein, we propose a supervised machine learning method, based on Transformers, for domain definition of a corpus. We argue why such automated definition of the domain's structure is beneficial both in terms of construction time and quality of the generated graph. The proposed method is extensively validated on three public datasets (WebNLG, NYT and DocRED) by comparing it with two reference methods based on CNNs and RNNs models. The evaluation shows the efficiency of our model in this task. Focusing on scientific document understanding, we present a new health domain dataset based on publications extracted from PubMed and we successfully utilize our method on this. Lastly, we demonstrate how this work lays the foundation for fully automated and unsupervised KG generation.
Knowledge Graphs (KGs) are among the most
popular data management paradigms and their application is
widespread across different fields, e.g., recommendation
systems, question-answering tools and knowledge discovery
applications. This is due to the fact that KGs share
simultaneously several advantages of databases (information
retrieval via structured queries), graphs (representing loosely
or irregularly structured data) and knowledge bases
(representing semantic relationship among the data). KG research
can be divided in two main streams
The knowledge acquisition process is either referred to
KG construction, where the KG is built from scratch
using a specific corpus, or KG specification, where a subgraph
of interest is extracted from an existing KG. In both cases,
the acquisition process can follow a bottom-up or top-down
approach
Herein, we focus on domain-specific, i.e., top-down,
acquisition for two main reasons. Firstly, the acquisition
process can be faster and more accurate in this way. By
specifying the schema of the domain of interest, then we only need
to select the proper and needed tools (i.e. pretrained
models) for the actual entity and relation extraction. Secondly,
such approach minimizes the presence of irrelevant data and
restricts queries and graph operations to a carefully tailored
KG. This generally improves the accuracy of KG
applications
The domain definition is usually performed by subject matter experts. Yet, knowledge acquisition by expert curation can be extremely slow as the process is essentially manual. Moreover, human error may affect the data quality and lead to malformed KGs. In this work, we propose to overcome these issues by introducing an automated machine learning-based approach to understand the domain of a collection of text snippets. Specifically, given sample input texts, we infer the schema of the domain to which they (a) Bottom-up pipeline. (b) Top-down pipeline. belong. This task can be incorporated into both domainspecific KG generation and KG specification process, where the domain definition is the essential first step. For the KG generation, the input texts can be samples from the corpus of interest, while for the KG specification, these text snippets can express possible questions that need to be answered from the specified KG. We introduce a seq2seq-based model relying on transformer architecture to infer the relation types characterizing the domain of interest. Such model lets us to define the domain’s schema including all the needed entity and relation types. The model can be trained using any available previous schema (i.e., schema of a general KG like DBpedia) and respective text examples for each possible relation type. We show that our proposed model outperforms other baseline approaches, it can be successfully utilized for scientific documents and it has interesting potential extension in the field of automated KG generation.
At the best of our knowledge, our method is the first attempt
to introduce a supervised machine learning based domain
understanding tool that can be incorporated into
domainspecific KG generation and specification pipelines.
Currently, the main research interest related to KG generation
workflows is associated with attempts to improve the named
entity recognition (NER) and the relation extraction tasks or
provide end-to-end pipelines for general or domain-specific
KG generation
As it concerns the KG specification field, the subgraph
extraction is usually based on graph traversals or more
sophisticated heuristics techniques and some providing initial
entities or entity types
The relation extraction task is also related to our work.
It aims at the extraction of triplets of the form of (subject,
relation, object) from the texts. The neural network based
methods, such as Nguyen and Grishman; Zhou et al.; Zhang
et al., dominate the field. These methods are CNN
While our work is linked to relation extraction it has two major differences. Firstly, we focus on the relation type and the entity types that compose a relation rather than the actual triplet. Secondly, the training process differs and requires coarser annotations. We solely provide texts and the respective existing sequence of relation types. Contrarily in a typical relation extraction training process, information about the position of the entities in the text is also needed. Here, we propose to improve knowledge acquisition by performing a data-driven domain definition providing an approach that is currently unexplored in KG research.
The domain understanding task attempts to uncover the structured knowledge underlying a dataset. In order to depict this structure we can leverage the so called domain’s metagraph. A domain’s metagraph is a graph that has as vertices all the entity types and as edges all their connections/relations in the context of this domain. The generation of such a metagraph entails obtaining all the entity types and their relations. Assuming that each of entity types that are presented in the domain has at least one interaction with another entity type, the metagraph of this domain can be produced by inferring all the possible relation types as all the entity types are included in at least one of them. Thus, our approach aims to build an accurate model to detect a domain’s relation types, and leverages this model to extract those relations from a given corpus. Aggregating all extracted relations yields the domain’s metagraph.
Sequence to sequence models (seq2seq)
We consider the domain’s relation type extraction task as a specific version of machine translation from the language of the corpus to the “relation” language that includes all the different relations between the entity types of the domain. A relation type R which connects the entity type i to j is represented as “i.R.j” in the “relation” language. In the case of undirected connections, “i.R.j” is the same as “j.R.i” and for simplicity we can discard one of them.
Seq2seq models have been designed to address tasks
where both the input and the output sequences are ordered.
In our case the target “relation” language does not have any
defined ordering as per definition the edges of a graph do
not have any ordering. In theory the order does not
matter, yet in practice unordered sequences will lead to slower
convergence of the model and requirements of more
training data to achieve our goal
According to
Based on the given training set, we consider that the model is aware of a general domain anatomy, i.e., the sets of possible entity types and relation types are known, and we would like to identify which of them are depicted in a given corpus. In both cases of EntityPairOverlap and SingleEntityOverlap type text snippets, there is one main entity type from which all the other entity types can be found by performing only one hop traversal in the general domain’s metagraph. The class of Normal text snippets is a broader case in which one can identify heterogeneous connectivity patterns among the entity types represented. Yet, a sentence typically describes facts that are expected to be connected somehow, thus the entity types included in such texts usually are not more than 1 or 2 hops away from each other in the general metagraph. In the light of the considerations above, we propose to sort the relations in a breadthfirst-search (BFS) order starting from a specific node (entity type) in the general metagraph. In this way, we confine the output in a much lower dimensional space by adhering to a semantically meaningful order.
Inspired by state-of-the-art approaches in the field of neural machine translation, our model architecture is a multi-layer bidirectional Transformer. We follow the lead of Vaswani et al. in implementing the architecture, with the only difference that we adopt a learned positional encoding instead of a static one (see Appendix for further details on the positional encoding). As the overall architecture of the encoder and the decoder are otherwise the same as in Vaswani et al., we omit an in-depth description of the Transfomer model and refer readers to original paper.
To boost the model’s performance, we also propose an ensemble approach exploiting different Transformers and aggregating their results to construct the domain’s metagraph. Each of the Transformers differs in the selected ordering of the “relation” vocabulary. The selection of different starting entity type for the breadth-first-search will lead to different orderings. We expect that multiple orderings could facilitate the prediction of different connection patterns that can not be easily detected using a single ordering. The sequence of steps for an ensemble domain understanding is the following: Firstly, train k Transformers using different orderings. Secondly, given a set of text snippets, predict sequences of relations using all the Transformers. Finally, use late fusion to aggregate the results and form the final predictions.
It is worth mentioning that in the last step, we omit the underlying ordering that we follow in each model and we perform a relation-based aggregation. We examine each relation
WebNLG
NYT
DocRED
PubMed-DU
separately in order to include it or not in the final metagraph.
For the aggregation step, we use the standard Wisdom of
Crowds (WOC)
We evaluate our Transformer-based approach against three
baselines on a selection of datasets representing different
domains. As baselines, we use CNN and RNN based methods
influenced by
To our knowledge, there is no standard dataset available
for the relation type extraction task in the literature.
However there is a plethora of published datasets for the
standard task of relation extraction that can be utilized for our
case with limited effort. For our task, the leveraged datasets
should contain tuples of texts and their respective sets of
relation types. We use WebNLG
To emphasize the application of such model in the
scientific document understanding, we produce a new
taskspecific dataset called PubMed-DU related to the general
health domain. We download paper abstracts from PubMed
focusing on work related to 4 specific health subdomains:
Covid-19, mental health, breast cancer and coronary heart
disease. We split the abstracts into sentences. The entities
and their types for each text have been extracted using
PubTator
For all datasets, we use the same model parameters.
Specifically, we use Adam
The evaluation of the models is performed both at instance
and graph level. In the instance level, we examine the ability
of the model to predict the relation types that exist in a given
text. To investigate this, we use F1-score and accuracy.
F1score is the harmonic mean of model’s precision and recall.
Accuracy is computed at an instance level and it measures
for how many of the testing texts, the model manage to infer
correctly the whole set of their relation types. For the
metagraph level evaluation, we use our model to predict the
metagraph of a domain and we examine how close to the actual
metagraph is. For this comparison, we utilize F1-score for
both edges and nodes of the metagraph as well as the
similarity of the distribution of the degree and eigenvector
centrality
CNN
Transformer - unordered Transformer - BFSrecord label
Transformer - WOC k=20
CNN
Transformer - WOC k=8
CNN
Transformer - WOC k=6
CNN
To study the performance of our model, we perform 10 independent runs each with different random splitting of the datasets into training, validation and testing set. Table 2 depicts the median value and the standard error of the baselines and our method for the two metrics. Our method is better in terms of accuracy for all the four datasets and in terms of F1score for the WebNLG and DocRED datasets. For the NYT and PubMed-DU datasets, the F1-score of CNN and RNN models outperform our approach. We observed that the baseline models profit from the fact that, in these datasets, the majority of the instances depict only one relation and many of the relations appear in a limited number of instances. In general, there is lack of sequences of relations that hinders the Transformer’s ability to learn the underlying distribution in these two cases(see Appendix). Lastly, the decreased performances of all the models in the DocRED dataset is due to the long tail characteristic that this dataset shows as 66% of the relations appeared in no more than 50 instances (see Appendix).
The above comparisons focus only on the ability of the model to predict the relation types given a text snippet. Since our ultimate goal is to infer the domain’s metagraph from a given corpus, we divide the testing sets of the datasets into small corpora and we attempt to define their domain using our model. For WebNLG, NYT and DocRED dataset, 10 artificial corpora and their respective domains have been created by selecting randomly 10 instances from each of the testing sets. We set two constraints into this selection to assure that the produced metagraphs are meaningful. Firstly, each subdomain should have a connected metagraph and secondly each existing relation type is appeared at least two times in the provided instances. For the PubMed-DU dataset, we already know the existence of 4 subdomains in it, so we focus on the inference of them. For each subdomain, we select randomly 100 instances from the testing set that belong to this subdomain and we attempt to produce the domain based on them. We infer the relation types for each instance and then we generate the domain’s metagraph by including all the relation types that were found in the instances. Then, we compare how close the actual domain’s metagraph and the predicted metagraph are.
Table 3 presents the results of the evaluation of the predicted versus the actual domain’s metagraph for 10 subdomains extracted from the testing set of the WebNLG, NYT and DocRED datasets. All the presented values for these datasets are the mean over all the 10 subdomains. For the PubMed-DU dataset, we include only the Covid-19 subdomain case. Results for the remaining subdomains of this dataset can be found in the Appendix. Our approach using Transformer + BFS based ordering outperforms or is close to the baselines for all cases in terms of edges and nodes F1score. Furthermore, the degree and eigenvector centralities distribution of the generated metagraphs using our method are closer to the groundtruth in comparison to other methods in all cases. This indicates that the graphs produced with our method are both element-wise and structurally closer to the actual ones. More detailed comparisons of the different methods at both instance and metagraph level have been included in the Appendix.
Transformer - BFSrecord label
Transformer - WOC k=5
CNN
RNN (Zhou et al. 2016)* Transformer - unordered Transformer - BFSperson
Transformer - WOC k=8
CNN
RNN (Zhou et al. 2016)* Transformer - unordered
Transformer - BFSPER
Transformer - WOC k=6
CNN
RNN (Zhou et al. 2016)* Transformer - unordered Transformer - BFSChemical Transformer - WOC k=5 0.9747 0.9639 0.9598 0.9806 0.9808 0.9059 0.9205 0.8184 0.8806 0.8672 0.4819 0.6823 0.7530 0.7830 0.8045 0.9140 0.9736 0.9631 0.9583 0.9789
The ensemble variant of our approach, based on the WOC consensus strategy, outperforms the simple Transformer + BFS ordering in all cases. Based on the evaluation at both instance and metagraph level, our ensemble variant seems to be the most reliable approach for the task of domain’s relation type extraction as it achieves some of the best scores for any dataset and metric.
The proposed domain understanding method enables the
inference of the domain of interest and its components. This
enables a partial automation and a speed up of the KG
generation process as, without manual intervention, we are able
to identify the metagraph, and inherently the needed
models for the entity and relation extraction in the context of the
domain of interest. To achieve this, we adopt a
Transformerbased approach that relies heavily on attention mechanisms.
Recent efforts are focusing on the analysis of such
attention mechanisms to explain and interpret the predictions
and the quality of the models
Herein, we proposed a method to speed up the knowledge acquisition process of any domain specific KG application by defining the domain of interest in an automated manner. This is achieved by using a Transformer-based approach to estimate the metagraph representing the schema of the domain. Such schema can indicate the proper and needed tools for the actual entity and relation extraction. Thus our method can be considering as the stepping stone in any KG generation pipeline. The evaluation and the comparison over different datasets against state-of-the-art methods indicates that our approach produces accurately the metagraph. Especially, in datasets where text instances contain multiple relation types our model outperforms the baselines. This is an important observation as text describing multiple relations is the most common scenario. Based on that and relying on the capability of the transformers to catch longer dependencies, future investigation of how our model performs in larger pieces of texts, like full paragraphs, could be interesting and indicate a clearer advantage of our work. The needed definition of a general domain for the training phase might be a limitation of this method. However, schema and data from already existing KGs can be utilized for training purposes. Unsupervised or semi-supervised extension of this work can also be explored in the future to mitigate the issue.
Our work paves the way towards an automated
knowledge acquisition, as our model minimizes the need of
human intervention in the process. So in the near future the
currently needed manual curation can be avoided and lead
to faster and more accurate knowledge acquisition.
Interestingly, using the PubMed-DU dataset, we underline that our
method can be utilized for scientific documents. The
inference of their domain can assist both in their general
understanding but also lead to more robust knowledge acquisition
from them. As a side effect, it is also important to notice that,
such attention-based model can be directly applied to triplet
extraction from the text without retraining and without
supervision. Triplet extraction in an unsupervised way
represents a breakthrough, especially if combined with most
recent advances in zero-shot learning for NER
In our model, we adopt a learned positional encoding instead of a static one. Specifically, the tokens are passed through a standard embedding layer as a first step in the encoder. The model has no recurrent layers and therefore it has no idea about the order of the tokens within the sequence. To overcome this, we utilize a second embedding layer called a positional embedding layer. This is a standard embedding layer where the input is not the token itself but the position of the token within the sequence, starting with the first token, the <sos> (start of sequence) token, in position 0. The position embedding has a ”vocabulary” size equal to the maximum length of the input sequence. The token embedding and positional embedding are element-wise summed together to get the final token embedding which contains information about both the token and its position within the sequence. This final token embedding is then provided as input in the stack of attention layers of the encoder.
For a better understanding of the datasets, we analyzed the distribution of occurrences for all relation types. These distributions are depicted in Figure 4. A percentage of relation types with number of appearances close or less to 10 is observed for all datasets. The lack of many examples can pose problems in the learning process for these specific relation types. This is highlighted especially in the DocRED case, as we attributed the decreased performances of all the models in this dataset in its long tail characteristic that it holds. Especially for the DocRED, almost the 50% of the relations appeared in no more than 10 instances and the 66% of the relations appeared in no more than 50 instances (4).
The Jensen-Shannon divergence metric between two probability vectors p and q is defined as: r D(p k m) + D(q k m)
2 where m is the pointwise mean of p and q and D is the Kullback-Leibler divergence.
The Kullback-Leibler divergence for two probability vectors p and q of length n is defined as:
D(p k q) = n X pilog2( pqi )
i i=1
The Jensen–Shannon metric is bounded by 1, given that we use the base 2 logarithm.
In this section, the procedure of automated triplets extraction based on the predicted relation types and the respective attention weights is described . We generate an undirected graph that connects the tokens of the sentence based on their syntax dependencies for each instance. Then for each different predicted relation type, we define the final attention weights of a token based on the attention weights of itself and its neighbors in the syntax dependencies graph. Let ar be the attention vector of a predefined model’s attention head, which contains all the attention weights related to the relation type r. The final attention weight w of the token i for the relation r is defined as: wir = 2 air +
X ajr j2neigi where neigi is the set containing all the neighbors of i in the syntax dependencies graph. Then for each noun chunk k (nk) of the text we compute its total attention weight for the relation type r as: nrk =
X f (wjr) j2nck where nck is the set of tokens which belong to the nk and f is a function defined as f (wjr) = wr
j 2 wr j if j is stop-word if j is not stop-word
Finally, we extract as entities which are connected via the relation type r the two noun chunks with the highest weight nr. As this work is a proof of concept rather than an actual method, the selection of the attention head is based on whichever gives as the best outcome. Yet, in actual scenarios it is recommended the use of a training set, based on which the optimal head will be identified. For the creation of the syntax dependencies graph and the extraction of the noun chunks of the text we use spacy1 and its en core web lg pretrained model. Table 4 includes all the texts that have used in the proof of concept that is presented in the main paper and the respective predicted triplets for each of them.
Table 5 presents a detailed evaluation of our approach and the baselines models. We have included the 3 best BFS ordering variants (in terms of accuracy) and 3 consensus variants. To cover the range of all the available values of k, [1, number of entity types], we select a case with just a few Transformers, one with a value close to half of the total number of entity types and one close to the total number of entity types. For each different k value, we utilize the top k best orderings based on their accuracy. In addition to the per instance accuracy and the per relation F1-score, the table also includes the per relation precision and the recall of each model. Our proposed method, especially its ensemble variant, produces the best outcome in all datasets apart from the NYT case where the CNN and RNN models manage to be more precise. This is attributed to the characteristics of NYT dataset, where there are many one-only relation instances.
Similarly, in tables 6 and 7 we perform an in-depth metagraph level evaluation of the models. For all three datasets, our proposed method and its ensemble extension produce the best or one of the top-3 best outcomes.
(a) WebNLG (b) NYT (c) DocRED (d) PubMed-DU
Figure 4: Appearances distribution for the relation types of all utilized datasets. Text acute deep vein thrombosis in covid-19 hospitalized patients. (ace-2) receptor for its attachment similar to sars-cov-1, which is followed by priming of spike protein by transmembrane protease serine 2 (tmprss2) which can be targeted by a proven inhibitor of tmprss2, camostat. temporal trends in decompensated heart failure and outcomes during covid-19: a multisite report from heart failure referral centres in london. prevalence, risk factors and clinical correlates of depression in quarantined population during the covid-19 outbreak. tocilizumab plus glucocorticoids in severe and critically covid-19 patients. effects of progressive muscle relaxation on anxiety and sleep quality in patients with covid-19. special section: covid-19 among people living with hiv. ace2 and tmprss2 variants and expression as candidates to sex and country differences in covid-19 severity in italy. did the covid-19 pandemic cause a delay in the diagnosis of acute appendicitis? practice observed in managing gynaecological problems in post-menopausal women during the covid-19 pandemic. hypogammaglobulinemia causing pneumonia: an overlooked curable entity in the chaotic covid-19 pandemic. chemokine receptor gene polymorphisms and covid-19: could knowledge gained from hiv/aids be important? serotonin syndrome in two covid-19 patients treated with lopinavir/ritonavir. mortality rate and predictors of mortality in hospitalized covid-19 patients with diabetes. in conclusion, self-reported depression occurred at an early stage in convalescent covid-19 patients, and changes in immune function were apparent during short-term follow-up of these patients after discharge. gb-2 inhibits ace2 and tmprss2 expression: in vivo and in vitro studies. preemptive interleukin-6 blockade in patients with covid-19. response to: ’clinical course of covid-19 in patients with systemic lupus erythematosus under long-term treatment with hydroxychloroquine’ by carbillon et al. obsessive-compulsive disorder during the covid-19 pandemic repeated monitoring of ferritin, interleukin-6, c-reactive protein, lactic acid dehydrogenase, and erythrocyte sedimentation rate during covid-19 treatment may assist the prediction of disease severity and evaluation of treatment effects. respiratory and pulmonary complications in head and neck cancer patients: evidence-based review for the covid-19 era. targeting the immune system for pulmonary inflammation and cardiovascular complications in covid-19 patients. risk of peripheral arterial thrombosis in covid-19. preadmission diabetes-specific risk factors for mortality in hospitalized patients with diabetes and coronavirus disease 2019.
Predicted triplets (thrombosis, Disease.to.Disease, COVID 19) (spike, Gene.to.Gene, Transmembrane protease serine 2) (heart failure, Disease.to.Disease, COVID-19) (depression, Disease.to.Disease, COVID-19) (Tocilizumab, Chemical.to.Species, patients) (anxiety, Disease.to.Disease, COVID-19) (people, Disease.to.Species, HIV) (ACE2, Gene.to.Gene, TMPRSS2) (COVID-19, Disease.to.Gene, TMPRSS2) (COVID-19, Disease.to.Disease, Acute Appendicitis) (COVID-19, Disease.to.Species, women) (hypogammaglobulinemia, Disease.to.Disease, COVID-19) (AIDS, Disease.to.Gene, Chemokine receptor) (lopinavir/ritonavir, Chemical.to.Species, patients) (COVID-19, Disease.to.Disease, Diabetes) (depression, Disease.to.Disease, COVID-19) (ACE2, Gene.to.Gene, TMPRSS2) (COVID-19, Disease.to.Gene, interleukin-6) (hydroxychloroquine, Chemical.to.Species, patients) (Obsessive-compulsive disorder, Disease.to.Disease, COVID-19) (COVID-19, Disease.to.Gene, interleukin-6) (head and neck cancer, Disease.to.Disease, COVID-19) (Cardiovascular Complications, Disease.to.Disease, COVID-19) (Thrombosis, Disease.to.Disease, COVID-19) (Diabetes, Disease.to.Disease, COVID-19)
NYT
CNN
RNN (Zhou et al. 2016)* 0.8517 0.0058 Transformer - unordered 0.8798 0.0053 Transformer - BFSoccupation 0.8987 0.0068 Transformer - BFSmusic genre 0.8983 0.0053 Transformer - BFSrecord label 0.9000 0.0046
Transformer - WOC k=5 0.9210 0.0017 Transformer - WOC k=20 0.9235 0.0014
Transformer - WOC k=45 0.9235 0.0002
CNN
RNN (Zhou et al. 2016)* 0.7520 0.0027 Transformer - unordered 0.7426 0.0061 Transformer - BFSlocation 0.7461 0.0053 Transformer - BFSperson 0.7491 0.0048 Transformer - BFScompany 0.7461 0.0081 Transformer - WOC k=4 0.7547 0.0038 Transformer - WOC k=8 0.7669 0.0011
Transformer - WOC k=12 0.7698 0.0007
CNN
RNN (Zhou et al. 2016)* 0.2178 0.0088 Transformer - unordered 0.4869 0.0069 Transformer - BFSLOC 0.5235 0.0049 Transformer - BFSPER 0.5234 0.0077 Transformer - BFSORG 0.5252 0.0048 Transfomer - WOC k=4 0.5678 0.0037 Transformer - WOC k=5 0.5697 0.0016
Transformer - WOC k=6 0.5722 0.0001
CNN
RNN (Zhou et al. 2016)* 0.5772 0.0041 Transformer - unordered 0.5693 0.0075 Transformer - BFSSpecies 0.5707 0.0060 Transformer - BFSChemical 0.5703 0.0048
Transformer - BFSGene 0.5718 0.0068 Transformer - WOC k=3 0.5909 0.0070 Transformer - WOC k=4 0.5685 0.0069 Transformer - WOC k=5 0.5960 0.0001
Precision WebNLG
NYT
DocRED
CNN
RNN (Zhou et al. 2016)* Transformer - unordered Transformer - BFSoccupation Transformer - BFSmusic genre Transformer - BFSrecord label
Transformer - WOC k=5 Transformer - WOC k=20
Transformer - WOC k=45
CNN
RNN (Zhou et al. 2016)* Transformer - unordered Transformer - BFSlocation Transformer - BFSperson Transformer - BFScompany Transformer - WOC k=4 Transformer - WOC k=8
Transformer - WOC k=12
CNN
RNN (Zhou et al. 2016)* Transformer - unordered Transformer - BFSLOC Transformer - BFSPER Transformer - BFSORG Transformer - WOC k=4 Transformer - WOC k=5 Transformer - WOC k=6 0.9879 0.9735 0.9775 0.9805 0.9746 0.9772 0.9772 0.9840 0.9916 0.9800
1 0.9800 0.9800 0.9666 0.9657 1 1 1 0.9019 0.9714 1 1 0.9777 0.9777 1 1 1
Transformer - BFSSpecies Transformer - BFSChemical
Transformer - BFSGene Transformer - WOC k=3 Transformer - WOC k=4
Transformer - WOC k=5
CNN
Transformer - BFSGene Transformer - WOC k=3 Transformer - WOC k=4
Transformer - WOC k=5
CNN
Transformer - BFSGene Transformer - WOC k=3 Transformer - WOC k=4
Transformer - WOC k=5
CNN
Transformer - BFSGene Transformer - WOC k=3 Transformer - WOC k=4 Transformer - WOC k=5 0.9140 0.9736 0.9631 0.9583 0.9583 0.9525 0.9531 0.9642 0.9789 0.9135 0.9730 0.9367 0.9531 0.9531 0.9379 0.9525 0.9584 0.9514 0.8902 0.9498 0.9484 0.9703 0.9703 0.9756 0.9644 0.9703 0.9644