We introduce the Collaborative Oriented Relation Extraction (CORE) system for Knowledge Base Construction, based on the combination of Relation Extraction (RE) methods and domain experts feedback. CORE features a seamless, transparent, and modular architecture that suits large-scale processing. Via active learning, the CORE system bootstraps Knowledge Bases (KBs) and then employs RE methods to scale to large text corpora. We employ CORE to build one of the largest KBs focusing on fine-grained gene expression- cancer associations, fundamental to complement and validate experimental data for precision medicine and cancer research. We conducted comprehensive experiments showing the robustness of the approach and highlighting the scalability of CORE to large text corpora with limited manual annotations.
In 2020 there were about 19.2 million cancer cases worldwide and the World Health
Organization estimates a 33% overall increase by 2040.1 With this growing global
burden, cancer prevention is one of the century’s most pressing public health challenges,
and data-driven research is crucial in assisting the development of medical solutions
to address it. In this regard, microarray and next-generation sequencing technologies
providing raw data about gene expression-cancer interactions [
Thus, alternative solutions have been proposed to address this limitation, such as distant
supervision [
To show the robustness of the proposed approach, we conducted extensive analyses that
highlight how CORE scales to large text corpora with little human annotations. Moreover,
to evaluate the system efectiveness against the state-of-the-art, we performed a knowledge
base completion task showing that CORE achieves top performances. The KB derived
by CORE storing fine-grained facts about gene expression-cancer associations is available
at https://zenodo.org/records/7577127. The KB can also be accessed via CoreKB [
The rest of the article is as follows: Section 2 reports on related work; Section 3 outlines the CORE system; Section 4 presents the experiments; Section 5 concludes the paper.
To date, there are a handful of knowledge resources containing data about gene
expressioncancer associations [
Beside resources based on experimental studies, there also exist a few literature-based
resources [
Hence, there is a need for KBC systems that can scale to large text corpora and stay up to date while generating fine-grained information about gene expression-cancer associations. These fine-grained associations are essential to complement and validate experimental data, fundamental for advancing cancer research.
Let us consider a directed graph = ( , ) , where ⊆ {( 1, 2) | ( 1, 2) ∈ × }
is the set of edges connecting ordered pairs of vertices. Given an edge = ( 1, 2) ∈ , we call 1 the source vertex and 2 the target vertex. In our context, the nodes of are entities and the edges are the relationships between them.
entities. An aspect has a name and a domain dom= { 1 , … , }, where ∈ is the ℎ aspect value of . Given an aspect , the function () = dom returns its domain.
When it is clear from the context, the aspect value ∈ is simply referred to as . Example 1. Let us consider the context of gene-cancer associations, where there are three aspects describing a possible relationship ( ) between gene ( 1) and cancer ( 2): the Change of Gene Expression (CGE), the Change of Cancer Status (CCS), and the GeneCancer Interaction (GCI). Following Definition
of the aspects with the following domains: ( CGE) = {up, down, notinf}, (
CCS) = {progression, regression, notinf}, and (
GCI) = {causality, correlation, notinf}. A detailed
description of these aspect domains can be found in the original paper [
Definition 2 (Multi-Aspect Relationship).
Given a graph ( , ) and a set of aspects = {
}=1 , then a tuple of aspect values ( 1 , … , ) associated with = ( 1, 2) ∈ a multi-aspect relationship to a type from , called the signature set.
Definition 3 (Signature Function).
Given a set of aspects = { we define s ∶ ∏=1 → ⊆ Σ ∗; s(( 1 , ..., )) ↦ type as the signature function that maps }=1 and an alphabet Σ,
The signature function defines a set of mapping rules depending on the domain of interest. In our setting, we refer to the mapping rules described in Table 1. That is, we use the signature function to map multi-aspect gene expression-cancer relationships to gene prospective roles in cancer. Gene roles allow to distinguish the genes that are responsible for oncogenesis from those that are not; these are essential information for efective for cancer research and therapy design [ 29].
Definition 4 (Tagging Function). Given an edge ∈ and the signature set . We define ∶ → ; () ↦ type as the function tagging an edge with a signature type.
The tagging function works on the graph and associates a signature type to an edge. Thus, we use it to label edges with gene prospective roles. In other words, the graph represents gene expression-cancer associations as gene prospective roles in cancer. Overview. The goal of the CORE system is to harvest facts from text corpora to populate KBs. We model a KB as a directed graph made up of entities connected by typed relationships. Facts (or statements) are ( 1, , 2) triples, where 1, 2 ∈ , = ( 1, 2) ∈ , and () ∈ .
To obtain facts, CORE collects scientific literature from diferent sources, identifies sentences containing pairs of entities relevant to the considered task, and extracts aspects from them. Depending on the combination of extracted aspect values, a sentence expresses a specific signature type. Note that, for a given pair of entities, diferent sentences can express various signature types, as we show in the next example.
Example 2. Let us consider the following sentences taken from the biomedical literature: A. Colorectal cancer (CRC) growth and progression is frequently driven by RAS pathway activation through upstream growth factor receptor activation or through mutational activation of KRAS or BRAF.
B. Somatic mutations of the BRAF gene, causing constitutive activation of BRAF, have been found in various types of human cancers such as malignant melanoma, and colorectal cancer.
In both sentences, the following entities are extracted 1 = BRAF and 2 = Colorectal Cancer. Considering the aspects introduced in Example 1, for sentence A we find CGE = up, CCS = progression, and GCI = causality, leading to the signature type s((up, progression, causality)) = oncogene. On the other hand, the aspect values of sentence bootstrapping (orange), deployment (blue), and active learning (purple).
B are CGE = up, CCS = progression, and GCI = correlation, leading to the signature type s((up, progression, correlation)) = biomarker.
From Example 2, we see that diferent sentences may lead to diferent signature types. In the scientific discourse, it is not surprising that there are diferent viewpoints and that various studies can lead to diferent conclusions – even in contradiction with each other. Hence, we need to consider this potential uncertainty when facts are extracted from the literature. The CORE system models this inherent uncertainty by assigning the likelihood of being true to each aspect value. This probability is based on the evidence we can extract from the literature. Given a set of sentences concerning the same two entities, the more an aspect value is consistent in the set, the higher the probability for that value to be true. Hence, we define the concepts of Aspect-Probability Set and Multi-Aspect Function.
value carries a likelihood Pr( ), we call = {( , Pr( ))}=1 its aspect-probability set.
Given an aspect = { }=1 such that each aspect returns the -tuple of aspect-probability sets.
Definition 6 (Multi-Aspect Function).
Let = ( , ) be a directed graph and = { ({( 1 , Pr( 1 ))}
}=1 a set of aspect-probability sets. We define ∶ → ∏=1 ; () ↦ |=11|, … , {( , Pr( ))}
|= 1 |) as the multi-aspect function that, given an edge,
Thus, for each pair of target entities, CORE computes the probabilities for all the aspect values and combines them into tuples of aspect-probability sets – which represent a probability distribution over multi-aspect relationships. In this way, sentences serve as supporting or contradicting evidence that strengthens or weakens the likelihood of a fact. Furthermore, aspect-probability sets drive another essential aspect of CORE: the data-driven, active learning approach used to bootstrap KBs. That is, through reliability tests based on aspect value likelihoods and inference rules, the system tags facts as reliable or unreliable. Part of the sentences associated with the most “highly” unreliable facts is then fed to a human-in-the-loop process that reinforces the RE methods for aspect extraction.
Figure 1 presents the system architecture. In the first module (module 1), the texts acquired from the literature are processed and normalized to obtain sentences, from which a NERD component extracts entity pairs. The entity-annotated sentences undergo two diferent processes: bootstrapping (orange workflow) and deployment (blue workflow). In the bootstrapping workflow, experts manually annotate multi-aspect relationships between the entities (module 2), producing a set of relation-annotated sentences.
The manual, relation-annotated sentences are then used to train RE methods (module 3) and to populate the KB (module 5). The RE methods are trained to predict the diferent aspects of multi-aspect relationships. Once trained, RE methods are employed in the deployment workflow to obtain automatic annotations expressing multi-aspect relationships between entities (module 4). Then, automatic, relation-annotated sentences are used to further populate the KB (module 5).
In the last module (module 5), relation-annotated sentences are grouped by entity pairs and used to generate facts. First, a knowledge enrichment component computes probabilities for all the aspect values and combines them into tuples of aspect-probability sets. Then, a reliability testing component uses these probabilities to perform multiple tests that tag facts as either reliable or unreliable. Only facts tagged as reliable are used to populate the KB.
When the deployment workflow is complete, unreliable facts are ranked by ascending reliability score and the top- automatically annotated sentences associated with them are re-annotated by experts – thus triggering an active learning process that reinforces the RE methods (purple workflow).
Versioning. The active learning workflow makes CORE suited to iterative KB versioning. We define a KB version as the graph = ( , ) obtained after the ℎ iteration of the bootstrap and deployment workflows. Once the ℎ version of the KB has been deployed, the active learning workflow starts by generating the batch of unreliable sentences for bootstrapping the ℎ + 1 version of the KB. The unreliable sentences are manually annotated and used to increase the size of the datasets to re-train the RE methods from scratch, which then generate a new set of automatic annotations to be included in the ℎ + 1 KB version. When the bootstrap and deployment workflows end, the ℎ + 1 version of the KB is re-built from scratch and comprises all the available annotations.
Knowledge Base Construction. We use diferent resources to build the KB, which
increase with each subsequent iteration of the KB construction process. The considered
resources are CoMAGC [
First, we can see that the ratio between the sentences stored in the KB (Table 3) and
the input ones (Table 2) decreases at each iteration. From the first iteration, where
the CORE system uses the 62% of the input sentences to build KB0, we move to the
52% to build KB1, 26% for KB2, and only 14% for KB3. Such a decrease reflects the
use of reliability tests and active learning, which make the system more selective and
accurate. In particular, active learning leads the CORE system to refine the RE methods
at each iteration, thus reducing false positives as well as unreliable facts. Secondly, the
large number of diferent genes and cancers in KB3 highlights the scalability of the
approach. In this regard, KB3 contains 21, 005 genes, which cover the 70% of the 30, 000
estimated genes in the human genome.3 On the other hand, through the integration
of DisGeNET data, KBs 1–3 contain most of the (known) cancer types involved in
gene expression-cancer associations. Combined, this large number of genes and cancer
types leads to more than 150, 000 reliable facts. Finally, compared to currently available
knowledge resources [
Knowledge Base Completion. We evaluate the efectiveness of the CORE system on a
KB completion task, in which we hold out a portion of an existing KB with associated
sentences and we assess CORE ability to recover it. To this end, we hold out from
3https://www.genome.gov/human-genome-project/
BioXpress [
For BioXpress completion, we use the three versions of the CORE system obtained after each (re-)training of the RE methods. Table 4 reports the CORE system performances on the BioXpress completion task after each (re-)training of the RE methods, as well as DEXTER performance on KB3.
We can see that each CORE version consistently achieves performances above 0.95 for each measure. In particular, CORE1 improves over CORE0 by about 2% and reaches a performance plateau, as shown by CORE2 performance. The results highlight the efectiveness of the CORE system in recovering BioXpress using a limited amount of manual annotations to train the RE methods. On the other hand, the poor performance of DEXTER on KB3 highlights a lack of flexibility that hampers its applicability to heterogeneous sentences. To further support this intuition, we observe that between precision and recall it is recall to have the worst performance, with a value of 0.3256. This underlines the expert system nature of DEXTER which, although precise, fails to generalize beyond its set of predefined patterns.
In this work we presented CORE, a KBC system based on the combination between automated RE methods and domain experts. The reliability tests and the active learning process make the system suited to iterative KB versioning. We used the CORE to build one of the largest KBs about gene expression-cancer associations. We conducted extensive experiments that (i) highlighted the ability of CORE to scale to large collections of heterogeneous data with limited human annotations and (ii) showed its generalizability and reliability compared to the current state-of-the-art.
Acknowledgments. This work is partially supported by the HEREDITARY Project, as part of the European Union’s Horizon Europe research and innovation programme under grant agreement No GA 101137074. with literature evidence, Nucleic Acids Res. 42 (2014) 416–421. [27] J. P. González, J. M. Ramírez-Anguita, J. Saüch-Pitarch, F. Ronzano, E. Centeno, F. Sanz, L. I. Furlong, The DisGeNET knowledge platform for disease genomics: 2019 update, Nucleic Acids Res. 48 (2020) D845–D855. [28] M. Bundschus, A. Bauer-Mehren, V. Tresp, L. I. Furlong, H. P. Kriegel, Digging for knowledge with information extraction: a case study on human gene-disease associations, in: Proc. of the 19th ACM Conference on Information and Knowledge Management, CIKM 2010, Toronto, Ontario, Canada, October 26-30, 2010, ACM, 2010, pp. 1845–1848. [29] D. Haber, J. Settleman, Cancer: Drivers and passengers, Nature 446 (2007) 145–146.