The "RNA world" represents a novel frontier for the study of fundamental biological processes and human diseases and is paving the way for the development of new drugs tailored to the patient's biomolecular characteristics. Although scientific data about coding and non-coding RNA molecules are continuously produced and made available from public repositories, they are scattered across different databases and in the scientific literature. A centralized, uniform, and semantically consistent representation of the knowledge on RNA is still lacking. We have recently constructed RNA-KG [1], a knowledge graph integrating biological knowledge about RNA molecules with their functional relationships with genes, proteins, and chemicals and biomedical ontological concepts. RNA-KG includes around 600K nodes and 9M RDF triples representing reliable interactions involving RNA molecules and related biomedical concepts extracted from more than 50 public data sources according to 11 bio-ontologies. RNA-KG is coupled with a meta-graph representing all the possible interactions involving RNA molecules. SPIRES (Structured Prompt Interrogation and Recursive Extraction of Semantics) [2] is a recently proposed approach to information extraction that exploits Large Language Models (LLMs) [3] to identify instances of a knowledge schema expressed in terms of LinkML [4] starting from plain texts. By identifying and extracting relevant information from an input text, it adopts zeroshot learning to identify and extract relevant entities and relationships among them, which are then normalized and grounded through ontologies and vocabularies. SPIRES is a general-purpose approach that can be used across a variety of domains and does not require specific training/tuning on the considered domain. SPIRES adopts an engineering approach for creating prompts for interacting with an LLM (like GPT [5], Llama 2 [6], Mistral [7], and Zephyr [8]) to improve the quality of the generated responses [9]. In this way, technical challenges for generative AI (e.g., constructing comprehensive real-world knowledge and improving the accuracy of automated responses) can be addressed.
In this paper, we discuss the initial experimental results that we obtained by applying SPIRES in the extraction of interactions among bio-entities involving RNA molecules in the context of the PNRR project "Gene Therapy and Drugs based on RNA Technology". The purpose of the experiments is to show the level of accuracy of the system in extracting interactions from the scientific literature and investigate the possibility of combining RNA-KG with LLMs. Note that the extraction of interactions involving RNA molecules is particularly challenging for two reasons. First, a well-recognized ontology for characterizing non-coding RNA molecules is still lacking, and then different identifiers for representing the same bio-entity are adopted. Even if a more systematic evaluation should be conducted, the initial results are very encouraging.
The paper is structured as follows. Section 2 describes the SPIRES approach and related approaches that integrate LLMs with knowledge data. Section 3 presents the LinkML schema that we have developed for interacting with SPIRES. Section 4 describes the experimental results, while Section 5 reports concluding remarks.
The population of a KG by extracting triples from unstructured texts is an interesting research activity and the advent of LLMs has boosted the interpretation of highly technical languages as shown on question-answering benchmarks [10]. However, these techniques have shown different limitations, such as generating incorrect statements due to hallucinations [11] and insensitivity to negations [12], that cannot be tolerated in sensitive domains like precision medicine. SPIRES adopts: π) the knowledge schema of a specific domain for the generation of prompts for reducing these drawbacks; and ππ) bio-ontologies for enhancing the quality of the produced information. Figure 1 outlines the SPIRES workflow. SPIRES requires the specification of the knowledge schema expressed in LinkML [4] to guide the system in the extraction of knowledge. A LinkML schema contains the classes of entities and relationships among them within the specified domain. Classes can also include attributes (e.g., name, type, and list of synonyms) to enrich entity description. The LinkML schema is automatically processed to generate a list of prompts through which SPIRES interacts with a LLM (e.g., GPT3, GPT4, Llama 2, Mistral, and Zephyr). Each prompt of the list is submitted to the LLM for collecting information that is exploited for completing the following prompt by eventually considering the bio-ontologies (e.g., for changing a protein symbol with the corresponding identifier in an ontology). This refinement recursive process improves the quality of the information gathered through the LLM.
Protein with its properties and the adopted identification scheme (See Figure 1). A prompt is then generated for this class and used for extracting proteins. However, the result obtained by ChatGPT alone (in this case COX20) is not compliant with the Protein class structure. Therefore, SPIRES exploits bio-ontologies (e.g. PRotein Ontology -PRO [13]) to obtain an adequate result.
Furthermore, in case relationships are identified, SPIRES selectively retains only those aligned with the predefined schema that can be grounded to the Relations Ontology (RO [14]). By exploiting standard identification schemes adopted by the reference bio-ontologies, the system guarantees the generation of triples that can be easily integrated into a biomedical KG.
SPIRES thus creates and refines prompts to maximize the effectiveness of LLMs by exploiting domain knowledge encapsulated through the description of the classes and relationships that we wish to include in the KG.
As outlined in [9], the explicit and structured information contained in KGs can also be used for improving the knowledge awareness of LLMs. KGs have been used: π) in the training of the LLM [15,16]; ππ) during the inference stage for making available to the LLMs the latest knowledge without retraining [17]; πππ) to improve the interpretability of LLMs by explaining the facts [18] and by enhancing the reasoning process of LLMs [19]. One of the main disadvantages of solution π) is that the enhancement of the knowledge contained in the KG requires a retraining of the model which is a time (and money) consuming activity. For this reason, approaches of solution ππ) are gaining momentum because they allow the separation of the text space and the knowledge space. In this case, knowledge is injected at the time of inference.
For the creation of the schema needed for the application of SPIRES, we considered the RNA-KG meta-graph [20] that represents all the kinds of relationships involving RNA molecules in the considered data sources. Starting from it, a UML class diagram was developed that formally describes the schema of the considered domain and can be used for identifying meaningful relationships in the considered domain. Figure 2 shows an excerpt of the generated UML class diagram that consists of four biological and biomedical classes (miRNA, gene, protein, and disease) with six kinds of RO relationships. miRNA molecules are small non-coding RNAs that play a central role in gene expression via interference pathways and their misregulation is associated with several diseases [21]. miRNA molecules can generically interacts with genes but also more precisely regulate the activity of a gene when a miRNA molecule blocks the transof a gene or promotes the degradation of gene's product. Moreover, miRNA molecules can regulate the activity of other miRNAs because they form basepairing interactions with complementary miRNA molecules according to [22,23]. The schema also contains the relationships involving genes and proteins. Specifically, the has gene product relation and its inverse gene product of are used for representing that different proteins are translated from the same gene (i.e. isoforms); while the regulates activity of is used for representing that a subclass of the proteins (transcription factors) regulates the activity of genes, promoting or down-regulating their activity acting as enhancers or repressors. Both proteins and miRNAs are connected to the disease class by the causes or contributes to condition relation. The diagram also contains the main properties that can be associated with these bio-entities (e.g., nucleotide/amino acid sequences, descriptions of molecules/diseases, synonyms).
The proposed UML class diagram was translated into a LinkML schema. Genes are annotated using HGNC [24] IDs. This choice is motivated by the stability of the HGNC IDs even if a gene name or symbol changes. Proteins are grounded to the PRotein Ontology (PRO) while diseases are grounded to both the Monarch Disease Ontology (Mondo [25]) and the Human Phenotype Ontology (HPO [26]). miRNAs were left with no semantic annotation since miRNA labels (e.g., hsa-let-7b-5p) and miRBase [27] accession identifiers (MIMAT0000063) are CURIE prefixes not included in default SPIRES annotators. We can manually retrieve miRNA molecules from relationships extracted from SPIRES since their labels follow a pattern (for instance, "hsa-" prefix indicates human miRNAs, "mmu-" prefix murine miRNAs, mature miRNA are designated with "miR-" substring whilst "mir" refers to the stem-loop primary transcript). Labels can be then easily translated into miRBase accession identifiers using a look-up table.
). In the expression, samples of the kinds of relationships that we wish to extract are reported. The prompt generated for this class relies on the prompts generated for the classes protein and disease and used for the identification of these bio-entities from the scientific literature. Figure 3 shows an output obtained by using SPIRES and the corresponding result obtained by the simple application of ChatGPT. In the SPIRES' output, the extracted interactions are already represented as triples that exploit the required identification scheme. Therefore, checking their presence in RNA-KG and, in case of new triples, their integration is facilitated.
In this section we discuss the experiments that we carried out to evaluate SPIRES for extracting interactions involving RNA molecules. Moreover, we compare SPIRES with ChatGPT (ver. GPT-3.5-turbo), which is the LLM internally integrated in SPIRES, and with Llama 2 (ver. llama-2-70b-chat), another well-known and used LLM.
To evaluate the extraction of relations aligned with the meta-graph depicted in Figure 2, we manually selected a Listing 1: LinkML template for protein-disease interaction. corpus of 60 scientific articles gathered from PubMed, Re-searchGate, and Google Scholar by specifying keywordbased queries like: "disease", "comorbidity", "protein", "miRNA", "miRNA regulation", "gene". From these documents, we identified paragraphs containing useful information to be extracted (e.g., abstract, discussion, or specific subsections within the domain of interest). In the identification of the paragraphs we have taken into account the following guidelines: π) the paragraph should contain different kinds of relations between bio-entities (e.g., "miRNA-interacts with-gene" and "miRNA-regulates activity of-gene") to evaluate the ability of SPIRES to identify the right relations according to the provided meta-graph; ππ) the paragraph might also contain irrelevant relationships that should be discarded; πππ) different identification schemes can be used in the paragraph to check the ability of SPIRES to correctly work with them.
Paragraphs have been classified according to the kind of bio-entities that they describe and associated with the list of relationships that should be identified according to the adopted meta-graph. For each kind of bio-entity, the following table shows the number of paragraphs containing relationships involving it (note that a paragraph can contain more than one).
Protein Disease miRNA Gene 44 58 37 21
In the considered paragraphs, we have identified six kinds of interactions among the considered bio-entities (reported in the y-axis of the diagram in Figure 4).
For evaluating the obtained predictions, we have used standard metrics (precision, recall, and F-score) by considering the True Positive (TP), False Positive (FP), and False Negative (FN) according to the manually tagged paragraphs. Table 1 reports the obtained results for the considered interactions ordered according to the F-score measure. The obtained results indicate a consistent trend where recall tends to be lower than precision due to the prevalence of false negatives over false positives. We think this behavior is due to the difficulty in accurately extracting precise relationships from text, especially in distinguishing specific types of relationships. Furthermore, we observe that disease-disease and miRNA-disease interactions present a high F-score. These kinds of interactions are widely studied in the literature and thus a higher number of publications are available with respect to other interactions (like miRNA-miRNA interactions). Consequently, the abundance of this kind of relationships contributes to a higher true positive rate. Conversely, the F-score for protein-disease relations is notably low because it is influenced by low recall. We noticed that many protein-disease relations are undetected, often because they are expressed in complex ways within the text. For instance, the interchangeable use of symbols like "/" and ", " (e.g., "overexpressions in IL6/MEGF8/RELA, and also TP53 are known to cause osteoporosis"). Additionally, mapping proteins to the PRO proves challenging when textual information is sparse or ambiguously expressed. For instance, the mention of "PMP-22" solely as "myelin protein 22" instead of "peripheral myelin protein 22" (due to assumptions made by authors) can lead to inaccurate grounding. Despite this, precision remains remarkably high and, in the biomedicine context, this is preferable because it prioritizes certainty over ambiguity.
We also compared our results with the average results achieved by SPIRES in other domains. A marginal improvement has been observed in the domain of name entity recognition for chemicals and diseases [2]. We believe that the slightly enhanced accuracy is due to the use of multiple ontology annotators such as PRO for proteins, Mondo and HPO for diseases, and RO for relations.
For assessing the performance of SPIRES with respect to ChatGPT and Llama 2, we focused on a subset of 20 This prompt does not guarantee to obtain the identifiers for the subject and the object of the triples. However, if we try to generate a further prompt with the explicit request of mapping the extracted concepts to appropriate terminologies, both ChatGPT and Llama 2 advise that the provided ontology identifiers are hypothetical and may not correspond to actual ontology identifiers (so, hallucinations can occur in this case). Therefore we decided to substitute the grounding process with our manually curated look-up tables [1].
When using ChatGPT (or Llama 2) alone, we do not have to specify the schema, and results are produced through a single interaction with the user. Avoiding the specification of the schema might be interpreted as an advantage of basic LLMs approaches, but it is not. Indeed, the schema allows us to reduce the relationships to be extracted to only meaningful ones in the considered domain. Finally, no lookup table can be exploited for translating class instance names with the corresponding identifiers in the bio-ontologies (thus requiring a manual identification of the identifiers). All these drawbacks are avoided by the use of SPIRES.
As shown in the bottom part of Figure 5, SPIRES outperforms ChatGPT or Llama 2 alone both in terms of precision and recall. The histogram in Figure 5 points out a high increment in TP rate and a sensible decrease in FP and FN rates when adopting SPIRES instead of Chat-GPT or Llama 2 alone for extracting relations that adhere to a specified schema within texts.
In this paper, we have reported the initial experimentation of the use of SPIRES for extracting triples from 1-6 the scientific literature related to RNA molecules by taking advantage of the meta-graph we have realized for the generation of RNA-KG. Even if a more systematic analysis is required, the initial results are quite encouraging. To facilitate the reproducibility of our results, our dataset and the LinkML template can be downloaded from: https://doi.org/10.5281/zenodo.10671796.
As future work, we would like to extend the approach by integrating the entire RNA-KG in different ways. First, we will exploit the RNA-KG triples for enhancing the prompts generated by SPIRES. Moreover, RNA-KG can be used for validating the plausibility of the generated triples by using RNA-KG as a gold standard in the area. Furthermore, we will explore the KG-enhanced LLM inference approaches in combination with SPIRES for further improving the precision of the system by injecting knowledge extracted from RNA-KG at inference time. Finally, we would like to create a web environment for graphically showing to the user the predicted triples directly in the graphical representation of the portion of the knowledge graph that will contain them. The user can thus manually check the proposed triples and provide feedback that will be handled afterward to improve the quality of the predictions.
This research was in part supported by the "National Center for Gene Therapy and Drugs based on RNA Technology", PNRR-NextGeneration EU program [G43C22001320007] and in part by the MUSA -Multilayered Urban Sustainability Action -Project, funded by the PNRR-NextGeneration EU program ([G43C22001370007], Code ECS00000037).