Ontologies and the knowledge graphs derived from them are valuable knowledge representation approaches in many application areas. Designing such ontologies is a laborious process requiring domain expertise and knowledge engineering skills. The process can be assisted by tapping into existing sources of knowledge containing a structured component. One example of this is using the labelled information shown in the little grey rectangle at the top of each Wikipedia page to produce a vast ontology known as DBpedia [1]. Domain-specific work in the biomedical, finance [2,3] or clinical. Most textual information is available in a non-structured form though, and extracting concepts and relations is a valuable but challenging task [4].
Retrieval Augmented Generation (RAG) offers a transformative approach to enhancing Question Answering systems by dynamically retrieving and incorporating external knowledge during the generation process. In the context of ontologies, RAG can be particularly powerful. By integrating RAG with a robust ontology, the system can access a structured repository of domain-specific knowledge, such as those concerning environmental policies and practices outlined in Nationally Determined Contributions (NDC) reports. This integration allows the QA system to fetch relevant information and contextually adapt its responses based on ontological relationships and entities. The methodology leverages the precision of ontology-based data retrieval and the flexibility of generative models to produce more accurate and contextually relevant answers. Studies such as those by Li et al. (2024) have demonstrated the efficacy of RAG in various domains by showing significant improvements in the accuracy and relevance of generated answers [5]. Applying these principles, our model aims to tackle environmental data's complex terminology and inter-relation characteristics, thereby enhancing the decision-making process in climate change mitigation efforts.
In this paper, we start with these ideas to investigate how LLM can construct domain-specific knowledge graphs with limited participation of domain experts. Then, we involve our constructed ontology into a retrieval augmentation generation (RAG) system for the question and answering(QA) system to answer a set of NDC-related questions and evaluate the answer. Our contributions are as follows:
• We implement a novel environmental knowledge graph framework manually and apply it to extract information from environmental reports on climate change. This approach aims to capture important information and organise it into an ontology.
• Based on the ontology we constructed, we developed a hybrid RAG system to retrieve information from both our ontology and NDC report. Our evaluation shows that our method performs better than the baseline method.
Large knowledge graph such as Freebase [6] and DBpedia [1] have shown a remarkable ability to provide a well-structured data source in a wide range of NLP tasks, including question-answering [7], information retrieval [8], chatbots [9], recommendation systems [10] and machine translation [11].
With structured vocabulary and predefined relationships, knowledge graphs and ontologies facilitate automated associations between data and knowledge, enabling nuanced analyses and hypothesis generation in various scientific domains such as finance [3] and health care [12]. However, constructing domain-specific ontology is a multifaceted, challenging task requiring the deep participation of domain experts. Researchers usually divided knowledge construction into separate NLP tasks, including named entity recognition(NER) and relation extraction(RE). Since models and NLP tools based on the general domain cannot satisfy the form of scientific corpus or data, all steps require independent fine-tuning or adjustment for different scientific requirements, as well as human annotation with domain experts [4]. Some joint relation extraction methods, such as NovelTagging [13] or TPLinker [14], demonstrate the concept of unifying NER and RE steps, which can mitigate the error propagation between different models and reduce the dataset annotation cost and also improve the performance for tackling overlapping or complex relation scenarios. These also push the work on scientific knowledge graph construction.
With the rapid growth of LLMs, the few/zero-shot learning abilities via prompt learning on LLMs are widely investigated by NLP researchers. Recent progress in large language models (LLMs) like GPT series(GPT-3 [15], ChatGPT, GPT-4 [16]), LLama-series(LLama [17], Alpaca [18], Vicuna [19]), PaLM [20] and GLM [21] has showcased remarkable performance across a spectrum of natural language processing (NLP) tasks. The ongoing expansion in the number of model parameters and the size of training datasets has endowed LLMs with emergent capabilities, facilitating them to partake in in-context learning (ICL). In ICL, these models can derive insights from a limited set of demonstrative examples presented within the input context and exceed the previous baselines in multiple NLP tasks under few-shot or zero-shot settings, as well as relation extraction and knowledge construction [22].
Based on the ICL paradigm, Wadhwa et al. [23] propose comparing few-shot LLM-based relation extraction and fine-tuned baselines. At the same time, the result indicates that Few-shot prompting with GPT-3 achieves near SOTA performance and can exceed the previous supervised SOTA performance when using the Chain-of-Thought (COT) [24] prompt for enhancement. In addition, in the domainspecific area, Agrawal et al. [25] point out that LLMs perform well at zero-and few-shot clinical relation extraction despite not being explicitly trained in clinical records. Rajpoot et al. [26] developed the GPT-FinRE model using ICL and achieved a 0.718 F1 score on their financial dataset, REFinD. Moreover, some works such as LLM2KB [27] attempt to connect LLM to the knowledge construction task, which aims to construct a knowledge base from input relation triples, and link to the external knowledge base such as Wikidata to capture the related information. They achieved an average F1 score of 0.6185 across 21 relations in the LM-KBC challenge held at the ISWC 2023 conference.
Retrieval-augmented generation (RAG) is a method that integrates traditional large language models (LLMs) with external knowledge sources to enhance their output quality and relevance. The primary goal of RAG systems is to overcome the limitations inherent in LLMs, such as content hallucination, outdated information, and lack of traceable reasoning processes by dynamically incorporating up-todate information from external databases or other structured knowledge forms during the generation process [28]. This methodology significantly improves the models' accuracy and reliability by ensuring that the generated content is contextually relevant and factually accurate.
The operational framework of an RAG system typically involves querying an external database using a retrieval mechanism that can pull relevant information based on the input query or context. This retrieved content is then fed into the LLM, enabling the model to produce outputs enhanced by the externally sourced data. The integration of retrieval capabilities allows RAG systems to remain dynamic, adapting to new data and evolving user requirements.
Using ontology in RAG systems introduces a structured way of organizing information that can significantly enhance the retrieval process. Ontologies provide a framework for representing knowledge as a set of concepts within a domain and the relationships between those concepts. This structured approach allows RAG systems to perform more precise and context-aware retrievals, essential for generating accurate and relevant outputs.
A notable example is presented in the study by Sabrina Toro et al., where they explore the "Dynamic Retrieval Augmented Generation of Ontologies using Artificial Intelligence (DRAGON-AI)" system. [29] This system utilizes ontologies to dynamically generate and update knowledge, illustrating a profound enhancement in managing and utilizing structured domain knowledge effectively in real-time applications. This methodology is particularly beneficial in fields requiring precise and up-to-date information, such as biomedical and environmental sciences, where it helps mitigate the substantial collaborative efforts typically required from domain experts.
Further, Julien Delile et al. introduce a graph-based retriever that captures the long-tailed data of biomedical knowledge. Their work emphasizes the utility of integrating graph-based knowledge representations with RAG systems to handle rare or less frequently seen information better. [30] By incorporating these knowledge graphs, the system achieves a more comprehensive retrieval coverage, significantly enhancing the generation capabilities of LLMs in the biomedical research domain. This approach enhances the retrieval of relevant data. It improves the overall accuracy and utility of the generated content by ensuring it reflects rare yet critical information that might otherwise be overlooked.
These studies demonstrate how ontologies and knowledge graphs can significantly enhance RAG systems by providing a more structured, nuanced, and comprehensive approach to information retrieval and generation. The structured nature of ontological frameworks allows for a deeper semantic understanding of queries and the relationships between different pieces of information, leading to more accurate and contextually appropriate outputs. However, there is still a lack of research which directly links to the ontology-enhanced RAG system for extracting information in the climate change area, which is the motivation for our research.
To demonstrate our framework, we considered the Nationally Determined Contributions (NDCs) to the United Nations Framework Convention on Climate Change (UNFCCC). These documents, which are updated every five years, outline countries' national strategies to mitigate and adapt to climate change. Specifically, we focused on elements of NDCs pertaining to Nature-based Solutions (NBS), which, broadly speaking, is the sustainable use or use of natural features and processes to tackle socio-environmental issues (in this case, climate change).
As the urgency to mitigate and adapt to climate change becomes increasingly pressing, consideration of climate strategies and policies by analysts is likely to become a more common process. As such, the development of ontologies on relevant issues is likely to be of great utility to those involved, as they will essentially allow rapid and automated comparisons to be made between countries, institutions and other entities. Moreover, given that climate change issues typically cut across multiple themes and sectors (e.g. energy, transport, or, in the case of NBS, biodiversity conservation and natural resource management), efforts to assess (and ultimately maximise) levels of cross-sectoral alignment between policies and intended activities may also be expected to increase, and can also be facilitated through the use of topic-specific ontologies. In this regard, we hope that our efforts to develop an ontology specific to NBS will have further utility beyond this demonstration.
We collected the environmental reports we needed from the Convention on Biological Diversity(CBD) and the United Nations Climate Change(UNFCCC) website. We chose 10 reports from all publications by different countries, including Albania, Angola, Antigua and Barbuda, Armenia, Australia, Azerbaijan, Bahrain, Bangladesh, Nigeria, and Papua New Guinea (PNG).
As figure 1 shows, our model mainly includes two main processes: Relation Triple Extraction aims to extract important features from the given NDC reports and output them as a fixed JSON format, while Relation Triple Alignment receive the structural result combining with a pre-designed ontology framework to assign entities to the specified class.
In the Relation Triple Extraction process, we follow the general RAG structure to load different NDC reports and split them into pages. Through one LLM as a paging encoder to generate different embeddings and store them in a vector database. Then, we design a structural prompt to let LLM extract featured information with JSON-style output by computing the similarity between the page's vectors. Our prompt and example result for the structural extraction are shown in Table 1.
Then, to mostly capture the relation between different initiatives from different countries, using Stanford's WebProtégé [31], we create our knowledge base framework with one domain expert. The main structure(components) of the ontology and some related examples are shown as follows:
• Object properties: Object properties link pairs of instances (individuals). They describe the relationship between two individuals, e.g. country A benefits_from project B. {{ "Initiative 1": {{ "Nature-Based Solution": "", "Actor": "", "Challenge": "", "Start Date": "", "End Date": "", "Location": "", "Budget": "" }}, and start with Initiative 1, Initiative 2, etc. Please gives out as many as initiatives that mentioned in the NDC document as you can find.
Here are some initiatives related to nature-based solutions in Papua New Guinea's NDC, based on the provided document: { "Initiative 1": { "Nature-Based Solution": "Reducing Emissions from Deforestation and Forest Degradation and Enhancement of Carbon Stocks (REDD+)", "Actor": "Papua New Guinea government, with support from development partners", "Challenge": "Forest degradation and deforestation, particularly due to logging and land use changes for agriculture", "Start Date": "2017", "End Date": "2027", "Location": "Nationwide", "Budget": "N/A" } ...
to represent specific objects or entities in the domain of interest. E.g. Indonesian is a named individual and belongs to the class Country.
Figure 2 shows the most general classes, as well as some of the class hierarchies in our ontology (known as Aris). We defined seven main class categories that conclude different kinds of national policy attributes in 'Action', 'Actor', 'Budget', 'Challenge', 'Document', 'Location' and 'Outcome'. Similarly, in figure 3, we show the basic design for object and data properties, which are defined to reflect the relationship between classes and individuals. Figure 4 indicates an example ontology individual called 'FoodSecurityInNigeria' that belongs to the 'FoodSecurity' subclass, which links through several different subclass under the main class 'Challenge' and connected to the other three different main classes: 'Outcome', 'Actions' and 'Actor'.
For the relation triple alignment process, based on the output of the last step, we develop another prompt combined with the hierarchy structure of our Aris ontology framework to let LLM assign an entity to a leaf subclass such as the 'FoodShortages' in figure 2. Specifically, given a set of tree-structure class-subclass names as {JSON}, the LLM will classify the entity to a certain subclass and give the thinking process. Our prompts and example result are shown in Table 2, with content in parentheses used as a placeholder:
Following the progress above, we constructed the fulfilled Aris ontology with 10 countries' NDC reports. The complete version of Aris ontology contains 5923 relation triples with 401 classes and subclasses and 19 different relations. We extracted 155 environmental initiatives across 10 countries that contain nature-based solutions, challenges, actors and time period information.
In the next step, to verify the performance of our ontology straightforwardly, we implement an RAGbased Q&A model that combines the retrieval information from both vector database and structure ontology. As the figure 5 shows, the model will retrieve a set of select page content ranking by the similarity score with cosine similarity function, as well as generate related SPARQL query to achieve information from the Aris ontology to generate answers. Specifically, our method will extract entities related to the country in the question and then generate SPARQL query to extract related information from ARIS ontology to generate the answer. To compare our result, we set the baseline as the general RAG framework with the retrieval vector from the page-split database.
To evaluate our result, we propose a set of questions across the climate-change-based information we are interested in. To perform a clear structure for evaluating the result, we design the prompt prefix for the questions and let the LLM answer it by listing its answer, reason and reference. Some examples are shown in Table 3.
Based on these questions, we formulate our dataset across 10 countries and their combinations for single and double-country queries, then generate 40 questions for a single query and 45 questions for a double-country query. The RAG-based model will answer the questions based on the prompt to list
Here is a list string that represents some classes in an ontology: {JSON}. Now, I want to give you a named individual:{Ind}, which should belong to one of these subclasses. If we consider the JSON I give to you as a tree structure, please find the most relevant class that the individual belongs to by the meaning. If you think no leaf subclass is relevant please answer "No match found". Please only give one answer for the suitable subclass, but not two or more. Please answer this question in this format precisely without any additional explanation or information: Thinking: {{The steps you are thinking}} Answer: {{The subclass name}} Example Answer from LLM: Thinking: The individual "Papua New Guinea government, with support from development partners" is a political entity at the national level. I will search for a matching subclass in the provided ontology that represents this concept. Answer: NationalGovernment points in enumerates. We manually verified our result based on the facts in the original NDC report against the following criteria: Precision: How many answers match the facts in the NDC report? Recall: How many answers from the model cover the points in the NDC reports?
We choose Mixtral 8*7B [32] as the core LLM for both embedding and ontology/RAG model. Compared to the other open-source model, the Mixtral 8*7B model has demonstrated superior performance across a variety of benchmarks, including code generation, math problem-solving, and general language understanding.
It matches or exceeds the capabilities of more extensive models like Llama 2 70B and GPT-3.5 with the efficient memory requirement and longer context length, up to 32000 tokens. We locally run the Mixtral 8*7B model with 1 H100 GPU for all the tasks we proposed above.
The result in table 4 shows the performance of our model. We separately test our model on the baseline model and the Hybrid RAG model with human evaluation to check the result. Compared to the baseline model, our model achieved 0.836 precision and 0.588 recall in the single-country query questions and 0.676 precision and 0.582 recall in the cross-country query questions, which indicates the success of the enhancement based on the ontology data during the RAG progress. Ontology data: {'solution': 'IntegratedWaterResourcesManagementAndWatershedProtection', 'challenge': 'WaterScarcity_Flooding_AndClimateChangeImpactsOnWaterResources', 'Actor': 'Ministry-OfEnergyAndWater', 'budget': 'N/A', 'start_date': '2020', 'end_date': '2030'}, ...
Here we can see that our model successfully answer the question with the ontology data even if the baseline hasn't retrieve the related information from the text. Our model suffer from the manual evaluation efficiency which led to a limited question dataset to expand the question type to other type of environmental problem. However, our result and example demonstrate that to integrate with the ontology we design, our model can exceed the raw RAG system and sufficient in answer the NDC-based environment questions.
While our study introduces a novel integration of ontological frameworks with retrieval-augmented generation models, it also encounters specific limitations that pave the way for future research opportunities. Firstly, the current ontology is restricted to extracting relational triples from Nationally Determined Contributions (NDC) reports across ten countries. This limitation confines the breadth of our ontological database and potentially impacts the model's applicability to global environmental data. Future efforts will aim to expand our method to include a wider range of countries, thereby enriching the ontology's diversity and representativeness.
Secondly, the scope of our evaluation is constrained by the manual effort required, limiting us to a smaller dataset of questions and answers. This has potentially restricted our ability to thoroughly assess the model's performance across more complex environmental queries and diverse data scenarios. Moving forward, we plan to develop more robust evaluation methodologies that can handle larger and more intricate datasets of environmental-related questions. This will not only improve the accuracy and reliability of our model but also enhance its utility for more comprehensive environmental information retrieval tasks.
By addressing these limitations, future research can significantly enhance the model's functionality and applicability, providing more detailed and extensive support for environmental decision-making processes.
This study presents a novel integration of Retrieval Augmented Generation (RAG) with a specialized ontology tailored for environmental policy documents, particularly focusing on Nationally Determined Contributions (NDC) reports. The development of this hybrid model showcases an innovative approach to improving the accuracy and relevance of question-answering systems in the environmental domain. Our primary contribution lies in the formulation of a domain-specific knowledge graph that significantly augments the question-answering capabilities of large language models (LLMs) through ontology-based queries. The combination of RAG embeddings with these queries allows our model to deliver responses that are not only accurate but also deeply contextualized, drawing on the structured knowledge embedded in environmental policies. Testing across different LLMs has confirmed that our model offers a clear improvement over traditional RAG systems in handling complex environmental data. The ability to integrate and manipulate domain-specific ontologies enables the model to address intricate queries about climate change mitigation strategies and policies with a higher degree of precision. Our future work will focus on expanding the ontology's scope, refining the integration mechanisms, and exploring the application of this model to other domains requiring high accuracy and context-specific responses. This endeavour not only advances the field of environmental informatics but also contributes to the broader discussion on the potential and limitations of applying advanced AI techniques in highly specialized and impactful areas.