The construction of Knowledge Graphs (KGs) has become increasingly important due to their ability to integrate and represent complex relationships across various domains, making them essential for applications like information retrieval and semantic search. Recently, Large Language Models (LLMs) have been utilized to enhance KGs creation by leveraging their advanced capabilities in understanding and generating human-like text. The Large Language Models for Knowledge Engineering (LLMKE) pipeline was introduced to combine knowledge probing with Wikidata entity mapping for knowledge engineering. Nevertheless, this approach has a limitation: it primarily relies on retrieval-augmented context drawn from the first paragraph and Wikipedia Infobox of the subject entity's page. This narrow focus can lead to incomplete knowledge representations, as relevant information is often spread throughout the text and linked pages. To address this issue, we propose the Knowledge Graph Construction from Large Language Model using Retrieval-Augmented Generation method (KGC-RAG). This method leverages web scraping to retrieve documents from the subject entity's Wikipedia page and to extend the search to include linked pages, thereby increasing the likelihood of capturing comprehensive and contextually rich information. We further enhance this approach by using LLMs in conjunction with cosine similarity to filter out irrelevant content, ensuring that only the most pertinent data are included in the relevant contexts. We conducted an experiment on datasets from ISWC 2024 LM-KBC Challenge and applied the meta-llama/Meta-Llama-3-8B-Instruct model as our pre-trained large language model along with the all-MiniLM-L6-v2 as our vector embedding model. We set a relevant score threshold of 0.5 to filter Wikipedia URLs. Our approach achieved macro average F1-scores of 0.695 and 0.698 on the validation and test sets, respectively. The implementation is available at https://github.com/jaejeajay/LM-KBC2024.
CEUR ceur-ws.org
Knowledge graphs [
https://github.com/jaejeajay/LM-KBC2024 (T. Prabhong) CEUR Workshop Proceedings (A. Trongratsameethong)
Constructing knowledge graphs (KGs) is highly challenging [
However, challenges in extracting knowledge from LLMs [
Retrieval-Augmented Generation (RAG) [
In this study, we explore the construction of knowledge graphs using knowledge extracted
from LLMs. We optimize LLM performance with RAG by improving the quality of relevant
context through web scraping and web crawling. The relevance score is determined by the path
names in the Wikipedia URLs. We utilized datasets from the ISWC 2024 LM-KBC Challenge [
2.1. Using Large Language Models for Knowledge Engineering (LLMKE): A
Case Study on Wikidata
In recent research, Zhang et al. [
Nonetheless, LLMKE operates under the assumption that the most relevant documents are primarily found in the first paragraph (Introduction) and the Wikipedia Infobox, which may not always be the case, as related information can be dispersed throughout various sections.
To address this limitation, we propose a method to enhance content extraction from Wikipedia by employing web scraping techniques to gather information from paragraphs, Infoboxes, and Wikitables on the Wikipedia page of the subject entity. Also, this approach extends the search to other linked Wikipedia pages, thereby capturing more comprehensive and relevant information.
One challenge of extending the search beyond the subject entity’s Wikipedia page is the
increased volume of documents, which can slow down the process. To mitigate this, we
implemented a solution using LLM and cosine similarity scores to filter and prioritize relevant
Wikipedia pages and documents, ensuring eficient and efective information retrieval.
2.2. Retrieval-Augmented Generation for Large Language Models
Yunfan Gao et al. [
This study focuses on utilizing the Naïve RAG process to explore its application in constructing
knowledge graphs through knowledge extraction from LLMs, using the ISWC 2024 LM-KBC
Challenge dataset [
The Knowledge Graph Construction from Large Language Model using Retrieval-Augmented Generation method (KBC-RAG) provides an overview as shown in Figure 1. This method is composed of two processes: RAG and Entity Mapping. In the RAG process, relevant contexts are identified and used to enhance the LLM’s knowledge extraction. In the entity mapping process, the answers from the LLM are used to construct knowledge graphs by mapping through API functions from Wikidata, resulting in completed knowledge tuples. The details of each process are as follows:
The overview of RAG process is presented in Figure 2. The goal of this process is to extract knowledge from LLM using RAG as an LLM optimization method. The outcome of this process will be LLM-generated answers, which will then be used for subsequent entity mapping process.
The entity ID query step involves locating the Wikipedia URLs of the subject entities using the provided SubjectEntityID, which is essential for the web scraping process. The SubjectEntityID is used to search for the URLs in Wikidata through an API called SPARQLWrapper 1, with a preference for English URLs. The outcome of this step is the subject entity’s Wikipedia URL, which is used to access the subject entity’s Wikipedia page, referred to as the main page in 1https://sparqlwrapper.readthedocs.io/en/latest/main.html this study. For example, if SubjectEntity is “Uruguay”, SubjectEntityID is “Q77”, and relation is “countryLandBordersCountry”, after using SubjectEntityID to search via SPARQLWrapper, the resulting Wikipedia URL for SubjectEntity would be https://en.wikipedia.org/wiki/Uruguay.
In web scraping step, the focus is on scraping data from the Wikipedia pages. The data targeted for scraping includes three tags: p (Paragraph), Infobox, and Wikitable. The outputs of this step are the scraped data from Wikipedia, which will later be assessed for relevance in the relevant identification step. awardWonBy “Who has won ” + SubjectEntity + “ ?” companyTradesAtStockExchange “Which stock exchange does ” + SubjectEntity + “ trade on ?” countryLandBordersCountry “Which country share land border with ” + SubjectEntity + “ ?” personHasCityOfDeath “What is the city of death of ” + SubjectEntity + “ ?” seriesHasNumberOfEpisodes “How many episodes does series ” + SubjectEntity + “ has ?”
The web crawling step is designed to broaden the scope of web scraping by extending the search from the main page to other linked Wikipedia pages, thereby increasing the chances of identifying relevant context. This is achieved by identifying <a> tags within the main page. The result of this step is a collection of linked Wikipedia URLs linked to the main page. For example, from the Wikipedia page of the SubjectEntity “Uruguay” at https://en.wikipedia. org/wiki/Uruguay, examples of linked Wikipedia pages include https://en.wikipedia.org/wiki/ Economy_of_Uruguay, https://en.wikipedia.org/wiki/Religion_in_Uruguay, and so on.
The primary aim of relevant scoring step is to filter out unnecessary linked URLs from the web crawling process by using a Relevant score, calculated via cosine similarity. This approach helps streamline the RAG process. The filtering method involves calculating the cosine similarity between the path name of each URL linked to the main page and the Prompt for web scraping, utilizing the vector model all-MiniLM-L6-v2. The templates of the prompt for web scraping are shown in Table 1. For instance, from the URL https://en.wikipedia.org/wiki/Economy_of_ Uruguay, only the part “Economy of Uruguay” is used to calculate the Relevant Score. URLs with a cosine similarity score above the threshold are selected for web scraping. The outcome of this step is a refined list of linked URLs for web scraping.
The relevant identification step focuses on filtering scraped data from Wikipedia by leveraging LLM responses to specific questions. The result is a list of relevant documents, which will be used in the subsequent step for determining the relevant score.
We begin by filtering each paragraph, Infobox, and Wikitable data using the Meta-Llama3-8B-Instruct Model through question-based evaluation. The format of question is: Is this information “[Paragraph/Infobox/Wikitable]” able to answer the question: “[Prompt for web scraping]”? Information deemed relevant by the LLM is added to the list of relevant documents. If the LLM determines that it can answer the question, it will return a response of 1; otherwise, it will return 0. The answer from the LLM is 1, meaning that this paragraph will be included in the list of relevant documents.
In relevant score ranking step, the goal is to retrieve the relevant context from the list of relevant documents by applying the same method used in the relevant scoring step, combined with relevant score ranking. Each document is compared to the web scraping prompt by calculating cosine similarity using the vector model all-MiniLM-L6-v2, as in the relevant scoring step, to determine its relevant score. The documents are then ranked from highest to lowest, and the Top documents are selected to form the relevant context, which will be crucial for knowledge extraction. The key diference between this step and the relevant scoring step lies in their objectives. In relevant scoring, the focus is on identifying relevant linked URLs using a threshold-based filtering technique, while this step selects relevant documents to create the relevant context through a Top filtering approach. The Top method is not used in the relevant scoring step to avoid restricting the number of links, thereby allowing broader web scraping and increasing the chances of finding relevant documents.
The purpose of knowledge extraction step is to extract knowledge from LLM by using relevant context. The result of this step will be LLM-generated answers that are ready for entity mapping. For example, for the question “Which countries share land borders with Uruguay with country name only with comma? If None, answer None.” the answer generated by the LLM is “Argentina, Brazil”.
To ask questions using the prompt for knowledge extraction, as shown in Table 2, and the obtained relevant context, the input message format consists of three parts: 1. Relevant Context: The first message provides the relevant context to the LLM in the following format: {“role”: “system”, “content”: “Using this context to answer the question: ” + [Relevant Context]}. 2. Behavior Setting: The second message sets the behavior of the Chatbot of LLM to ensure responses are as required: {“role”: “system”, “content”: “You are a chatbot who always responds an answer in english with comma and no explanation. If u don’t know the answer, answer None”}. 3. Question Prompt: The third message is used to ask the question and is formatted as: {“role”: “user”, “content”: [Prompt for Knowledge Extraction]}.
This structured approach ensures that the LLM has the necessary context and guidance to provide accurate and concise answers.
The goal of entity mapping process is to construct a knowledge graph using the answers from the LLM obtained in the previous session. The result of this process will be completed knowledge tuples based on the given subject entity and relation. For example, after receiving the answer from the question “Which countries share land borders with Uruguay with country name only with comma? If None, answer None.” from the LLM as “Argentina, Brazil”, we map the answer using wbsearchentities. The final result is a completed tuple: {“SubjectEntity”: “Uruguay”, “Relation”: “countryLandBordersCountry”, “ObjectEntitiesID”: [“Q414”, “Q155”]}.
We begin by mapping the answers to create a knowledge graph using wbsearchentities, an API function provided by the Wikidata Query Service 2. This function searches for entities in Wikidata using labels or keywords and returns a list of matching entities ranked by relevance. The method used to select entities for linking as objects is “Choose First” for all relations. After selecting an entity, its validity is verified by checking the range of each relation. As shown in Table 3, if the entity’s “instance of” is within the same range of relations as the subject, the entity is selected for mapping.
In cases where responses from LLM exhibit ambiguity, it may be due to the LLM relying too heavily on context. For instance, in the relation companyTradesAtStockExchange, stock exchange information for a company on Wikipedia is often presented in the format “Traded as”, such as “West Japan Railway Company traded as TYO: 9021”, where TYO is the stock exchange name and 9021 is the stock exchange code. The LLM might provide an answer like “TYO: 9021”.
2https://www.wikidata.org/wiki/Wikidata:SPARQL_query_service To address this ambiguity, we employ hard coding to filter and refine the responses from the LLM, thereby improving the accuracy and clarity of the output.
The datasets used in this study were obtained from ISWC 2024 LM-KBC Challenge [
The LLM used in this study is Llama-3-8B-Instruct. Its parameter count also falls within the required limit of the task. Wikipedia was the primary data source, and the vector model chosen for cosine similarity calculations was all-MiniLM-L6-v2. This model was selected for its compact size, speed, and reasonable performance, particularly given the resource-intensive nature of web scraping and crawling, which may involve processing numerous linked URLs and a large volume of documents. It helps eficiently manage this challenge. The relevant score threshold for filtering Wikipedia URLs is set at 0.5 to select URLs similar to the web scraping prompt, which functions as a user query. A cosine similarity of 0.5 reflects a moderate level of similarity, balancing relevance and coverage to avoid overly filtering out pertinent URLs. Each document was divided into segments of 4,500 tokens, and a Top approach, with =20, was applied to combine documents into a single context within the relevant score ranking pipeline. In the relevant identification step, we used max_new_tokens=1, temperature=0.1, and top_p=0.9. max_new_tokens=1 was chosen because this step of the pipeline only returns a 0 or 1 for ifltering. In the knowledge extraction step, we set max_new_tokens=3000, temperature=0.1, and top_p=0.9, with max_new_tokens=3000 being an estimated value for the maximum length of the LLM’s responses. The choice of using =20, temperature=0.1, and top_p=0.9 was based on preliminary experiments, which demonstrated better results compared to not setting these values. Moreover, the values of temperature=0.1 and top_p=0.9 are suitable for tasks requiring logical consistency.
Our study used the baseline of meta-llama/Meta-Llama-3-8B-Instruct from ISWC 2024 LMKBC Challenge1. The baseline was derived from the use of a Masked Language Model, an Autoregressive Language Model, and Llama-3 chat models.
In our study, we used three evaluation metrics: Macro-Precision (Macro-P), Macro-Recall (Macro-R), and Macro-F1. Macro-P is the average precision score across all classes, calculated by determining the precision for each class and averaging the values. Macro-R represents the average recall score across all classes, computed similarly by averaging the recall for each class. Finally, Macro-F1 is the average F1 score across all classes, derived by calculating the F1 score for each class and averaging the results.
We conducted experiments on our system using the validation set, with the results presented in Table 5. The experimental results demonstrated that our approach outperformed the baseline, achieving an average F1-score of 0.695. For the test set, we submitted the results to the ISWC 2024 LM-KBC Challenge. The test results, shown in Table 6, indicate an average F1-score of 0.698, which is consistent with the performance on the validation set.
The source of context information plays a critical role in determining the coverage score of
the context [
While performing the task, we observed that although the responses generated by the LLM were consistent with the context derived from Wikipedia, they did not align with the information Baseline KGC-RAG awardWonBy companyTrades AtStockExchange countryLand BordersCountry personHas CityOfDeath seriesHas NumberOfEpisodes All Relations awardWonBy companyTrades AtStockExchange countryLand BordersCountry personHas CityOfDeath seriesHas NumberOfEpisodes
All Relations referenced in Wikidata. This discrepancy afected the system’s performance. For example, in the training data for the subject: Love & Hip Hop: New York, relation: seriesHasNumberOfEpisodes, the LLM predicted that the series had 143 episodes, based on Wikipedia. However, Wikidata indicated that the series had 82 episodes, which was outdated. Although both Wikipedia and Wikidata are open-source platforms where users can update information, the discrepancy in information between them still exists. Helping to update information or reporting issues to the community could help reduce this discrepancy.
High-quality context can improve the performance of question-answering in LLMs [
It was observed that for certain relations, particularly awardWonBy, the coverage score was
not high. This may be due to the fact that Wikipedia often presents award winners in tables
that include additional information. Although the award winner’s name is present, the presence
of other noise in the data can impact the relevant score [
Additionally, comparing the diference in coverage score between using web crawling and not using it reveals that web crawling generally improves the coverage score for most relations. However, in some cases, it does not. This may be due to the presence of noise in the documents obtained from web scraping. Expanding the scope of web scraping increases the likelihood of encountering noisy data, which can afect the cosine similarity score. As a result, some documents with higher cosine similarity scores may be selected over those that actually contain the correct answers to the query.
This study aims to construct KGs from LLM by using RAG to optimize knowledge extraction
from LLM and to involve the large language model in finding relevant documents using ISWC
2024 LM-KBC Challenge datasets [
This work was supported by JSPS Grant-in-Aid for Early-Career Scientists (Grant Number 24K20834).