Knowledge graphs (KGs) are a powerful tool for representing semantic information. Existing methods depend on the use of human annotation or semi-structured automated methods based on basic metadata. However, academic papers and their abstracts are still the main way to carry academic information. The development of LLM leads to new tools to solve semantically heavy problems, so LLM can help to create KGs from texts automatically. This study comparatively evaluated LLMGraphTransformer, KGGen, and GT2KG, three LLM-based KG construction methods, using three computer science abstracts. We assessed performance via precision, recall, and F1-score against a gold standard and analyzed diferences in knowledge representation. Our findings revealed a trade-of between precision and recall in the extracted triples. Furthermore, GT2KG extracted hierarchical and definitional triples, whereas LLMGraphTransformer and KGGen Pro identified causal and functional relationships. Divergent predicate structures-simple in the gold standard vs. complex in some LLM outputs-suggest varied KG objectives, from traditional knowledge sharing to Retrieval-Augmented Generation (RAG) context capture. These results indicate that LLM-based KG construction is promising but requires further research to enhance accuracy and robustness, emphasizing that methodology choice should align with the intended application.
Knowledge graphs (KGs) are essential for structuring information to support complex problem-solving
in intelligent systems [
The recent advancements in Large Language Models (LLMs) ofer a powerful solution for extracting granular entities and relationships directly from unstructured text. This paper presents a comparative evaluation of state-of-the-art LLM-based methods for automatically constructing KGs from scientific papers, bridging the gap between metadata-level graphs and deep content understanding.
We assessed three distinct KG construction methods: LLMGraphTransformer [
Our evaluation reveals a clear trade-of between precision and recall across the tested methods. While GT2KG achieved the highest precision, LLMGraphTransformer demonstrated superior recall. Ultimately, LLMGraphTransformer (with Gemini Flash) and KGGen (with Gemini Pro) obtained the highest F1-scores, representing the most efective balance of precision and recall.
The comparable performance across all methods indicates that while LLM-based KG construction is promising, significant challenges remain. This highlights a critical need for further research to improve the accuracy and robustness of automated knowledge extraction from complex scholarly texts.
Various approaches have been proposed to convert natural language into knowledge graphs. Early
work by Hearst, developed in the early 1990s, introduced rule-based and mechanical transformation
methods for information extraction [
Pretrained language models (PLMs) have been employed for knowledge graph construction, as
demonstrated in works such as [
Alongside construction, a parallel line of work asks how to evaluate the resulting KGs. Benchmarkbased evaluations rely on curated datasets: CARB provides a crowdsourced OpenIE benchmark for triple extraction quality [14]; WebNLG measures graph–text fidelity via RDF-to-text generation [ 15]; DocRED [16] and TACRED [17] target the relation extraction task.
Beyond static benchmarks, other evaluation approaches were proposed. Diferential testing [ 18] trains multiple KG embedding models, runs head-prediction, and computes a diferential score from the proximity of model outputs. Another, downstream-utility evaluation [19], judges a KG by how much it improves fixed tasks (e.g., classification, clustering, recommendation). Finally, an LLM-as-judge approach has been proposed, where GraphJudge [20] first filters noise with an entity-centric strategy and then uses a fine-tuned LLM to assess the correctness and consistency of triples and entities.
We evaluated several LLM-based tools for knowledge graph construction from text using three randomly
selected abstracts (Document IDs 23, 438, and 519) from the G-T2KG Computer Science benchmark [
The evaluation was conducted manually, focusing on the semantic equivalence of the generated triples. Specifically, a predicted triple was considered correct if it semantically matched a gold-standard triple, regardless of diferences in wording (e.g., synonyms), structure (e.g., active/passive voice), or morphology (e.g., plural/singular forms).
We select three methods to compare: LLMGraphTransformer, KGGen and G-T2KG.
LLMGraphTransformer from LangChain transforms text into a KG by employing a pipeline of predefined prompts to
extract entities as nodes and their corresponding relationships as edges [
Our experimental setup involved the following models: Gemini 2.5 Flash was used for LLMGraphTransformer, while KGgen was tested with both Gemini 2.5 Flash and Gemini 2.5 Pro. For a baseline comparison, the results for the G-T2KG method, which employed the GPT-4 model, were extracted from its source publication.
Model
26 438 519
0.60 0.15 0.30 0.14 0.47 0.40 0.54 0.57 0.48 0.37 0.41 0.82 0.55 0.18 0.55 0.09 0.53 0.40 0.47 0.27 0.43 0.48 0.74 0.39 0.57 0.17 0.39 0.11 0.50 0.40 0.50 0.36 0.45 0.42 0.53 0.53
As shown in Table 1, LLMGraphTransformer and KG-Gen Pro demonstrate the best balance between precision and recall, achieving the highest recall in most cases while maintaining comparable precision. Conversely, G-T2KG exhibits the highest overall precision but sufers from low recall in most instances.
A comparison between KGGen Flash and KGGen Pro suggests that larger LLM models can lead to improved performance. However, LLMGraphTransformer, which also utilizes Gemini Flash, achieved similar results, indicating that the choice of methodology is as crucial as the model size.
G-T2KG, relying on an initial extraction phase with OpenIE, ofers the key advantage of strictly adhering to information explicitly stated in the text, thereby efectively preventing hallucinations. However, it struggles with the accurate identification of entities and predicates, often leading to truncated or overly extended entity spans. This limitation persists even after cleaning algorithms are applied, particularly in complex sentences involving conjunctions or coreference. In contrast, LLMs, leveraging their deep semantic understanding, are generally more efective at capturing relevant entities in diverse and intricate contexts. However, this capability can come at the cost of lower precision and potential hallucinations.
A manual comparison of the generated triples against the gold standard is necessary, as the
terminology adopted by each system can vary significantly. For instance, the gold standard almost exclusively
employs predicates consisting of a single verb. In contrast, systems such as KGGen and G-T2KG
generate more complex, compound-verb predicates. These are challenging to normalize and unify
for downstream applications, including knowledge graph-based search systems. This divergence in
predicate structure likely originates from the fundamental purpose for which KGs are created. As
discussed by Hogan et al. [
In this research, we investigated three diferent models that leverage LLMs to generate KGs from text. We evaluated their performance using abstracts from scientific papers to determine their capability to capture rich semantic context. Our findings indicate that these models show considerable promise, though there are clear areas for improvement. Furthermore, we discovered that diferent triple extraction methods generate distinct yet complementary sets of triples. This suggests that LLM-based approaches can serve a diferent purpose in KG creation compared to conventional methods. For future work, we plan to extend our evaluation by incorporating a wider range of methods and LLMs, testing on abstracts from diverse scientific domains, and extending the analysis to multiple languages.
During the preparation of this work, the authors used Gemini and Grammarly in order to: paraphrase and reword, improve writing style, and grammar and spelling checking. After using these tools, the authors reviewed and edited the content as needed and take full responsibility for the publication’s content. [14] S. Bhardwaj, S. Aggarwal, et al., Carb: A crowdsourced benchmark for open ie, in: Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), 2019, pp. 6262–6267. [15] C. Gardent, A. Shimorina, S. Narayan, L. Perez-Beltrachini, The webnlg challenge: Generating text from rdf data, in: Proceedings of the 10th International Conference on Natural Language Generation, ACL Anthology, 2017, pp. 124–133. [16] Y. Yao, D. Ye, P. Li, X. Han, Y. Lin, Z. Liu, Z. Liu, L. Huang, J. Zhou, M. Sun, Docred: A large-scale document-level relation extraction dataset, arXiv preprint arXiv:1906.06127 (2019). [17] Y. Zhang, V. Zhong, D. Chen, G. Angeli, C. D. Manning, Position-aware attention and supervised data improve slot filling, in: Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, 2017. [18] J. Tan, D. Wang, J. Sun, Z. Liu, X. Li, Y. Feng, Towards assessing the quality of knowledge graphs via diferential testing, Information and Software Technology 174 (2024) 107521. [19] N. Heist, S. Hertling, H. Paulheim, Kgreat: A framework to evaluate knowledge graphs via downstream tasks, in: Proceedings of the 32nd ACM International Conference on Information and Knowledge Management, 2023, pp. 3938–3942. [20] H. Huang, C. Chen, C. He, Y. Li, J. Jiang, W. Zhang, Can llms be good graph judger for knowledge graph construction?, arXiv preprint arXiv:2411.17388 (2024). [21] J. L. Martinez-Rodriguez, I. López-Arévalo, A. B. Rios-Alvarado, Openie-based approach for knowledge graph construction from text, Expert Systems with Applications 113 (2018) 339–355.