Evaluation of Language Models in Knowledge Engineering (ELMKE) Workshop

Abstract

This preface outlines the scope of the Second Workshop on Evaluation of Language Models in Knowledge Engineering (ELMKE 2025) and provides essential details for inclusion in the ESWC 2025 joint proceedings. Held in Portorož, Slovenia, and co-located with ESWC 2025, ELMKE focused on topics such as evaluation methodologies, knowledge and ontology engineering, large language models, benchmark creation, human-centered evaluation, and aspects of trustworthiness, interpretability, and explainability.

1. Introduction

2. Accepted Papers

3. Program Committee

The workshop organizers would like to express their sincere appreciation to the Program Committee members for their valuable contributions and support: Terry Payne (University of Liverpool), Nitisha Jain (King’s College London), Daniel Dobriy (Vienna University of Economics and Business), Hamed Babaei Giglou (TIB), Chiara Di Bonaventura (King’s College London), Ariana Graciotti (University of Bologna), George Hannah (University of Liverpool), Arunav Das (King’s College London), Anna Sofia Lippolis (University of Bologna), Mohammad Javad Saeedizade (Linköping University), Yihang Zhao (King’s College London)

References

[1] Y. He , J. Chen , D. Antonyrajah , I. Horrocks , BERTMap: A BERT-based Ontology Alignment System , CoRR abs/2112 .02682 ( 2021 ). URL: https://arxiv.org/abs/2112.02682. arXiv: 2112 . 02682 .

[2] B. Zhang , V. A. Carriero , K. Schreiberhuber , S. Tsaneva , L. S. González , J. Kim , J. de Berardinis, OntoChat: A Framework for Conversational Ontology Engineering Using Language Models, in: The Semantic Web: ESWC 2024 Satellite Events : Hersonissos, Crete, Greece, May 26 -30, 2024 , Proceedings, Part I , Springer-Verlag, Berlin, Heidelberg, 2025 , p. 102 - 121 . URL: https://doi.org/10. 1007/978-3- 031 -78952-6_ 10 . doi: 10 .1007/978-3- 031 -78952-6_ 10 .

[3] D. Alivanistos , S. B. Santamaría , M. Cochez , J.-C. Kalo , E. van Krieken, T. Thanapalasingam , Prompting as Probing: Using Language Models for Knowledge Base Construction , 2023 . URL: https://arxiv.org/abs/2208.11057. arXiv: 2208 . 11057 .

[4] B. Zhang , I. Reklos, N. Jain , A. Meroño-Peñuela , E. Simperl , Using Large Language Models for Knowledge Engineering (LLMKE): A Case Study on Wikidata , CoRR abs/2309 .08491 ( 2023 ). URL: https: //doi.org/10.48550/arXiv.2309.08491. doi: 10 .48550/arXiv.2309.08491. arXiv: 2309 . 08491 .

[5] S. Tsaneva , D. Dessì , F. Osborne , M. Sabou , Knowledge graph validation by integrating LLMs and human-in-the- loop , Information Processing & Management 62 ( 2025 ) 104145 . URL: https://www.sciencedirect.com/science/article/pii/S030645732500086X. doi:https://doi.org/ 10.1016/j.ipm. 2025 . 104145 .

[6] B. Zhang , E. Koutsiana, Y. Zhao , A. Meroño-Peñuela , E. Simperl, Trustworthy Knowledge Graphs: Practices and Approaches, in: Handbook on Neurosymbolic AI and Knowledge Graphs , IOS Press, 2025 , pp. 363 - 384 . doi: 10 .3233/FAIA250215.

[7] B. P. Allen , L. Stork , P. Groth , Knowledge Engineering Using Large Language Models , Transactions on Graph Data and Knowledge 1 ( 2023 ) 3: 1 - 3 : 19 . URL: https://drops.dagstuhl.de/entities/document/ 10.4230/TGDK.1. 1 .3. doi: 10 .4230/TGDK.1. 1 .3.

[8] E. Koutsiana , J. Walker , M. Nwachukwu , A. Meroño-Peñuela , E. Simperl, Knowledge Prompting: How Knowledge Engineers Use Large Language Models , 2024 . URL: https://arxiv.org/abs/2408.08878. arXiv: 2408 . 08878 .

[9] J. Frey , L.-P. Meyer, F. Brei , S. Gründer-Fahrer , M. Martin, Assessing the Evolution of LLM Capabilities for Knowledge Graph Engineering in 2023 , in: A. Meroño Peñuela , O. Corcho , P. Groth , E. Simperl , V. Tamma , A. G. Nuzzolese , M. Poveda-Villalón , M. Sabou , V. Presutti , I. Celino , A. Revenko , J. Raad , B. Sartini , P. Lisena (Eds.), The Semantic Web: ESWC 2024 Satellite Events , Springer Nature Switzerland, Cham, 2025 , pp. 51 - 60 .