Pretrained language models (LMs) have significantly advanced a variety of semantic tasks and have shown promise as sources of knowledge elicitation. While prior work has studied this ability through probing or prompting, the potential of LMs for large-scale knowledge base construction remains underexplored. The fourth edition of the LM-KBC Challenge invited participants to build knowledge bases directly from LMs, given specific subjects and relations. Unlike existing probing benchmarks, the challenge imposed no simplifying assumptions on relation cardinality-allowing a subject entity to be linked to zero, one, or multiple object entities. To ensure accessibility, the challenge featured a single track based the same LLM to be used by all participants. Five submissions were received, which explored a variety of ideas from self-consistency, self-RAG, reasoning, and prompt optimization.
Large language models (LLMs) such as Qwen [
Traditional approaches to knowledge base construction (KBC) have leveraged unstructured
text [
This challenge seizes the opportunity to bridge the gap by exploring how LLMs can contribute to practical KB construction. Continuing previous eforts [ 21, 22, 23, 24], the 4th edition focuses on leveraging a single locally runnable LLM to construct KBs without prior knowledge of relation cardinalities. Specifically, given a subject–relation pair, participants were asked to design an LLM-based system that generates candidate subject–relation–object triples, and decides whether to accept or reject each one. The predictions were evaluated using F1-score.
In the LM-KBC Challenge, the knowledge base construction task is defined as follows: given a subject entity and a relation , the goal is to generate all correct object entities [1, 2, . . . , ] by probing language models. For example, given the tuple (Greece, countryLandBorderCountry), a participant might query the language model with a prompt such as “Greece shares a border with [MASK]”. The system should then output country entities like [Albania, North Macedonia, Bulgaria, Turkey], in any order. Similarly, for numeric-answer relations, such as (Wembley Stadium in London, hasCapacity), the expected output would be [“90000”].
Participants are required to build LM-based systems that produce entity labels without relying on external resources (e.g., web search engines, retrieval-augmented generation), i.e., submitted systems had to be fully self-contained. For comparison, we released a baseline method based on prompt templates, covering both question-style prompts and fill-in-the-blank templates.
awardWonBy companyTradesAtStockExchange countryLandBordersCountry
hasArea hasCapacity personHasCityOfDeath
The dataset was built by querying Wikidata and manually refining the results to reduce errors and improve quality. Compared with previous editions [21, 22], the 2025 version focuses on six challenging relations with distinctive characteristics, enabling participants to design approaches tailored to specific problem types. These relations fall into the following categories: 1. Relations with many missing objects (e.g., a person’s place of death, or the stock exchange where a company is listed). 2. Relations with long object lists (e.g., the list of award winners in a given field). 3. Standard relations carried over from the previous edition.
For each relation, up to 100 subject entities are provided for the training, validation, and hidden test sets used in challenge evaluation. The relations were carefully selected to ensure diversity, with subject entities spanning diferent types such as persons, countries, and organizations. The subject–object pairs were automatically sampled from Wikidata under the following constraints: 1. Balanced object list lengths: Longer lists were oversampled to avoid dominance by single-object examples. 2. Balanced subject popularity: Using proxies such as total Wikidata statements or web hits, we ensured roughly a 50/50 split between popular and long-tail subject entities for each relation. 3. Balanced object complexity: Both single-token and multi-token object entities were included. edarsem JingboHe acmc isam aclay Relation-Wise Self-consistency
Self-RAG and DaC
Soft Thinking
LLM-as-a-Judge Prompt optimization
The 2025 dataset is publicly available on GitHub1. Participating systems submit their predictions on the CodaLab platform2 [25], where final scores are computed on the hidden test set.
Compared with previous editions of the LM-KBC challenge [21, 22], the 4th edition introduced the following characteristics: 1. More diverse relations: As in the 3rd edition, we continue to use a smaller, representative set of relations grouped into topical categories, enabling participants to better tailor their approaches.
Compared with the 3rd edition, the numeric-answer relations were replaced by two new ones. 2. Single parameter-bounded track: The challenge continues to follow a single-track format.
Building on last year’s fixed limit of 10 billion parameters for LLMs, we further standardized the
setting by requiring participants to use Qwen3-8B [
1https://github.com/lm-kbc/dataset2025 2https://codalab.lisn.upsaclay.fr/competitions/23218
From the detailed per-relation results in Table 3, we observe that no single method consistently outperforms the others across all relations. The ReWiSe system proposed by AlbertRoulhac and Zouaq achieved the highest F1-score in three out of six relations. The SelfRAG and DaC, Soft Thinking, and LLM-as-a-Judge methods achieved the best performance on awardWonBy, companyTradesAtStockExchange, and hasCapacity, respectively. Overall, the relations awardWonBy, hasArea, and hasCapacity remain particularly challenging. Compared with the baseline, system improvements on these relations were marginal, and their absolute F1-scores stayed relatively low.
edarsem JingboHe acmc isam aclay baseline edarsem JingboHe acmc isam aclay baseline edarsem JingboHe acmc isam aclay baseline edarsem JingboHe acmc isam aclay baseline edarsem JingboHe acmc isam aclay baseline edarsem JingboHe acmc isam aclay baseline
The authors have not employed any Generative AI tools. in Natural Language Inference, in: ACL, 2019. URL: https://aclanthology.org/P19-1334/. doi:10. 18653/v1/P19-1334. [19] N. Kassner, H. Schütze, Negated and Misprimed Probes for Pretrained Language Models: Birds Can Talk, But Cannot Fly, in: ACL, 2020. URL: https://aclanthology.org/2020.acl-main.698/. doi:10.18653/v1/2020.acl-main.698. [20] B. Cao, H. Lin, X. Han, L. Sun, L. Yan, M. Liao, T. Xue, J. Xu, Knowledgeable or Educated Guess? Revisiting Language Models as Knowledge Bases, in: C. Zong, F. Xia, W. Li, R. Navigli (Eds.), ACL, Online, 2021. URL: https://aclanthology.org/2021.acl-long.146/. doi:10.18653/v1/2021. acl-long.146. [21] J.-C. Kalo, S. Singhania, S. Razniewski, J. Z. Pan, et al., LM-KBC 2023: 2nd Challenge on Knowledge Base Construction from Pre-trained Language Models, in: Joint proceedings of 1st workshop on Knowledge Base Construction from Pre-Trained Language Models (KBC-LM) and the 2nd challenge on Language Models for Knowledge Base Construction (LM-KBC), 2023. URL: https: //ceur-ws.org/Vol-3577/paper0.pdf. [22] J.-C. Kalo, T.-P. Nguyen, S. Razniewski, B. Zhang, Preface: LM-KBC Challenge 2024, in: Joint Proceedings of the KBC-LM Workshop and the LM-KBC Challenge 2024, 2024. URL: https:// ceur-ws.org/Vol-3853/paper0.pdf. [23] D. Alivanistos, S. B. Santamaría, M. Cochez, J.-C. Kalo, E. van Krieken, T. Thanapalasingam, Prompting as Probing: Using Language Models for Knowledge Base Construction, 2023. URL: https://arxiv.org/abs/2208.11057. arXiv:2208.11057. [24] B. Zhang, I. Reklos, N. Jain, A. M. Peñuela, E. Simperl, Using Large Language Models for Knowledge Engineering (LLMKE): A Case Study on Wikidata, 2023. URL: https://arxiv.org/abs/2309.08491. arXiv:2309.08491. [25] A. Pavao, I. Guyon, A.-C. Letournel, D.-T. Tran, X. Baro, H. J. Escalante, S. Escalera, T. Thomas, Z. Xu, CodaLab Competitions: An Open Source Platform to Organize Scientific Challenges, Journal of Machine Learning Research (2023). URL: http://jmlr.org/papers/v24/21-1436.html.