2025 continues the mission of benchmarking semantic table interpretation systems in an era shaped by large language models (LLMs) and retrieval-augmented strategies. This edition introduces two complementary tracks. The first is the large-scale MammoTab dataset for cell entity annotation (CEA) aligned to Wikidata. The second is the new Secu-Table Track designed to evaluate robustness under noisy, adversarial, and security-domain table conditions. This paper presents an overview of the challenge setup, datasets, evaluation methodology, and the key outcomes of this edition.
Tabular data continues to serve as a foundational format across Web datasets, enterprise data lakes, and
analytics pipelines. However, semantic interpretation of tabular data remains a dificult challenge in
many real-world scenarios. A central task is the annotation of table elements with entities, classes, and
relationships from a knowledge graph (KG). This process, known as Semantic Table Interpretation (STI),
supports data discovery, integration, analytics, and knowledge-based augmentation. STI research has
been shaped by both early foundational work on semantic table annotation [
Since its inception in 2019, the SemTab Challenge has provided an annual benchmarking platform
for evaluating STI systems across multiple subtasks. The challenge has expanded over the years and
has been featured at ISWC through all editions from 2019 to 2024 [
This paper presents the 2025 edition of the challenge, summarizing its tracks, datasets, evaluation metrics, and results.
The 2025 challenge includes two main tracks. The first is the MammoTab Track, which focuses on large-scale Cell Entity Annotation (CEA). The second is the Secu-Table Track, which evaluates robustness under noisy or adversarial table conditions and includes CEA, CTA, and CPA subtasks.
Task. Participants perform Cell Entity Annotation by linking entity-bearing table cells to Wikidata entity identifiers (QIDs), or marking them as NIL when no matching entity exists.
Dataset. MammoTab is a large-scale dataset composed of more than one million Wikipedia tables
annotated using Wikidata [
The Round 1 subset contains about 870 tables with 84,907 annotated cells. It reflects natural complexity,
such as alias variation, homonymy, and NIL mentions. These challenges are motivated by earlier SemTab
editions [
An example of the annotation format is: LYQZQ0T5,1,1,Q3576864 indicating that table LYQZQ0T5’s cell in row 1 column 1 needs to be mapped to Wikidata entity Q3576864.
Evaluation. Each system outputs one entity prediction per cell. Evaluation uses:
Task. The Secu-Table Track evaluates the robustness of systems under noise, incomplete context, perturbations, and adversarial scenarios. It includes:
Two knowledge graphs are used: Wikidata for general-purpose tasks and the SEPSES Computer Security KG for security-domain CEA, CTA and CPA. Robust STI has been highlighted in past studies [14, 15].
Formats: • CEA: filename,row_id,column_id,entity_id • CTA: filename,column_id,type_id • CPA: filename,col0,col1,property_id • CEA: Cell Entity Annotation, • CTA: Column Type Annotation, • CPA: Column Property Annotation. • misspellings and alias variants, • missing or incomplete context, • added ambiguity or homonymy, • NIL mentions.
Dataset. The dataset includes 1,554 tables, including 76 gold tables and 1,478 test tables [
2 × Precision × Recall 1 = Precision + Recall
A selective mode allows predicting “I do not know” when confidence is low.
• ADFr [16]: This system is built on GRASP [17], a framework for interactive KG reasoning and SPARQL generation. GRASP enables the LLM to retrieve and validate KG entities before producing annotations. It was originally introduced for SPARQL question answering and KG exploration. The system iteratively searches, validates, and assigns entity predictions using evidence from the KG. • RAGDify [18]: This is a multi-stage retrieval and generation pipeline. It cleans table data, retrieves candidates using exact and fuzzy methods, performs LLM-based debate ranking, and applies LLM cross-level verification for NIL detection and consistency. • ditlab [19]: This framework uses iterative refinement by annotating both original and transposed table orientations, improving contextual consistency. It integrates CEA and CTA through multistage candidate generation and unsupervised entropy-based scoring. • Kepler-aSI: This system combines KG querying and LLM-based disambiguation. Earlier versions were presented in previous SemTab editions [20, 21].
System ADFr RAGDify ditlab Kepler-aSI
Precision 0.758 0.603 0.549 0.403
We thank all contributors to the SemTab Challenge across its editions from 2019 to 2025, including dataset creators, system developers, track organizers, and community members who shaped the evolution of the tasks, benchmarks, and evaluation methodology. We are also grateful to the ISWC 2025 organizers for their collaboration and support throughout the coordination of this year’s challenge.
During the preparation of this work, the authors used an LLM to assist with text generation based on website materials and informal notes. All content was reviewed and edited by the authors, who take full responsibility for the final version. for evaluating semantic table interpretation systems, 2025. URL: https://arxiv.org/abs/2511.06301. arXiv:2511.06301. [14] B. Foko, A. Jiomekong, H. Tapamo, J. Buisson, S. Tiwari, Exploring Naive Bayes Classifiers for Tabular Data to Knowledge Graph Matching, in: SemTab’23: Semantic Web Challenge on Tabular Data to Knowledge Graph Matching 2023, co-located with the 22nd International Semantic Web Conference (ISWC), 2023. [15] J. P. Bikim, C. Atezong, A. Jiomekong, A. Oelen, G. Rabby, J. D’Souza, S. Auer, Leveraging GPT Models For Semantic Table Annotation, in: SemTab’24: Semantic Web Challenge on Tabular Data to Knowledge Graph Matching 2024, co-located with the 23rd International Semantic Web Conference (ISWC), 2024. [16] S. Walter, H. Bast, Knowledge graph entity linking via interactive reasoning and exploration with grasp, in: Proceedings of the 20th International Workshop on Ontology Matching (OM 2025) co-located with the 24th International Semantic Web Conference (ISWC 2025), Nara, Japan, 2025. [17] S. Walter, H. Bast, Grasp: Generic reasoning and sparql generation across knowledge graphs, in: The Semantic Web – ISWC 2025: 24th International Semantic Web Conference, Nara, Japan, November 2–6, 2025, Proceedings, Part I, Springer-Verlag, Berlin, Heidelberg, 2025, p. 271–289.
URL: https://doi.org/10.1007/978-3-032-09527-5_15. doi:10.1007/978-3-032-09527-5_15. [18] K. Bar, T. Sagi, Llm-driven retrieval, debate, and verification for robust table-to-knowledge-graph matching, in: Proceedings of the 20th International Workshop on Ontology Matching (OM 2025) co-located with the 24th International Semantic Web Conference (ISWC 2025), Nara, Japan, 2025. [19] Y. Tachioka, Y. Terao, Cell entity annotation for semtab 2025 mammotab via iterative refinement with transposed contexts and unsupervised scoring, in: Proceedings of the 20th International Workshop on Ontology Matching (OM 2025) co-located with the 24th International Semantic Web Conference (ISWC 2025), Nara, Japan, 2025. [20] W. Baazouzi, M. Kachroudi, S. Faiz, Kepler-aSI at SemTab 2023, in: SemTab’23: Semantic Web Challenge on Tabular Data to Knowledge Graph Matching 2023, co-located with the 22nd International Semantic Web Conference (ISWC), 2023. [21] W. Baazouzi, M. Kachroudi, S. Faiz, Kepler-aSI : Semantic Annotation for Tabular Data, in: SemTab’24: Semantic Web Challenge on Tabular Data to Knowledge Graph Matching 2024, colocated with the 23rd International Semantic Web Conference (ISWC), 2024.