Introduction

Tabular data is ubiquitous across the Web, enterprise data lakes, data catalogs, and other repositories, serving as a foundational format in data science and analytics. However, a significant gap often exists between those producing tabular data and those consuming it. Data producers focus on storing, maintaining, and ensuring the availability of raw data, frequently sharing it with minimal metadata or metadata in non-standard or textual forms. In contrast, data consumers must locate the data they need, extract relevant subsets, and refine and integrate the raw data to render it suitable for their applications. Achieving this transformation is often impractical without automated solutions. A cornerstone of such automation is the annotation of data elements with entities, classes, and relationships from a knowledge graph (KG). These annotations facilitate knowledge-based data discovery [1,2,3,4], organization [5], integration [6,7], and augmentation [8]. Automating the task of linking tabular data to KGs, commonly known as Semantic Table Interpretation (STI), has been extensively studied in the literature [9,10,11,12,13,14,15,16,17,18,19].

The Semantic Web Challenge on Tabular Data to Knowledge Graph Matching (SemTab) started in 2019 with the goal of providing an avenue for benchmarking and evaluation of various STI solutions. Over the years, the SemTab participants have proposed a range of solutions incorporating a variety of approaches to automated matching, with their key strengths and weaknesses analyzed using different datasets and rounds of each of the SemTab editions. In this paper, we provide a high-level summary of the 2024 edition of the SemTab challenge, along with the results.

The Challenge

The SemTab 2024 challenge comprised three tracks. The Accuracy Track, which evaluated the accuracy of semantic table interpretation solutions, continued as in previous years. Two new tracks were introduced this year: the STI vs LLMs Track, which focused on assessing cell entity annotation solutions leveraging large language models (LLMs), and the Table Metadata to KG Track, which addressed the challenge of matching tabular data using only table metadata. Although a call for datasets was issued, no submissions were received, and the datasets track was consequently omitted from this year's challenge.

Accuracy Track

The Accuracy Track consisted of two rounds, each featuring three datasets. This year, all datasets were aligned with the same target knowledge graph, Wikidata [20]. Similar to last year, participants submitted their solutions via a submission form, and the results were evaluated at the conclusion of each round. This year we also performed additional rounds of evaluation, with the last round right before the conference.

Datasets

Table 1 provides an overview of the datasets used in the Accuracy Track, along with their corresponding statistics. Similar to the previous year-and in contrast to the earlier editions where ground truth was kept hidden-participants were provided with partial ground truth data during the challenge in the form of training and/or validation sets. These labels enabled teams to evaluate their methods locally. All datasets are openly available on Zenodo. Across the two rounds, three groups of datasets were utilized:

• WikidataTables https://doi.org/10.5281/zenodo.14207232

This dataset comprises tables generated using an enhanced version of our data generator, which produces realistic-looking tables through SPARQL queries [21]. The target knowledge graph (KG) for this dataset is Wikidata, and, as in previous years, the tasks include Cell Entity Annotation (CEA), Column Type Annotation (CTA), and Column Property Annotation (CPA). As detailed in Table 1, the test set for Round 1 consists of 30,000 tables with an average of 2.5 columns and 61.7 rows, while the Round 2 dataset consists of 78,745 tables with an average of 2.5 columns and 11.6 rows. For these collections, the dataset generator was configured to produce a large number of small to medium-sized tables with high ambiguity in entity columns. This ambiguity was introduced by filtering for labels that can refer to multiple entities in Wikidata.

• tBiodiv https://doi.org/10.5281/zenodo.10283015 tBiodivL https://doi.org/10.5281/zenodo.10283083 These datasets, generated using KG2Tables [22] for the biodiversity domain, include two types of tables: 1) "horizontal" relational tables, where each table represents a collection of entities, and 2) "entity" tables, where each table represents a single entity. Ground truth mappings to Wikidata were provided for the CEA, CTA, and CPA tasks, as well as for the Topic Detection (TD) task, which focuses on annotating an entire table to instances/entities or types/classes, and the Row Annotation (RA) task, which involves mapping each row to an entity. As shown in the statistics in Table 1, the relational table datasets are wider and exhibit greater variation in the number of columns.

• tBiomed https://doi.org/10.5281/zenodo.10283103 tBiomedL https://doi.org/10.5281/zenodo.10283119 These datasets, also generated using KG2Tables [22] for the biomedical domain, include both relational and entity tables. They are accompanied by ground truth mappings for the CEA, CTA, CPA, TD, and RA tasks. As indicated in Table 1, the tBiomed datasets contain a larger number of tables but are comparable to the tBiodiv datasets in terms of the average number of rows and columns.

Evaluation measures

As in prior editions, systems were evaluated based on a single annotation for each specified target across all tasks. For CEA, this meant annotating target cells with a single entity from the target KG. In CPA, the task involved assigning a single property to the target column pairs. For CTA, the goal was to annotate target columns with a single type from the target KG, selecting the most specific or fine-grained type in the hierarchy. Similarly, the TD and CQA tasks required a single annotation to be provided as output.

The evaluation metrics for CEA, CPA, and CTA were Precision, Recall, and F1-score, defined as follows in Equation 1:

𝑃 = |Correct Annotations| |System Annotations| , 𝑅 = |Correct Annotations| |Target Annotations| , 𝐹 1 = 2 × 𝑃 × 𝑅 𝑃 + 𝑅(1)

In this context, target annotations refer to the designated target cells for CEA, target columns for CTA, and target column pairs for CPA. An annotation is considered correct if it matches any entry in the ground truth set. Due to redirect links or same-as links in KGs, some target cells may have multiple valid annotations in the ground truth.

For CTA evaluation, a modified version of Precision and Recall was applied, given the detailed type hierarchy in Wikidata [23]. This adaptation accounts for partially correct annotations, such as those that are ancestors or descendants of the ground truth (GT) classes. The correctness score 𝑐𝑠𝑐𝑜𝑟𝑒 for a CTA annotation 𝛼 is based on its distance from the GT classes within the hierarchy and is defined as follows: ) , if 𝛼 is in GT, or an ancestor of the GT, with 𝑑(𝛼) ≤ 5 0.7 𝑑(𝛼) , if 𝛼 is a descendant of the GT, with 𝑑(𝛼) ≤ 3 0, otherwise;

cscore(𝛼) = ⎧ ⎪ ⎨ ⎪ ⎩ 0.8 𝑑(𝛼

(

Here, 𝑑(𝛼) denotes the shortest distance from 𝛼 to one of the GT classes. CTA ground truth columns can include multiple valid classes. For example, if 𝛼 is a GT class (𝑑(𝛼) = 0), the correctness score is cscore(𝛼) = 1. If 𝛼 is a grandchild of a GT class (𝑑(𝛼) = 2), the correctness score is cscore(𝛼) = 0.49. Types from higher levels of the KG type hierarchy, such as Q35120 [entity] in Wikidata, were excluded from the evaluation.

Using the correctness score 𝑐𝑠𝑐𝑜𝑟𝑒, the approximated Precision (AP), Recall (AR), and F1-score (AF1) for CTA were calculated as follows:

𝐴𝑃 = ∑︀ 𝑐𝑠𝑐𝑜𝑟𝑒(𝛼) |System Annotations| , 𝐴𝑅 = ∑︀ 𝑐𝑠𝑐𝑜𝑟𝑒(𝛼) |Target Annotations| , 𝐴𝐹 1 = 2 × 𝐴𝑃 × 𝐴𝑅 𝐴𝑃 + 𝐴𝑅(3)

STI vs LLMs Track

This track investigates the exclusive use of LLMs for performing the CEA task on Wikidata. Participants are tasked with either fine-tuning an LLM or employing prompting techniques on a dataset enriched with semantic annotations. The task presents several challenges, including integrating factual knowledge from a knowledge graph (KG) into an LLM, devising strategies for handling Wikidata QIDs, enhancing the training dataset to improve disambiguation accuracy, mitigating hallucination issues, and designing effective prompts for fine-tuning or annotation purposes. The primary objective is to leverage the capabilities of LLMs to generate high-quality annotations for the CEA task, advancing their applicability in semantic enrichment. Participants are required to submit their annotations for evaluation on the test set, demonstrating the practicality and effectiveness of their approaches. The provided tabular datasets consist of columns with entity mentions, which must be annotated with the corresponding Wikidata entities. These annotations should include the entity's URI, though the prefix http://www.wikidata.org/entity/ is optional. The evaluation metrics-Precision, Recall, and F1-are consistent with those used for CEA in the Accuracy Track.

Datasets

Table Metadata to KG Track

This track challenges participants to match limited table metadata, such as table names and column headers, to knowledge graphs without access to the actual table data or content. The task is inherently difficult due to the limited context available for annotation systems to perform semantic linking. LLMs offer a promising solution to address this challenge, providing flexibility in their application. The datasets for this track are adapted from our previous work on matching table metadata with business glossaries using large language models [25].

Datasets

Link: https://doi.org/10.5281/zenodo.14207376

• Round 1: This dataset consists of metadata from selected web tables that need to be mapped to the DBpedia ontology. The target ontology (also referred to as the glossary) contains 2,881 terms from the DBpedia ontology. The test dataset includes metadata (table and column labels) for 141 table columns. A small test set with metadata for 9 table columns, along with an evaluation script, was provided. • Round 2: This dataset consists of metadata from selected open data tables that need to be mapped to a custom glossary containing 1,192 terms, semi-automatically derived from the available metadata. The provided table metadata includes metadata (table and column labels) for 1,192 table columns.

We use "Hit@1" and "Hit@5" as evaluation metrics, representing the percentage of table columns correctly matched to the ground truth glossary item within the top 1 and top 5 predictions in the system outputs, respectively.

Results

Tables 2, 3, and 4 summarize the results for each of the three tracks. Overall, seven participants submitted solutions to at least one dataset across any round:

• TSOTSA [26] explores building an STI solution using a GPT-3-based model through both few-shot and zero-shot prompting techniques and participated in the Accuracy Track as well as the STI vs LLMs Track. • DREIFLUSS [27] employs a minimalist approach that carefully utilizes resources such as Wikidata APIs for the annotation process. • Kepler-aSI [28] leverages SPARQL queries, embeddings, custom index structures, and a NoSQL database to address the CEA, CTA, and CPA tasks. • MetaLinker [29] investigates the use of various LLMs and sentence embeddings for the Metadata to KG Track. • Adwan [30] combines Retrieval-Augmented Generation (RAG), Chain-of-Thought (CoT) prompting, Self-Consistency (SC), and Reciprocal Rank Fusion (RRF) to develop an LLM-based solution for the Metadata to KG Track. • GRAMS+ (ISI KG) [31] constructs a prediction model for CPA and CTA tasks using distant supervision. • CitySTI [32] participated in the STI vs LLMs Track, utilizing a two-stage approach where LLMs were used for data cleaning and matching, executed entirely through prompting techniques.

In the Accuracy Track, the TSOTSA system participated in the largest number of datasets, while other systems focused on only one or two datasets. TSOTSA demonstrated promising performance on several tasks, including TD, RA, CEA, and CTA, in the tBiodiv-Relational and tBiomed-Relational datasets, as well as the CEA task in the tBiodiv-Entity and tBiomed-Entity datasets. However, it struggled with certain tasks, even on the simpler WikidataTables datasets, suggesting potential scalability challenges in its LLM-based solution. In contrast, ISI-KG delivered exceptional results on the WikidataTables datasets, showcasing the effectiveness of building a prediction model using distant supervision. The DREIFLUSS and Kepler-aSI systems also achieved notable results on the larger tBiodiv-Large-Relational and tBiomed-Large-Relational datasets.

In the STI vs LLMs Track, TSOTSA achieved the best performance on the SuperSemtab 24 Round 1 dataset, while the CitySTI system showed promising results across both datasets.

Finally, the two solutions in the Metadata to KG Track offered valuable insights into how various LLM models and prompting techniques can address the challenge of matching table metadata to knowledge graphs or business glossaries without access to table contents. The Adwan solution achieved outstanding Hit@5 scores of 0.92 in Round 1 and 0.98 in Round 2.