SemTab 2023 was the fifth edition of the Semantic Web Challenge on Tabular Data to Knowledge Graph Matching, collocated with the 22nd International Semantic Web Conference (ISWC) and the 18th Ontology Matching (OM) Workshop. SemTab provides a framework to conduct a systematic evaluation of state-of-the-art semantic table interpretation systems. In this paper, we give an overview of the 2023 edition of the challenge and summarize the results.
Tabular data is prevalent on the Web and in enterprise data lakes, data catalogs, and other data
repositories, and is often the primary data format used in data science and data analytics solutions.
In practice, there is often a wide gap between the producers and consumers of tabular data. Data
producers have the primary role of storage, maintenance, and availability of the raw data and
often share the data without much metadata or with metadata in non-standard or textual form.
On the other hand, data consumers need to identify the data they require, select relevant portions
of the data, and refine and integrate the raw data to make the data usable in their application.
This process of making the data consumable is often not feasible without the aid of automated
solutions. A key enabler of automated solutions is the annotation of data elements with entities,
classes, and relations in a knowledge graph (KG). Such annotations enable knowledge-based data
discovery [
The Semantic Web Challenge on Tabular Data to Knowledge Graph Matching (SemTab) started in 2019 with the goal of providing an avenue for benchmarking and evaluation of various STI solutions. Over the years, the SemTab participants have proposed a range of solutions incorporating a variety of approaches to automated matching, with their key strengths and weaknesses analyzed using different datasets and rounds of each of the SemTab editions. In this paper, we provide a high-level summary of the 2023 edition of the SemTab challenge, along with the results.
The SemTab 2023 challenge included two tracks: the Accuracy Track, which is the standard track for evaluation of the accuracy of the solutions; and the Datasets Track, which focuses on new datasets and applications. The datasets track was also open to the submission of revisions of the existing datasets. As in 2022, SemTab 2023 also featured an Artifacts Availability Badge.
The Accuracy Track included 2 rounds, running from April 14 to June 22, 2023. The different rounds of SemTab 2023 have been organised to evaluate participating systems on various datasets, tasks, and target KGs, with variable difficulty. As with last year and unlike the initial editions of the challenge, where the rounds were run with the support of AIcrowd, we asked the participants to submit their solutions using a submission form, and the outcome was evaluated at the end of each round.
The different datasets used to run SemTab 2023 rounds are reported in Table 1, with some
statistics available in Tables 2 and 3. As with last year and unlike the initial editions where
the ground truth was hidden from the participants, we provided partial ground truth data to the
participants during the challenge in the form of a training and/or validation set. The teams could
use these ground-truth labels to evaluate their methods locally. All the datasets are available in
Zenodo. We used four groups of datasets across the two rounds:
• WikidataTables: datasets with tables generated using an improved version of our data
generator that creates realistic-looking tables using SPARQL queries [
Link: https://doi.org/10.5281/zenodo.8393535
• tFood [
Link: https://doi.org/10.5281/zenodo.7828163
• SOTAB [
Link: https://doi.org/10.5281/zenodo.8422037
• CQA [
Link: https://doi.org/10.5281/zenodo.8398347
As per the previous editions, systems have been evaluated on a single annotation for each provided target for all the tasks. This means that in CEA, target cells are to be annotated with a single entity from the target KG. In CPA the target column pairs are to be annotated with a single property. In CTA, target columns are to be annotated with a single type from the target KG, which should be WikidataTables tFood SOTAB CQA
CQA
DBpedia ✓ ✓
Target KGs Wikidata ✓ ✓ ✓
Schema.org ✓ as fine-grained as possible, i.e., the most specific type or the lowest applicable type in the type hierarchy. Similarly, TD and CQA tasks required a single annotation in the output.
The evaluation measures for CEA, CPA and CTA are the standard Precision, Recall and F1-score, as defined in Equation 1: = |Correct Annotations| , = |Correct Annotations| , 1 = 2 × × |System Annotations| |Target Annotations| + (1) where target annotations are the target cells for CEA, the target columns for CTA, and the target column pairs for CPA. We consider an annotation as correct if it is in the ground truth set. A target cell may have multiple annotations in the ground truth, because of redirect and same-as links in KGs.
As in the past editions, given the fine-grained type hierarchy in Wikidata, we used a modified
notion of Precision and Recall in the CTA evaluation [
otherwise; where ( ) is the shortest distance to one of the ground truth classes. As for the CEA, CTA ground truth columns may have multiple classes. For example, ( ) = 0 if is a class in the GT (cscore( ) = 1), and ( ) = 2 if is a grandchild of a class in the GT (cscore( ) = 0.49). We do not consider types in the higher level(s) of the KG type hierarchy, e.g., Q35120 [entity] in Wikidata. Given the correctness score , the approximated Precision (AP), Recall (AR), and F1-score (AF1) for the CTA evaluation are calculated as follows: =
∑︀ ( ) |System Annotations| , =
∑︀ ( ) |Target Annotations| , 1 = 2 × × + (3)
Finaly, CQA results are simply evaluated based on their accuracy. That is, the score for CQA is the number of correct property-qualifier pairs in the output divided by the total number of pairs in the ground truth. For convenience of display, we show the CQA accuracy scores in the Precision column in Table 7.
The data that table-to-Knowledge-Graph matching systems are trained and evaluated on is critical for their accuracy and relevance. We invited dataset submissions that provide challenging and accessible new datasets to advance the state-of-the-art of table-to-KG matching systems. We encouraged datasets that provide tables along with their ground truth annotations for at least one of CEA, CTA and CPA tasks. The datasets could be general or specific to a certain domain.
Submissions were evaluated according to provide the following: • Description of the data collection, curation, and annotation processes. • Availability of documentation with insights in the dataset content. • Publicly accessible link to the dataset (e.g. Zenodo) and its DOI. • Explanation of maintenance and long-term availability. • Clear description of the envisioned use-cases.
• Application in which the dataset is used to solve an exemplar task.
Wikidata Tables. The top-performing system over Wikidata Tables on all three CEA, CTA, and
CPA tasks was the SemTex system, with F1 score of 0.885 on CEA, 0.934 on CTA, and 0.964 on
CPA. In comparison, last year’s top-performing system, DAGOBAH [
This year we received two submissions, out of which one was accepted for presentation at SemTab
2023: TSOTSATable Dataset [
In 2021, SemTab included a new track focusing on system usability. The main goal of this
track was to mitigate a pain point in the community: the lack of publicly available, easy-to-use,
and generic solution to address the needs of a variety of applications and settings. Since 2022,
the usability track has been replaced by the Artifacts Availability Badge, that applies to both
Accuracy and Datasets tracks. For SemTab 2023, the Artifact Availability Badge is awarded
to DREIFLUSS [
We would like to sincerely thank all the challenge participants of this year and all the previous
editions, who have all helped shape this challenge with their valuable feedback, participation at
various discussions, and all their technical contributions [