SemTab 2022 was the fourth edition of the Semantic Web Challenge on Tabular Data to Knowledge Graph Matching, successfully collocated with the 21st International Semantic Web Conference (ISWC) and the 17th Ontology Matching (OM) Workshop. SemTab provides a common framework to conduct a systematic evaluation of state-of-the-art systems. In this paper, we give an overview of the 2022's edition of the challenge and summarize the results.
Tabular data are the most frequent input to data analytics pipelines, thanks to their high storage
and processing efficiency. Also, the tabular format allows users to represent the information in a
compacted way, by exploiting the clear data structure defined by rows and columns. However, such
clear structure does not imply a clear understanding of the semantic structure (e.g., relationships
between columns), as well as the meaning of the content (e.g., if data are about a specific topic).
The lack of understanding hinders data analytics processes, requiring additional effort to properly
understand the data first. Gaining the semantic understanding is valuable for many applications,
including data cleaning, data mining, data integration, data analysis and machine learning, and
knowledge discovery. For example, the semantic understanding can help in assessing what kind
of transformations are more appropriate for a dataset, or which datasets can be integrated to
enable new analytics (e.g., marketing analysis) [
In addition to their efficiency, the huge availability of tabular data on the Web makes Web tables a valuable source to consider for data miners (e.g., open data CSV files). Adding semantic information to Web tables is useful for a wide range of applications, including web search, question answering, and knowledge base construction.
Tabular data to Knowledge Graph (KG) matching is the process of clarifying the semantic meaning of a table by mapping its elements (i.e., cells, columns, rows) to semantic tags (i.e., entities, classes, properties) from KGs (e.g., Wikidata, DBpedia). The task difficulty increases when table metadata (e.g., table captions, table description, or column names) being missing, incomplete or ambiguous.
The tabular data to KG matching process is typically broken down into the following tasks: • cell to KG entity matching (CEA task), • column to KG class matching (CTA task), and • column pair to KG property matching (CPA task).
Over the last decade several approaches made advances in addressing one or several of above
tasks, also constructing benchmark datasets ([
The SemTab 2022 challenge has been organised into two different tracks: the Accuracy Track, which is the standard track proposed in previous editions; and the Datasets Track, which focuses on applications in real-world settings where the output of matching systems can contribute. The datasets track was also open to the submission of novel benchmark datasets. SemTab 2022 also featured an Artifacts Availability Badge that was applicable to both tracks.
The Accuracy Track included 3 rounds, running from May 26 to October 15. Different target
KGs were used across rounds (see Table 1):
• Wikidata (WD, W) [
The different rounds of SemTab 2022 have been organised to evaluate participating systems on various datasets with variable difficulty. Unlike the previous editions of SemTab where the rounds were run with the support of AIcrowd, this year, we asked the participants to submit their solutions once per week via a Google Form. We created a submission form for each round and we evaluated the submitted results at the beginning of each week during each round.
The different datasets used to run SemTab 2022 rounds are reported in Table 1, with some
statistics available in Table 2. Unlike the previous editions where the ground truth was hidden
from the participant, this year, we provided a partial ground truth data to the participants during
the challenge itself. The teams could use these ground-truth labels for evaluating their methods
locally. Thus, we report datasets’ statistics per split. All the datasets are available in Zenodo as
follows:
• Tough Tables (2T) [
Link: https://doi.org/10.5281/zenodo.7419275
• HardTables (HT) [
Link: https://doi.org/10.5281/zenodo.7416036
• BiodivTab [
Original tables have been adapted for the SemTab challenge. This year featured DBpedia as a target KG instead of Wikidata
Link: https://doi.org/10.5281/zenodo.7319654
• GitTables [
Link: https://zenodo.org/record/7091019 4Target column train and test splits for CTA are distributed across all tables.
# Tables
# CTA Targets # CPA Targets
As per the previous editions, systems have been evaluated on a single annotation for each provided target, for all the tasks; i.e., in CEA, target cells are to be annotated with a single entity from the target KG; in CTA, target columns are to be annotated with a single type from the target KG (as ifne-grained as possible).
5.54 2.56 5.57 2.6
2T Round 2
36 683.3 1, 373 4.31 4.42 81, 126 177, 453 97 111
NA
NA
NA
NA where target annotations refer to the target cells for CEA, the target columns for CTA, and the target column pairs for CPA. We consider an annotation as correct when it is included within the ground truth set (a target cell usually has multiple annotations in the ground truth, because of redirect and same-as links in KGs).
Given the fine-grained type hierarchy in Wikidata, we adopted approximations of Precision and
Recall in the CTA evaluation [
⎧⎪0.8( ), if is in GT, or an ancestor of the GT, with ( ) ≤ 5 cscore( ) = ⎨0.7( ), if is a descendant of the GT, with ( ) ≤ 3
⎪⎩0, otherwise; where ( ) is the shortest distance to one of the GT classes (as for CEA, also CTA GT columns may have multiple classes). For example, ( ) = 0 if is a class in the ground truth (cscore( ) = 1), and ( ) = 2 if is a grandchild of a class in the ground truth (cscore( ) = 0.49). Types in the higher level(s) of the KG type hierarchy are not considered in the GT (e.g., Q35120 [entity] in Wikidata). Given the correctness score , approximated Precision (AP), Recall (AR), and F1-score (AF1) for the CTA evaluation are as follows: 5 7 NA 7
NA 5 4 NA 5 (1) (2) 5 6 =
∑︀ ( ) |System Annotations| , =
∑︀ ( )
|Target Annotations|
, 1 =
2 × ×
+
(3)
2.1.4. Results
that the dataset is almost the same as in SemTab 2020.5 The BiodivTab and GitTables datasets
brought additional complexity in Round 3, highlighting that real-world tables are challenging.
CEA task. Results for the CEA task are reported in Figure 1 for all the datasets. In Round 1,
almost all the systems performed well on the HardTables dataset (automatically generated).
Starting from Round 2, datasets have been selected to increase the number of tricky cases to solve.
We used a new version of HardTables, mainly built of tables that we knew participants failed to
annotate in previous rounds and editions, in addition to Tough Tables. As expected, performance
on HardTable decreases, showing that systems still fail in annotating tricky cases (even if they
have been already seen); by comparing results on Tough Tables with previous SemTab editions,
more systems achieved an F1-score over 0.8, possibly because of the availability of the validation
set. In this round, systems focused on Wikidata-based datasets; indeed, just 4 out of 8 systems
submitted a solution for the DBpedia-based Tough Table dataset. In Round 3, the complexity
brought by the (relatively small) tables in the BiodivTab dataset still represents a new problem
to solve, showing a reduced performance by all systems. However, the scores are generally
improved compared to the last year’s version of such a dataset [
CTA task. In Figure 2, the results in the CTA task resemble the trend already seen from the CEA
results. This is an indicator that most of the systems solve the CTA task based on annotations
found in the CEA. Additional challenges have been included in Round 3 with the GitTables
dataset, where we can see a critical performance drop for all the involved systems. The column
type annotations from DBpedia seem harder than annotations from Schema.org. However, the
performance of systems for both ontologies improved compared to last year [
This new track aims at addressing applications in real-world settings that take advantage of the
output of the matching systems. Challenging dataset proposals have also been accepted and may
be included in future editions of SemTab.
2.2.1. Results
We opened the call for this track on 25 August 2022. We received four submissions, three of
them were accepted and the fourth one was accepted as a poster (2 pages) since it is still at a
0.8
preliminary stage. We give an overview of each of them as follows:
• Wikary [
It proposes a promising expansion from binary to n-ary tuples to reflect the conditional
properties of certain (object, predicate, subject) predicates. This will likely be helpful for
matching tables with n-ary entities and lower KG coverage. Wikary contributes a large,
diverse, multilingual and metadata-rich set of Wikipedia tables that are matched to n-ary
statements and qualifiers from Wikidata. A subset of the dataset underwent a manual
quality evaluation by asking annotators to annotate some tables.
• MammoTab [
In 2021, SemTab included a new track focusing on system usability. The main goal of this track was to mitigate a pain point in the community: the lack of publicly available, easy-to-use, and generic solution to address the needs of a variety of applications and settings. In 2022, the usability track was replaced by the Artifact Availability Badge, that applies to both tracks in 2022. This badge is given to submissions (regardless of the track) if all dependencies are verified to be accessible and sufficiently reusable. The goal of this badge is to motivate authors to publish and document their code and data, so that others can (re)use these artifacts and potentially reproduce the results.
The criteria used to assess submissions for eligibility of the Artifacts Availability Badge are: • Publicly accessible data (if applicable). • Publicly accessible source code. • Clear documentation of the code and data.
• Open-source dependencies. 2.3.2. Results Almost all core participants obtained good results in this track, by performing well on at least two criteria (open-source code and clear documentation). We report the evaluation details in Table 5. In general, tools requirements vary in complexity, but they are reasonable overall (e.g., pre-processing required, like creating new indexes or embeddings). JenTab provides its pre-computed lookups and indices, thus, it has a checkmark under open-source data. Considering the other criteria, the evaluation panel concluded that most of the tools are pre-configured and can potentially be used out of the box: for example, JenTab has been packaged with Docker to ease deployment on local premises. In addition, JenTab is the only system released as open source under a permissive license (Apache 2.0).
For the Datasets Track, all submissions have open-source data and code and vary in the details of the provided documentation. FCTables is an exception, since the released code under its GitHub repository is for the table parsing component only, and not for their entire pipeline. 2.4. Awards ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ 3 5 As in previous editions, the best systems in each track were awarded across different tracks: • Accuracy Track: KGCODE-TAB and DAGOBAH (1 prize) were the highest performing systems in most of the tasks, showing appreciable improvements over previous years. • Dataset Track: SOTAB won the first prize. • Artifacts Availability Badge: SOTAB and Wikary got the badge from the dataset track, while JenTab was the only system to get this badge.
Challenging HT Datasets. We have been using the same automated dataset generation process, with some variations that make it more challenging, since the first SemTab challenge. However, we applied a kind of filtration step to control the balance between the easy and hard cases. Unlike the previous editions of the challenge, this year the average F1-score is below 90% for the HT datasets (see Table 4). This proves the effectiveness of our filtration process and makes the dataset much harder to solve.
Ground-truth Quality. An accurate evaluation of systems is important for any benchmark or challenge and relies on the quality of the ground truth annotations. SemTab features diverse sets of tables that all are extracted and annotated differently. Some annotation procedures may yield inconsistent or erroneous annotations, which introduces noise in the development and evaluation of systems. Moreover, obtaining perfect ground truth for tables is hard as cell entities (CEA), column types (CTA), and column pair relations (CPA) could possibly correspond to multiple labels due to, for example, synonymy or hierarchy relations. In line with these challenges, a few participants indicated that some ground truth annotations in SemTab datasets might be questionable as well. This motivates an effort to improve the ground truth annotations for future editions, possibly in collaboration with the community, as suggested by a participant. Artifacts Availability Badge. The introduction of the Artifacts Availability Badge extends the Usability Track from SemTab 2021. It encouraged the participating systems to provide publicly accessible resources. Our goal was exactly to emphasize this, despite the competitive nature that a challenge may have. We note that the current conditions for receiving this badge may have been too restrictive, so we are considering providing more badges of a narrower scope in the future (e.g., one badge for publicly accessible code, one for reproducible results, etc). Dataset track. We believe that the call of the dataset track has grasped more attention from the community by introducing their own datasets. Compared to the last year, “Applications Track”, this year we received more contributions, four datasets, while last year we had only two submissions. Not all of them are ready to be used in the challenge, but they show a promising interest within the community. Such contributions from the community like Wikary and SOTAB help in extending the SemTab benchmark with new challenges that are hard to reproduce in synthetic datasets like HT. Thus, this new track has been an important addition to SemTab. Visibility & Increased Impact. SemTab gained an increasing and broader attention from the community. This year, before the official start of the challenge, we presented SemTab in the Knowledge Graph Construction (KGC)6 Workshop, co-hosted with the ESWC conference. In addition, SemTab has contributed for four years to the Ontology Matching (OM)7 workshop, co-hosted with the ISWC conference. Such dissemination activities, in combination with the new Datasets Track, resulted in more contributions to SemTab 2022 overall. The proceedings in 2022 contains 12 papers (vs. 8 in 2021) with four of them belonging to the Datasets track. This shows diverse artifacts from SemTab 2022. We also plan to continue the challenge and propose a workshop with KG matching specificity. Either by organizing our own workshop or by joining one of the existing ones.
We thank the challenge participants, the ISWC & OM organisers, and our sponsor IBM Research that played a key role in the success of SemTab. We also thank Paul Groth and Çag˘atay Demiralp for their contribution to GitTables, and Sirko Schindler and Birgitta König-Ries for their contribution to BiodivTab. This work was also supported by the SIRIUS Centre for Scalable Data Access (Research Council of Norway), Samsung Research UK, the EPSRC projects UK FIRES and ConCur, and the HFRI project ResponsibleER (No 969). Finally, we like to acknowledge that the organization was greatly simplified by using the EasyChair conference management system and the CEUR-WS.org open-access publication service. 6https://kg-construct.github.io/workshop/2022/ 7http://www.om2022.ontologymatching.org/