SemTab 2021 was the third edition of the Semantic Web Challenge on Tabular Data to Knowledge Graph Matching, successfully collocated with the 20th International Semantic Web Conference (ISWC) and the 16th Ontology Matching (OM) Workshop. SemTab provides a common framework to conduct a systematic evaluation of state-of-the-art systems.
Data in tabular format are the most frequent input to data analytics pipeline, thanks to
their high storage and processing efficiency. Also, the tabular format allows users to
represent the information in a compacted way, by exploiting the clear data structure
deifned by rows and columns. However, such clear structure does not imply a clear
understanding of the semantic structure (e.g., relationships between columns), as well as the
meaning of the content (e.g., if data are about a specific topic). The lack of
understanding hinders data analytics processes, requiring additional effort to properly understand
the data first. Gaining the semantic understanding is valuable for many applications,
including data cleaning, data mining, data integration, data analysis and machine
learning, and knowledge discovery. For example, the semantic understanding can help in
assessing what kind of transformations are more appropriate for a dataset, or which
datasets can be integrated to enable new analytics (e.g., marketing analysis) [
In addition to their efficiency, the huge availability of tabular data on the Web makes Web tables a valuable source to consider for data miners (e.g., open data CSV files). Adding semantic information to Web tables is useful for a wide range of applications, including web search, question answering, and knowledge base construction.
Tabular data to Knowledge Graph (KG) matching is the process of clarifying the semantic meaning of a table by mapping its elements (i.e., cells, columns, rows) to semantic tags (i.e., entities, classes, properties) from KGs (e.g., Wikidata, DBpedia). The task difficulty increases when table metadata ( e.g., table captions, table description, or column names) being missing, incomplete or ambiguous.
The tabular data to KG matching process is typically broken down into the following tasks: (i) cell to KG entity matching (CEA task), (ii) column to KG class matching (CTA task), and (iii) column pair to KG property matching (CPA task).
Over the last decade several approaches made advances in addressing one or several
of above tasks, also constructing benchmark datasets ([
The SemTab 2021 challenge has been organised into 3 different tracks: the Accuracy Track, which is the standard track proposed in previous editions; the Usability Track, a new track addressing the lack of publicly available, easy-to-use and generic solutions; and the Applications Track, which focuses on applications in real-world settings where the output of matching systems can contribute. The application track was also open to the submission of novel benchmark datasets. 2.1
The Accuracy Track included 3 rounds, running from June 30 to October 15. Different
target KGs were used across rounds (see Table 1):
– DBpedia [
The different rounds of SemTab 2021 have been organised to evaluate participating
systems on different datasets with variable difficulty. All the rounds were run with the
support of AIcrowd;4 SemTab 2021 also used the STILTool system [
Link: https://doi.org/10.5281/zenodo.6211551 – BioTable: a dataset focused on molecular biology data covering different entities.
It has the larges number of rows per table in the challenge.
Link: https://doi.org/10.5281/zenodo.5606585 – Automatically Generated (AG):5 a synthetic dataset with tables generated automatically by means of SPARQL queries. AG is the largest dataset used in SemTab.
Link: https://zenodo.org/record/6154708 – BiodivTab: a dataset with tables from real-world biodiversity research datasets.
Original tables have been adapted for the SemTab challenge.
Link: https://doi.org/10.5281/zenodo.5584180 – GitTables: a large-scale corpus of relational tables extracted from CSV files in GitHub. The main purpose of this dataset is to facilitate learning table representation models and applications in e.g., data management. A subset of tables has been curated for benchmarking column type detection methods in SemTab. Link: https://doi.org/10.5281/zenodo.5706316 Evaluation measures As per the previous editions, systems have been evaluated on a single annotation for each provided target, for all the tasks; i.e., in CEA, target cells are to be annotated with a single entity from the target KG; in CTA, target columns are to be annotated with a single type from the target KG (as fine-grained as possible). 5 In SemTab 2021, also referred to as Hard Tables. 6 AIcrowd leaderboard scores 23 participants because of test submissions. where target annotations refer to the target cells for CEA, the target columns for CTA, and the target column pairs for CPA. We consider an annotation as correct when it is included within the ground truth set (a target cell usually has multiple annotations in the ground truth, because of redirect and same-as links in KGs).
Given the fine-grained type hierarchy in Wikidata, we adopted approximations of Precision and Recall in the CTA evaluation. Approximations adapt their numerators to consider partially correct annotations, i.e., annotations that are ancestors or descendants of the ground truth (GT) classes. The correctness score cscore of a CTA annotation α considers the distance between the annotation and the GT classes in the type hierarchy, and it is defined as
0.8d(α ), if α is in GT, or an ancestor of the GT, with d(α ) ≤ 5 cscore(α ) = 0.7d(α ), if α is a descendant of the GT, with d(α ) ≤ 3 0, otherwise; (2) where d(α ) is the shortest distance to one of the GT classes (as for CEA, also CTA GT columns may have multiple classes). For example, d(α ) = 0 if α is a class in the ground truth (cscore(α ) = 1), and d(α ) = 2 if α is a grandchild of a class in the ground truth (cscore(α ) = 0.49). Types in the higher level(s) of the KG type hierarchy are not considered in the GT (e.g., Q35120 [entity] in Wikidata). Given the correctness score cscore, approximated Precision (AP), Recall (AR), and F1-score (AF1) for the CTA evaluation are as follows: AP =
|System Annotations| , AR =
|Target Annotations| , AF 1 = 2 × AP × AR
AP + AR (3) Results Table 4 contains the average F1-score achieved by the 11 participating systems. The Tough Tables dataset still represent a challenge for almost all the systems, specially considering the fact that the the dataset is the same as in SemTab 2020. The BiodivTab and GitTables datasets brought additional complexity in Round 3, highlighting that realworld tables are challenging.
CEA task. Results for the CEA task are reported in Figure 1 for all the datasets. The Round 1 used the same 2T tables from last year edition,7 raising the difficulty bar at the very beginning. Most of the systems faced important challenges when dealing with 2T tables, with only 2 systems managing to achieve an F1-score over 0.8 and several of them participating in only one of the tasks. It is worth noting the work of the DAGOBAH team, which improved their system over the last year, being able to achieve higher scores on 2T this year. Starting from Round 2, systems have been evaluated on datasets never seen before. The AG datasets aimed at bringing new challenges in each round, and we can observe than only the best systems managed to maintain almost the same score on the two different versions of this dataset. Concerning bio-related datasets, performance in Round 2 were positive (slightly below 0.9 on average), confirming that tables with many rows (∼ 2,500 on average) do not represent a problem for most of all the systems. Instead, the complexity brought by the (relatively small) tables in the BiodivTab dataset represented a new problem to solve, showing significantly reduced 7 The Wikidata targets have been updated to the current Wikidata live version. 1.0 0.8 roe0.6 c S 1 F0.4 0.2 1.0 0.8 re0.6 o c S -1F0.4 0.2 0.0 MTab MAGIC DAGOBAH MantisTable V JenTab Kepler-aSI
MTab MAGIC DAGOBAH JenTab Kepler-aSI
R1 2T-DBP
R1 2T-WD
R2 AG
R2 BIO
R3 AG
R3 BIODIV performance (none of the systems scored over 0.6). The JenTab system ranked 1st over a very difficult dataset. It is worth noting, however, that members of the JenTab team are also the providers of the BiodivTab dataset.
CTA task. As shown in Figure 2, the results in the CTA tasks resemble the trend already seen from the CEA results. This is an indicator that most of the systems solve the CTA tasks based on annotations found in the CEA. Additional challenges have been included in Round 3 with the GitTables dataset, where we can see a critical performance drop for all the involved systems. It is worth emphasising that, given the general picture provided by the results in CTA, more research is needed to make existing systems able to deal with real-world tables, where the cells may be missing a correspondence to the target KG.
CPA task. Results for the CPA tasks are plotted in Figure 3. Currently, only BioTables and the AG datasets provide a GT for CPA. Results are overall positive for all the tasks, with a general improvement from Round 2 to Round 3 for all the involved systems, except for MAGIC, whose performance dropped a bit during the last round. 1.00 0.95 e r co0.90 S 1 F 0.85 0.80 MTab MAGIC DAGOBAH MantisTable V JenTab Kepler-aSI R2 AG
R2 BIO
R3 AG Starting from SemTab 2021, the organisation committee agreed to include a new track focusing on system usability. The main goal of this track is to mitigate a pain point in the community: the lack of publicly available, easy-to-use, and generic solution that will address the needs of a variety of applications and settings.
Evaluation measures Deeply evaluating the usability of a system requires user studies
to monitor different parameters [
The evaluation panel concluded that most of the tools are pre-configured and can potentially be used out of the box: for example, JenTab has been packaged in Docker containers to ease the deployment and execution of the tool on local premises. In general, tools requirements vary in complexity, but they are reasonable overall (e.g., preprocessing required, like creating new indexes or embeddings).
Considering the other criteria, JenTab is the only system released as open source under a permissive license (Apache 2.0). The MTab tool has been made publicly available as a Web service, free to use (MIT license); but the back-end application has not been disclosed. However, having a public API enables MTab serving third-party application (with no rate limit), and this was a key point in declaring MTab the most usable tool. Systems like DAGOBAH and MantisTable delivered a framework with impressive GUIs, while others (e.g., MAGIC) opted for a lightweight application. This new track aims at addressing applications in real-world settings that take advantage of the output of the matching systems. Challenging dataset proposals have also been accepted and included within the SemTab 2021 rounds.
Results A specific application has been identified within the biological domain, where new data are constantly produced thanks to the advances in the field. The domain is particularly challenging from the semantics standpoint because of the the complexity of the biological relations between entities. Within SemTab, the data representation significantly impact the systems performance since entities are usually represented by codes (e.g., chemical formulas or gene names). Two different datasets have been submitted related to the biological domain; the first one, BioTables, is a dataset focused on molecular biology data; the second, BiodivTab, is a dataset focused on biodiversity research data and data augmentation.
Along side the above domain, a different dataset has been submitted to this track and also included in Round 3, GitTables. This dataset includes relational tables extracted from CSV files hosted at GitHub, and it comes with a peculiarity: the GT for CTA uses a mixture of classes and properties to annotate columns (both for the DBpedia and Schema.org versions).
The three datasets brought new complexity and contributed to increment the data diversity among the SemTab benchmark datasets. 2.4
As in previous editions, IBM Research8 sponsored SemTab 2021 and awarded the best systems in each track with the following prizes: – Accuracy Track: DAGOBAH (1st prize) was the top system in most of the tasks, showing appreciable improvements over the last years. Honorary mention to MTab – Usability Track: MTab team (1st prize), for providing the easy-to-use MTab tool9 along with Web services to lookup entities and annotate tables; JenTab (2st prize), for being the only open-source system with a permissive license. Honorary mentions to DAGOBAH, MAGIC and MantisTable. – Applications Track: BiodivTab dataset (1st prize), for having brought new challenges in CEA and CTA tasks. Honorary mention to GitTables. 3
Avoiding over-fitting to AG. We have been using the same automated dataset generation process, with some variations that make it more challenging, since the first SemTab challenge. This may be resulting to participating systems that explicitly target datasets with characteristics similar to those of the AG datasets. This becomes evident from the almost perfect results shown in Table 4. For that reason, this year we have introduced several new datasets, while we are also planning to use as much as possible real data, rather than synthetic, in the future versions of the challenge.
System generalizability beyond KGs. Many systems currently rely on matching table
values to entities in KGs. In this version of SemTab, we challenged the participating
systems on their ability to detect the semantic types of table columns even when their
values are not linkable to KG entities. We conclude that most systems do not generalize
well in this scenario as indicated by the performance drop on the CTA task for GitTables
(see Section 2.1). Improving systems to this end would make them useful for expanding
KG coverage by matching tables from novel data sources to KGs in order to populate the
8 https://www.research.ibm.com/
9 https://github.com/phucty/mtab tool
“unknown unknowns” [
Usability track. We believe that the introduction of the usability track has contributed
to making participating systems publicly accessible. Our goal was exactly to encourage
this, despite the competitive nature that a challenge may have. Thus, we consider this
new track to be a very important one and we are planning to keep it in the next
challenges. Next SemTab editions may consider to improve the evaluation of this track,
for example by adopting the System Usability Scale (SUS) [
We would like to thank the challenge participants, the ISWC & OM organisers, the AIcrowd team, and our sponsor IBM Research that played a key role in the success of SemTab. We also thank Paul Groth and C¸ ag˘atay Demiralp for their contributions to GitTables. Moreover, we would like to thank Sirko Schindler and Birgitta Ko¨nig-Ries for their contribution to BiodivTab. This work was also supported by the SIRIUS Centre for Scalable Data Access (Research Council of Norway), Samsung Research UK, the EPSRC projects UK FIRES and ConCur, and the HFRI project ResponsibleER (No 969). DO and CP were supported by FCT through LASIGE (UIDB/00408/2020 and UIDP/00408/2020). We would also like to acknowledge that the work of the challenge organisers was greatly simplified by using the EasyChair conference management system and the CEUR-WS.org open-access publication service.