SemTab 2020 was the second edition of the Semantic Web Challenge on Tabular Data to Knowledge Graph Matching, successfully collocated with the 19th International Semantic Web Conference (ISWC) and the 15th Ontology Matching (OM) Workshop. SemTab provides a common framework to conduct a systematic evaluation of state-of-the-art systems. ? Copyright c 2020 for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0). 1 AIDA project: https://www.turing.ac.uk/research/research-projects/ artificial-intelligence-data-analytics-aida
Tabular data in the form of CSV files is the common input format in a data analytics pipeline. However a lack of understanding of the semantic structure and meaning of the content may hinder the data analytics process. Thus gaining this semantic understanding will be very valuable for data integration, data cleaning, data mining, machine learning and knowledge discovery tasks. For example, understanding what the data is can help assess what sorts of transformation are appropriate on the data.1
Tables on the Web may also be the source of highly valuable data. The addition of semantic information to Web tables may enhance a wide range of applications, such as web search, question answering, and knowledge base (KB) construction.
Tabular data to Knowledge Graph (KG) matching is the process of assigning semantic tags from Knowledge Graphs (e.g., Wikidata or DBpedia) to the elements of the table. This task however is often difficult in practice due to metadata (e.g., table and column names) being missing, incomplete or ambiguous.
Tabular data to KG matching tasks typically include (i) cell to KG entity matching (CEA task), (ii) column to KG class matching (CTA task), and (iii) column pair to KG property matching (CPA task).
There existed several approaches that aim at addressing one or several of above tasks
and datasets with ground truths that can serve as benchmarks (e.g., [
The SemTab 2020 challenge started on May 26 and closed on October 27. The target KG in this edition was Wikidata [18]: – Wikidata Dump (April 24, 2020): https://doi.org/10.5281/zenodo. 4282941
SemTab 2020 was organised into four evaluation rounds where we aimed at
testing different datasets with increasing difficulty. Rounds 1-3 were run with the support
of AIcrowd5, which provided an automatic evaluation of the submitted solutions, and
relied on an automatic dataset generator [
Table 2 shows the participation per round. We had a total of 28 different
systems6 participating across the four rounds; however only 11 systems have produced
results in 3 or more rounds, which we identify as the SemTab 2020 core
participants:7 MTab4Wikidata [15], LinkingPark [
The evaluation measures for CEA and CPA are the standard Precision, Recall and F1-score, as defined in Equation 1:
P = jCorrect Annotationsj ; R = jCorrect Annotationsj ; F 1 = 2 jSystem Annotationsj jTarget Annotationsj
P P + R
R (1) where target annotations refer to the target cells for CEA and the target column pairs for CPA. An annotation is regarded to be correct, if it is contained in the ground truth. Note that it is possible that one target cell or column pair may have multiple annotations in the ground truth.
For the evaluation of CTA, we used approximations of Precision and Recall, by adapting their numerators to consider as partially correct annotations, columns annotated with one of the ancestors or descendants of the ground truth (GT) classes. In that sense, we define the correctness score cscore of a CTA annotation as 8>0:8d( ); if is in GT, or an ancestor of the GT, cscore( ) = <0:7d( ); if is a descendant of the GT, (2) >:0;
otherwise; where d( ) is the shortest distance to one of the ground truth classes. For example, d( ) = 0 if is a class in the ground truth, and d( ) = 2 if is a grandparent of a class in the ground truth. In the former case, the scoring of will be 1, while in the later 0:64. Then, our approximations of Precision (AP), Recall (AR), and F1-score (AF1) for the evaluation of CTA are computed as follows: AP =
jSystem Annotationsj ; AR =
jTarget Annotationsj ; AF 1 = 2
AP AR AP + AR MTab4Wikidata LinkingPark MantisTable
SSL DAGOBAH JenTab bbw AMALGAM LexMa
CEA task. Figure 1a shows the results for the CEA task over the AG datasets. The dataset in each round aimed at bringing new challenges. This is somehow reflected in MantisTable, LinkingPark, LexMa and SSL. Nevertheless, the overall results for the AG datasets where very positive. In contrast, as shown in Figure 1b, the performance over the 2T dataset is significantly reduced where only three systems (MTab4Wikidata, bbw and Linkingpark) managed to maintain and F1-score over 0.8. It is worth mentioning the performance of the LexMa system in the 2T dataset, where it ranked 4th, while it was among the last ranked systems with the AG dataset.
CTA task. As shown in Figure 2, the results in the CTA tasks are relatively similar to the CEA task, where the average performance against the AG datasets is very good. However, the F1-score with the 2T dataset is dramatically reduced for all the systems (see Figure 2b). It is worth emphasising a general improvement from Round 1 to Round 2 (see Figure 2a).
CPA task. Figure 3 summarises the results for the CPA tasks over the AG datastes. Note that, currently, the 2T dataset is only available for the CEA and CTA tasks. It 1.00 0.95 bbw AMALGAM 1.00 0.95 0.90 0.85 0.80 0.75 1.0 0.9 0.8 roe0.7 cS1 F0.6 0.5 0.4 is worth mentioning the improvement of systems like JenTab and bbw in Round 4. MTab4Wikidata, LinkingPark and DAGOBAH maintained a constant performance during the last rounds. 2.3
Prizes IBM Research8 sponsored the prizes for the best systems in the challenge. This sponsorship was important not only for the challenge awards, but also because it shows a strong interest from industry.
– 1st Prize: MTab4Wikidata was the top system in all tasks and the least impacted by the 2T dataset. 8 https://www.research.ibm.com/ – 2nd Prize: LinkingPark had a very good and constant performance just below
MTab4Wikidata. – 3rd Prize: DAGOBAH and bbw. DAGOBAH had overall very positive results, apart from the CEA task in the 2T dataset. On the other hand, bbw had an outstanding performance in the last CEA round. 3
As in SemTab 2019 [
Data shifting. Several participants preferred the options of having a fixed target KG, given as a data dump, instead of using the SPARQL endpoints and related services to access the latest version of the KG. This concerns was specially important as Wikidata is continuously updated. This, however, may be challenging as it may require to “locally” store and process a large amount of data, and it may hinder the participation of systems that rely on online lookup services (e.g., DBpedia Lookup), since the exposed KG may differ from the fixed target KG.
Blind evaluation. Some participants proposed to have both a public and private
leaderboard in AIcrowd to cover the rounds with a blind evaluation. It is unclear if AIcrowd
provides this service, but systems like STILTool [
The user as a metric. In addition to the standard evaluation measures, it was proposed to include tasks that allow measuring the productivity from a user point of view. For example, a user may be more interested in a system that is easy to setup and run, than in a sophisticated system that provides better results but requires a non-trivial effort for installation or execution.
Complexity of annotations. In future editions, SemTab should also consider datasets that involve more complex annotations to reflect more realistic use cases. Domain-specific datasets. It was also proposed to use in the future domain-specific datasets and KGs (e.g., biomedical) as targets, in addition to cross-domain KGs like DBpedia and Wikidata and related datasets.
We would like to thank the challenge participants, the ISWC & OM organisers, the AIcrowd team, and our sponsors (SIRIUS and IBM Research) that played a key role in the success of SemTab. Special mention require Federico Bianchi and Matteo Palmonari who contributed to the creation of the Tough Tables dataset. This work was also supported by the AIDA project (Alan Turing Institute), the SIRIUS Centre for Scalable Data Access (Research Council of Norway), Samsung Research UK, Siemens AG, and the EPSRC projects AnaLOG, OASIS and UK FIRES. 15. P. Nguyen, I. Yamada, N. Kertkeidkachorn, R. Ichise, and H. Takeda. MTab4Wikidata at the SemTab 2020: Tabular Data Annotation with Wikidata. In Semantic Web Challenge on Tabular Data to Knowledge Graph Matching (SemTab). CEUR-WS.org, 2020. 16. R. Shigapov, P. Zumstein, J. Kamlah, L. Oberla¨nder, J. Mechnich, and I. Schumm. bbw: Matching CSV to Wikidata via Meta-lookup. In Semantic Web Challenge on Tabular Data to Knowledge Graph Matching (SemTab). CEUR-WS.org, 2020. 17. S. Tyagi and E. Jime´nez-Ruiz. LexMa: Tabular Data to Knowledge Graph Matching using Lexical Techniques. In Semantic Web Challenge on Tabular Data to Knowledge Graph Matching (SemTab). CEUR-WS.org, 2020. 18. D. Vrandecic and M. Kro¨ tzsch. Wikidata: a free collaborative knowledge base. Commun.
ACM, 57(10):78–85, 2014. 19. S. Yumusak. Knowledge graph matching with inter-service information transfer. In Semantic Web Challenge on Tabular Data to Knowledge Graph Matching (SemTab). CEUR-WS.org, 2020.