Semantic Table Annotation (STA) denotes the process of annotating a tabular dataset with concepts and relations from a given Knowledge Graph. The objective is to map individual table elements to their counterparts from the Knowledge Graph. The Semantic Web Challenge on Tabular Data to Knowledge Graph Matching (SemTab) aims to establish a common framework for systems that tackle the process of STA. Since 2019, it has provided a set of benchmarks each year for evaluation. However, most of the provided datasets in the rst two incarnations of the challenge are Automatically Generated (AG) and general domain datasets. This leaves the question open whether the developed systems can similarly be applied to real-world datasets that provide a di erent set of challenges. In this paper, we try to address this gap by introducing a domain-speci c benchmark named BiodivTab. It consists of 50 datasets based on real-world biodiversity research data that have further been augmented. BiodivTab was made available to SemTab participants during Round 3 in the 2021 edition of the challenge.
The Semantic Web Challenge on Tabular Data to Knowledge Graph Matching (SemTab)
has worked on establishing a community for Semantic Table Annotation (STA) tasks
over the course of so far three editions: 2019 [
Country
Area
Capital
Country
Area
Capital
Country
Area
Capital Egypt
1,010,408 Germany 357,386
Cairo Berlin
Egypt 1,010,408
Cairo
Egypt 1,010,408
Cairo Germany 357,386
Berlin
Germany 357,386
Berlin https://www.wikidata.org/wiki/Q79 https://www.wikidata.org/wiki/Q183 https://www.wikidata.org/wiki/Q6256 https://www.wikidata.org/wiki/Q5119 (a) CEA (b) CTA (c) CPA
The challenge establishes common standards for systems that tackle the problem of
STA. Among the best-performing participants from the 2020 are MTab4Wikidata [
The ultimate goal is systems that can annotate real-world datasets. However, the
datasets introduced in the rst two years of the challenge are Automatically
Generated (AG) derived from di erent KGs [
In this paper, we introduce a domain-speci c tabular benchmark named BiodivTab.
We have collected real tables from the biodiversity domain and manually annotated
them using the live edition of Wikidata [
In this section, we explain the creation steps of BiodivTab, the data sources we have used, and the biodiversity-speci c challenges encountered. Moreover, we describe the annotation phase, the data augmentation step, and the nal assembly of the benchmark.
The included datasets were collected from three public repositories of biodiversity
data: data.world5, BEFChina6 [
a large collection of biodiversity-related datasets and veri ed their licenses8 to ensure that they allow for the use in such a benchmark. Subsequently, we manually checked each of them concerning their suitability to the semantic table annotation tasks. In the process, we discarded datasets that predominantly contained, e.g., internal database \ID" columns, generic headers (e.g., \BEX 12"), or numerical columns without any further explanation or context. We consider those datasets next to impossible to annotate automatically and of little bene t to the community. Consequently, we decided to include only datasets containing a substantial amount of categorical information.
The datasets collected this way feature unique characteristics that can be summarized as follows: { Specimen Data: The collected datasets contain observations of a particular specimen, e.g., a speci c individual of a given species. This includes a multitude of properties of the specimen and their particular environment. The assembled data can only rarely be attributed to the general species. { Numerical Data: Most of the collected datasets describe the specimen by various measurements in numerical form. { Abbreviations: Species names may be given in an abbreviated format. For example, \Canna glauca", a particular kind of ower, is often referred to as \C.glauca" or \Ca.glauce". { Special Format : Species names are identi ed by a combination of species and subspecies. For example, \species:Atrichum sub:subserratum" may be used instead of \Atrichum subserratum".
The identi ed challenges increase the di culty during the semantic annotation process. Commonly, data is characterized by a single subject column (usually the leftmost one) with a group of other columns representing properties to the respective subject. Given that most of the data represent specimen data, those numerical/object elds do not necessarily relate to the general properties of species. As of the time of writing, Wikidata, the target KG of BiodivTab, contains no direct equivalent to the specimen data contained in the selected datasets. Thus, we could not annotate column-relations in the fashion of a CPA-task. As a consequence, BiodivTab does not include a CPAtask as of now. However, it might change in the future, if other KGs are supported, or Wikidata is extended accordingly. Instead, we provide ground truths only for CEA and CTA tasks. The special format found in the species names impedes the direct matching of cell values to labels of individual entities in the KG. One possible approach might be to handle these cases with a particular variety of misspellings. In fact, part of the cell values might be considered additional noise that has to be removed before matching.
After the data collection phase, we picked 13 tables to use in the annotation
process. Our naming convention follows the schema of \dataSource id", e.g., \befchina 1"
represents the rst dataset collected from BEFChina. The annotation itself is the most
time-consuming part of the benchmark creation. To ensure the quality of mappings, we
manually annotated the selected tables with entities assembled from Wikidata during
September 2021, resulting in ground truth data for both CEA and CTA tasks.
However, another set of annotations and checking the Inter-annotator Agreement [
Licenses.md ones to disambiguate the cell's semantics. If this still leaves more than one candidate, we keep them all and consider them true matches. Consequently, the provided ground truth contains all possible candidates that di erent systems could generate. We followed the same procedure for CTA. We maintain separate ground truth les to ease manual inspection, revision, and quality assurance for each table. E.g., \befchina 1" is annotated by two such les: \befchina 1 CEA" and \befchina 1 CTA". Biodiversity experts have partially revised these annotations. The structure of the ground truth les follows the format of SemTab. In particular, the solution les for CEA use a format of lename, column id, row id, and ground truth, whereas the ones for CTA employ a structure of lename, column id, and ground truth.
To increase the number of tables in our benchmark and reduce the human e ort needed, we further resorted to data augmentation. It is a technique to increase the amount of data by adding slightly modi ed copies of already existing entries. In our context, we introduced challenges to the existing dataset based on our ndings during the data collection and analysis phase (see the rst paragraph). Since abbreviations are a common issue in biodiversity datasets, a column containing full species names was thus abbreviated accordingly. This strategy allowed us to (i) increase the number of included tables to 50 (almost 4 the number of just real tables), (ii) reduce the required human e ort during annotation, and (iii) produce a benchmark that relies on real challenges instead of arti cial ones.
The benchmark dataset consists of a set with 13 real tables and 37 augmented ones. We anonymized the le names of tables to use unique identi ers using Python's uuid functionalities in the process. Subsequently, we aggregated the individual solutions of CEA and CTA into one le per task resulting in CEA biodivtab 2021 gt.csv and CTA biodivtab 2021 gt.csv, respectively. From this, we generated corresponding \target- les" by removing the ground truth columns from these solution les. The benchmark tables and targets can now be published during the challenge for participants to solve. This follows the general approach of SemTab that hides the ground truth of STA tasks from participants during the challenge.
In 2021, the organizers published a call for domain-speci c benchmarks to be used in that year's challenge. BiodivTab was submitted and accepted as one of these benchmarks. As a result, BiodivTab represented one of three benchmarks posed during the third round of 2021's SemTab challenge. 3
We have introduced a tabular benchmark derived from biodiversity research data
named BiodivTab. It consists of a collection of 50 tables. We have created BiodivTab
by manually annotating 13 tables from real-world biodiversity datasets and adding 37
more tables by augmenting them with noise based on previously observed challenges.
BiodivTab was submitted to and subsequently used in Round 3 of the 2021 SemTab
challenge. Our benchmark is publicly available [
Future Work We see multiple directions to continue this work. We plan to include more biodiversity tables from other projects to cover a broader spectrum of the domain. In addition, ground truth data from other KGs, in particular domain-speci c ones, can be provided.
The authors thank the Carl Zeiss Foundation for the nancial support of the project \A Virtual Werkstatt for Digitization in the Sciences (P5)" within the scope of the program line \Breakthroughs: Exploring Intelligent Systems" for \Digitization - explore the basics, use applications". We would like to especially thank our Biodiversity experts Cornelia Furstenau and Andreas Ostrowski for feedback and validation of the created annotations. Last but not least, we would like to thank Samira Babalou for the fruitful discussions during the work.
The tables provided in this challenge are based on real-world biodiversity research datasets, but have been adapted for the challenge. In the form provided here, they may be used for the challenge, only. Any publication on challenge results needs to contain citations of the underlying datasets. The list of original datasets is available within our GitHub repository.