Systems that annotate tabular data semantically have witnessed increasing attention from the community in recent years; this process is commonly known as Semantic Table Annotation (STA). Its objective is to map individual table elements to their counterparts from a Knowledge Graph (KG). Individual cells and columns are assigned to KG entities and classes to disambiguate their meaning. STA-systems achieve high scores on the existing, synthetic benchmarks but often struggle on real-world datasets. Thus, realistic evaluation benchmarks are needed to enable the advancement of the field. In this paper, we detail the construction pipeline of BiodivTab, the first benchmark based on real-world data from the biodiversity domain. In addition, we compare it with the existing benchmarks. Moreover, we highlight common data characteristics and challenges in the field. BiodivTab is publicly available 1 and has 50 tables as a mixture of real and augmented samples from biodiversity datasets. It has been applied during the SemTab 2021 challenge, and participants achieved F1-scores of at most ∼ 60% across individual annotation tasks. Such results show that domain-specific benchmarks are more challenging for state-of-the-art systems than synthetic datasets.
e.g., by evaluation campaigns in other domains like semantic web services evaluations [
In this paper, we focus on one important domain: Biodiversity is the assortment of life
on Earth covering evolutionary, ecological, biological, and social forms. It is imperative to
monitor the current state of biodiversity and its change over time and understand the forces
driving it to preserve life in all its varieties. The recent IPBES worldwide evaluation3 predicts a
dramatic decrease in biodiversity, causing an obvious decay in vital ecological functions. An
expanding volume of heterogeneous data, especially tables, is produced and publicly shared in
the biodiversity domain. Tapping into this wealth of information requires two main steps: On
the one hand, individual datasets have to be fit for (re)use – a requirement that resulted in the
FAIR principles [
Our unique contributions in this paper over our previous work [16] are as follows: • Detailed explanation of the creation and data augmentation of BiodivTab. • An extensive discussion of idiosyncrasies and challenges in biodiversity datasets. • The creation of a new ground truth based on DBpedia. • A characterization of BiodivTab including concepts covered. • Evaluation of BiodivTab compared to other existing benchmarks.
• Applications of BiodivTab.
The remainder of this paper is organized as follows: Section 2 summarizes the required background. We detail the construction of BiodivTab in Section 3. Section 4 provides an evaluation of BiodivTab. Finally, we conclude in Section 5.
Semantic Table Annotation: The SemTab challenge has provided a community forum for
STA tasks over the course of so far four editions: 2019-2021 [
• Column Property Annotation (CPA) links column pairs using a semantic property.
Existing Benchmarks: The ultimate goal for STA-systems is to annotate real-world datasets.
However, the datasets introduced in the first two years of the challenge are synthetic derived
from diferent KGs [
Biodiversity Data Sources
benchmark
Orchis
Orchidinae
Lilium Lilioideae 2752977 Lilium Lilioideae 2752977 wwdd::QQ5116914761247 ((""OLriclihuims"")) twexdt:Q34740 ("genus")
(a) CEA (b) CTA Figure 1: STA-tasks as defined by SemTab using a biodiversity example5.
Orchis
Orchidinae
Lilium Lilioideae
wdt:P846 ("GBIF taxon ID")
2752977
(c) CPA
disambiguation of possible annotation solutions. The datasets employed, so far, adhere to no
particular domain but represented a sample from a wide range of general-purpose data. On the
other hand, domain-specific datasets pose specific challenges as witnessed, e.g., by evaluation
campaigns in other domains like semantic web services evaluations [
In this section, we explain the creation of BiodivTab, and the data sources used. Moreover, we describe the manual annotation phase involving biodiversity experts, the data augmentation step, and the final assembly and release of the benchmark. Figure 2 summarizes the construction of BiodivTab, we detail in the following.
We decided on three data repositories that are well established for the ecological data: BExIS6, BEFChina7, and data.world8. We queried these portals using 20 keywords, e.g., abandance, and species, from our previous work [21]. Subsequently, we manually checked all of them regarding their suitability to the STA-tasks. We discarded datasets that contained a majority of, 5We use the following prefixes throughout this paper: dbr: http://dbpedia.org/resource/, dbo: http://dbpedia.org/ ontology/, rdf: http://www.w3.org/1999/02/22-rdf-syntax-ns#, rdfs: http://www.w3.org/2000/01/rdf-schema#, wd: http://www.wikidata.org/entity/, wdt: http://www.wikidata.org/prop/direct/, and owl: http://www.w3.org/ 2002/07/owl# 6https://www.bexis.uni-jena.de/ 7https://data.botanik.uni-halle.de/bef-china/ 8https://data.world/ e.g., internal database “ID” columns or numerical columns without any explanation or context. We consider those datasets are impossible to annotate automatically and of little benefit to the community. Consequently, we decided to include only datasets containing essential categorical information. We selected 6 out of 32 dataset from data.world, 4 out of 15 from BExIS, and 3 out of 25 from BEFChina. data.world provides the most suitable datasets for STA, thus, it contributes about half of the datasets in BiodivTab. Our analysis of the collected data shows that, in addition to common challenges, real-world datasets feature unique characteristics. We enumerate the encountered challenges in our sample of datasets. We summarize their prevalence in Table 1. • Nested Entities: more than one proper entity in a single cell, e.g., a chemical compound is combined with a unit of measurements. • Acronyms: Abbreviations of diferent sorts are common, e.g., “Canna glauca”, a particular kind of flower, is often referred to as “C.glauca” or “Ca.glauce”. • Typos: Data is predominantly collected manually by humans, so misspellings will occur, e.g., “Dead Leav” is used for “Dead Leaves”. • Numerical Data: Most of the collected datasets describe the specimen by various measurements in numerical form. • Missing Values: Data collected can be sparse and may include gaps, e.g., a column “super kingdoms” may consist of “unknown” values for the most part. • Lack of Context: The collected data may barely provide any informative context for semantic annotations. e.g., a column with a missing or severely misspelled header. • Synecdoche: Scientists may use a general entity as a short form to a more particular one, e.g., “Kentucky” is used instead of “Kentucky River”. • Specimen Data: The collected datasets contain observations of particular specimens or groups, but do not pertain to the species as a whole.
The annotation phase is the most time-consuming part of the benchmark creation since it included multiple rounds of revision. To ensure the quality of mappings, we manually annotated the selected tables with entities assembled from the live edition of Wikidata during September 2021, resulting in ground truth data for both CEA and CTA tasks. Concerning CEA, we have marked possible candidate columns, typically those with categorical values, to annotate their cells. For each cell value, we assembled possible matches via Wikidata’s built-in search. We manually selected the most suitable matches to disambiguate the cells semantically if we found multiple matches. If we could not have chosen only one annotation, we pick all possible ones and consider them true matches. Thus, the provided ground truth contains all proper candidates for a given cell value. Biodiversity experts reviewed around 1/3 of the annotations. This revealed an error rate of about 1%. Because of the low error rate, the efort of this step outweighs the benefits. Thus, we have decided to continue annotating the remainder without further revisions.
We followed the same procedure for CTA. For categorical columns, we looked for a common type among column cells, taking into consideration the header value, to decide on the semantic type from Wikidata. Most of these columns are identified by the value of ( wdt:P31, instance of) as the perfect annotation. However, finding such perfect annotation for taxon-related columns is not that easy. Since all taxon-related fields are instance of taxon. We believed it might not be distinguishable enough. In the biodiversity domain, experts are keen on more fine-grained modeling. E.g., species, genus, and class would be diferent types in their opinion. We established a simple one-question questionnaire for our biodiversity experts to select the perfect semantic type for a given taxonomic term as shown in Table 2. The first column shows the cell values with the corresponding mapping entities. The question is to select either which type is the most accurate, A, or B. We derive Type A from (wdt:P105, taxon rank) and Type B from (wdt:P31, instance of) in Wikidata. Based on their answers, the most fine-grained classification is (Type A); however, they consider (Type B) as a correct type as well. Thus, we have selected the perfect types for taxons through (wdt:P105, taxon rank). For numerical columns, most of them are identified by the column headers.
We maintain separate ground truth files to ease manual inspection, revision, and quality assurance for each table. So, “befchina_1”, e.g., is annotated by two such files: “befchina_1_ CEA” and “befchina_1_CTA”. The structure of the ground truth files follows the format of SemTab challenge. In particular, the solution files for CEA use a format of filename, column id, row id, and ground truth, whereas the ones for CTA employ a structure of filename, column id, and ground truth.
We further used data augmentation to increase the number of tables in our benchmark and reduce the human efort needed. In our context, we introduced challenges to the existing datasets based on our findings during the data collection and analysis phase, thus we rely on real-world challenges that we added programatically to increase the amount of the data. Table 3 shows our used data augmentation techniques per dataset and the number of variations derived from it. In the following, we list techniques used and how they relate to the collected data issues: • Merge and Separate Columns we either by introduced new nested entities or splited them up into separate columns. • Add and Fix Typos we added noise to categorical cell values and, on rare occasions, fixed them. • Disambiguate we replaced concepts with more accurate ones, e.g., the state is replaced by the river it stands for. • Abbreviate we introduced more abbreviations especially with taxon-related values. • Alter Columns we removed one or more data columns. This results in less informative and sparse datasets.
We managed to create the most variations from data.world since its datasets contain more categorical data that can be mapped to KG entities. Our data augmentation strategy increased the number of tables to 50 with less manual efort of the annotation.
To enable approximation of CTA F1, Precision and Recall scores [
Initially, we collected all unique column types from manually assigned perfect annotations. These are used to initialize a dictionary. Afterwards, we ran a sequence of three SPARQL queries sent to the public endpoint to retrieve related classes for each of them. For the first level, we query for direct types via (wdt:P31, instance of). We call them “E1”. For the second level, we query for further parent classes via (wdt:P279, subclass of) of the previous E1, resulting in “E2”. For the third and last level, we repeat the last process using the entities in E2, yielding “E3”. If the initial column type is a class (e.g., wd:Q60026969, unit of concentration) we skip the first step and only use the latter two. The resultant hierarchy consists of one perfect annotation with up to three levels of classes that are considered okay annotations. For taxon-related columns, we marked the (wdt:P105, taxonRank) as perfect annotation to follow the biodiversity experts’ recommendation. However, we have included (wd:Q16521, taxon) and (wd:Q21871294, living organism) as okay classes.
For publication, we anonymized the file names of tables to use unique identifiers using Python’s uuid functionalities. Subsequently, we aggregated the individual solutions of CEA and CTA-tasks into one file per task resulting in CEA_biodivtab_2021_gt.csv and CTA_biodivtab_2021_gt.csv respectively. We generated the corresponding “target-files” by removing the ground truth columns from these solution files. We provided anonymized tables alongside the target files to evaluate a particular system. The ground truth files alongside the dictionary for related classes, CTA-ancestors, are subsequently used to evaluate the results. Such way this follows the general approach of SemTab hiding the ground truth of STA-tasks from participants during the challenge. BiodivTab is awarded the first prize of IBM Research 9 at the third round of 2021’s SemTab challenge [17] for its new challenges in CEA and CTA tasks.
In 202210, we included annotations from DBpedia that are based on the Wikidata annotations in two ways: First, we exploited the link between Wikidata entities and corresponding Wikipedia pages. As there is a one-to-one correspondence between Wikipedia pages and DBpedia entities, we generated a Wikidata-DBpedia-mapping for them. Second, we extracted owl:sameAs mappings between Wikidata and DBpedia to complete our mapping from DBpedia itself. Despite these direct mappings appeared promising to begin with, they contain serious data quality issues. As of April 2022, L-glutamic acid (wd:Q26995161) is mapped to 1772 entities within the DBpedia graph using owl:sameAs. Thus, the resulting mappings were again manually verified to ensure the overall quality of the final DBpedia ground truth data. Generated types for CTA contained only instances/resources from DBpedia. During the manual verification, we further added classes from the DBpedia ontology as well. We attempted to replicate our approach from Wikidata using rdf:type and rdfs:subClassOf to retrieve the CTA-ancestors. However, some relations in the DBpedia ontology seemed unreasonable to us. For example, DBpedia at the time of writing contains a triple dbr:Species rdf:type dbo:MilitaryUnit. For these and other similar scenarios, we decided to not include an ancestor file for DBpedia.
In this section, we give a detailed overview of BiodivTab in terms of the size and content compared to existing benchmarks. In addition, we show the most and least frequent types of CTA. Finally, we demonstrate the application of our benchmark using the results of STA-systems during SemTab’s 2021 edition.
Table 4 summarizes the selected datasets in terms of their original and selected size, and the number of CEA and CTA mappings. For large datasets, e.g., dataworld_4 and dataworld_27, we selected a subset of rows that retain the table characteristics. Most of the redundant species were dropped. Nevertheless, we kept the entire extent of BExIS datasets, including the redundant
10The new ground truth data from DBpedia is going to be used in SemTab 2022, thus we release a new benchmark after the conclusion of the challenge. entries, to achieve a good balance between the large tables and those with the reasonable length for STA-systems. The column mappings show the characteristic of specimen data, those columns with only local measurements and with local database names that could not be matched to the KG. For example, only 4 out of 18 columns in dataworld_1 could be matched to KG-entities.
Figure 3 shows the domains distribution of the 83 unique semantic types in the CTA-solutions.
Approximately two-thirds of these types belong to the biodiversity domain. The distinction
into the biodiversity-related, general domain, and mixed types was made according to the
definitions introduced in [ 21, 22]. General domain types include, e.g., visibility, scale, cost,
and airport. Mixed domain types contain examples like river, temperature, or sex of humans.
Biodiversity-related types include taxon, chemical compounds, and soil type. In addition, Table 5
provides a list of most and least frequent semantic types in BiodivTab. Species (wd:Q7432) is
the most frequent type, which reflects its importance in biodiversity research.
11Calendar year, wd:Q3186692, is equivalent to year, wd:Q577.
Wikidata, or both. T2Dv2 [
Table 7 shows a comparison between BiodivTab and existing benchmarks in terms of the average number of rows, columns, and cells. It also gives an overview of the targets for CEA, CTA, and CPA. BiodivTab is the smallest in terms of the number of tables. However, BiodivTab has the maximum average number of columns, and average number of rows except for SemTab 2021, Round 1, and BioTables in Round 2. This poses an additional challenge for STA systems. For CTA targets, BiodivTab is a middle point among the existing benchmarks.
Table 8 shows the scores from SemTab2021 top participants on BiodivTab and HardTables during Round 3. Scores have been published by the organizers of SemTab2021 [17]. The details about the mentioned systems using BiodivTab are beyond the scope of this paper. For BiodivTab, CEA has maximum F1-score by JenTab [20] of 60.2%, while the CTA has a maximum score with 59.3% by KEPLER [26]. In contrast, for the synthetic dataset, HardTables, DAGOBAH achieved the maximum F1-score 97.4%, and 99% for CEA, and CTA respectively. These results show that annotating real-world, domain-specific tables is far from solved by state-of-the-art STA-systems. This underlines the importance of benchmarks like BiodivTab further to foster the transfer of academic projects to real-world applications.
Resources should be easily accessible to allow replication and reuse. We follow the FAIR
(Findable, Accessible, Interoperable, and Reusable) guidelines to publish our contributions [
We introduced BiodivTab, the first biodiversity tabular benchmark for Semantic Table Annotation tasks. It consists of a collection of 50 tables. BiodivTab as created manually by annotating 13 Comparison with existing benchmarks. ST19 - ST21 (SemTab editions). *_W and *_D use Wikidata and DBpedia as targets. ST21-H2, and H3 are HardTables for Round 2 and 3 during SemTab2021. ST21-Bio is BioTables at SemTab2021 Round 2. ST21-Git is the published version of GitTables during SemTab2021 Round 3.
Future Work We see multiple directions to continue this work. We plan to include more biodiversity tables from other projects to cover a broader domain spectrum. We also plan to apply further quality checks of the annotations like multiple-annotators annotation and validation via the interrater agreement. In addition, we plan to provide ground truth data from other knowledge graphs, particularly domain-specific ones. Moreover, we analyze the performance of STA-systems on the BiodivTab.
The authors thank the Carl Zeiss Foundation for the financial support of the project “A Virtual Werkstatt for Digitization in the Sciences (P5)” within the scope of the program line “Breakthroughs: Exploring Intelligent Systems” for “Digitization - explore the basics, use applications”. We thank our biodiversity experts Cornelia Fürstenau and Andreas Ostrowski for feedback and validation of the created annotations. ing Principles for scientific data management and stewardship, Scientific data 3 (2016) 1–9. [15] L. M. Gadelha Jr, P. C. de Siracusa, E. C. Dalcin, L. A. E. da Silva, D. A. Augusto, E. Krempser, H. M. Afe, R. L. Costa, M. L. Mondelli, P. M. Meirelles, et al., A survey of biodiversity informatics: Concepts, practices, and challenges, Wiley Interdisciplinary Reviews 11 (2021) e1394. [16] N. Abdelmageed, S. Schindler, B. König-Ries, BiodivTab: A table annotation benchmark based on biodiversity research data, in: Proceedings of the Semantic Web Challenge on Tabular Data to Knowledge Graph Matching co-located with the 20th International Semantic Web Conference (ISWC 2021), Virtual conference, October 27, 2021, volume 3103 of CEUR, CEUR-WS.org, 2021, pp. 13–18. [17] V. Cutrona, J. Chen, V. Efthymiou, O. Hassanzadeh, E. Jiménez-Ruiz, J. Sequeda, K. Srinivas, N. Abdelmageed, M. Hulsebos, D. Oliveira, et al., Results of SemTab 2021, Proceedings of the Semantic Web Challenge on Tabular Data to Knowledge Graph Matching 3103 (2022) 1–12. [18] V.-P. Huynh, J. Liu, Y. Chabot, T. Labbé, P. Monnin, R. Troncy, DAGOBAH: Table and Graph Contexts For Eficient Semantic Annotation Of Tabular Data., in: SemTab@ ISWC, 2021. [19] P. Nguyen, I. Yamada, N. Kertkeidkachorn, R. Ichise, H. Takeda, SemTab 2021: Tabular
Data Annotation with MTab Tool., in: SemTab@ ISWC, 2021. [20] N. Abdelmageed, S. Schindler, JenTab Meets SemTab 2021’s New Challenges., in: SemTab@
ISWC, 2021. [21] N. Abdelmageed, A. Algergawy, S. Samuel, B. König-Ries, BiodivOnto: Towards a core ontology for biodiversity, in: European Semantic Web Conference (ESWC), Springer, 2021, pp. 3–8. [22] F. Löfler, V. Wesp, B. König-Ries, F. Klan, Dataset search in biodiversity research: Do metadata in data repositories reflect scholarly information needs?, PloS one 16 (2021) e0246099. [23] C. S. Bhagavatula, T. Noraset, D. Downey, Methods for exploring and mining tables on wikipedia, in: Proceedings of the ACM SIGKDD workshop on interactive data exploration and analytics, 2013, pp. 18–26. [24] M. J. Cafarella, A. Halevy, D. Z. Wang, E. Wu, Y. Zhang, Webtables: exploring the power of tables on the web, VLDB Endowment 1 (2008) 538–549. [25] M. Hulsebos, Çağatay Demiralp, P. Demiralp, Gittables for SemTab 2021 - cta task, 2021.
doi:10.5281/zenodo.5706316. [26] W. Baazouzi, M. Kachroudi, S. Faiz, Kepler-aSI at SemTab 2021., in: SemTab@ ISWC, 2021. [27] B. Steenwinckel, F. De Turck, F. Ongenae, MAGIC: Mining an Augmented Graph using
INK, starting from a CSV., in: SemTab@ ISWC, 2021. [28] R. Avogadro, M. Cremaschi, MantisTable V: A novel and eficient approach to Semantic
Table Interpretation., in: SemTab@ ISWC, 2021. [29] N. Abdelmageed, S. Schindler, B. König-Ries, fusion-jena/BiodivTab, 2022. doi:10.5281/ zenodo.6461556. [30] N. Abdelmageed, S. Schindler, B. König-Ries, fusion-jena/biodivtab: Benchmark data and code, 2021. doi:10.5281/zenodo.5749340.