Understanding the semantics of table elements is a prerequisite for many data integration and data discovery tasks. Table annotation is the task of labeling table elements with terms from a given vocabulary. This paper presents the WDC Schema.org Table Annotation Benchmark (SOTAB) for comparing the performance of table annotation systems. SOTAB covers the column type annotation (CTA) and columns property annotation (CPA) tasks. SOTAB provides ∼50,000 annotated tables for each of the tasks containing Schema.org data from diferent websites. The tables cover 17 diferent types of entities such as movie, event, local business, recipe, job posting, or product. The tables stem from the WDC Schema.org Table Corpus which was created by extracting Schema.org annotations from the Common Crawl. Consequently, the labels used for annotating columns in SOTAB are part of the Schema.org vocabulary. The benchmark covers 91 types for CTA and 176 properties for CPA distributed across textual, numerical and date/time columns. The tables are split into fixed training, validation and test sets. The test sets are further divided into subsets focusing on specific challenges, such as columns with missing values or diferent value formats, in order to allow a more fine-grained comparison of annotation systems. The evaluation of SOTAB using Doduo and TURL shows that the benchmark is dificult to solve for current state-of-the-art systems.
Benchmark
Tables containing structured data are widely used on the Web. Understanding the semantics of
tables is useful for a variety of data integration and data discovery tasks such as knowledge
base augmentation [
This paper presents the WDC Schema.org Table Annotation Benchmark (SOTAB) for comparing
the performance of table annotation systems on the CTA and CPA tasks. The CTA dataset
consists of 59,548 tables covering 17 Schema.org [
This section provides an overview of the existing benchmarks for evaluating table annotation
systems and compares them to SOTAB in Table 1. The ToughTables, HardTables, BioDiv and
GitTables datasets are used by the Semantic Web Challenge on Tabular Data to Knowledge Graph
Matching (SemTab) which is a benchmark competition for table annotation systems that takes
place every year as part of the International Semantic Web Conference [
This section describes the selection of the SOTAB tables from the WDC Schema.org Table Corpus, the assignment of lables to the tables, as well as well the selection of challenging columns into specific subset of the test set.
The WDC Schema.org Table Corpus was created using Schema.org data that was extracted from the December 2020 version of the Web Data Commons Microdata and JSON-LD corpus. The Schema.org Table Corpus consists of 4.2 million relational tables covering 43 Schema.org types. Each table in the corpus contains the descriptions of all entities of a specific Schema.org type that were extracted for a specific host, e.g. all movie records from imdb.com or product records from ebay.com. All extracted entities of one Schema.org type are collected per host and subsequently passed through a pipeline of processing and cleansing steps. For more details about the Schema.org Table Corpus we refer the reader to the project website1.
Table Selection. We begin building SOTAB with a language identification phase on the
tables from the Schema.org Table Corpus. The language identification phase aims to filter out
non-English rows. For this purpose, we use the fastText language identification model [
Label Generation. As the WDC Schema.org Table Corpus uses Schema.org properties as column headers, these properties can be directly used as labels for the CPA task. We derive the CTA label for a column from its CPA label using the Schema.org vocabulary definition which specifies the types that are allowed as property values. In cases where the vocabulary definition allows multiple types, a manual selection of the most appropriate type is done. Lastly, 1http://webdatacommons.org/structureddata/schemaorgtables/ we add some CTA annotations that are not included in the Schema.org vocabulary such as IdentifierAT , MusicArtistAT and Museum/name. This is done with the purpose of including more ifne-grained labels instead of for example simply name, so that an annotation system needs to better understand the semantics to select the correct annotation. After assigning the labels, another filtering step is performed based on label frequency: We only keep the columns that have CPA and CTA labels that are used at least 50 times.
Test Sets for Specific Challenges. In addition to the full test sets for both tasks, we provide subsets of the test sets that measure how good the systems can handle specific annotation challenges. We provide test sets for the following challenges: (i) Missing Values: This set contains columns having a value density between 10 and 70 percent and are thus harder to predict, (ii) Format Heterogeneity: columns whose values are represented using diferent value formats such as date or weight columns and (iii) Corner Cases: columns that are dificult to annotate as their values are very similar to the values of other columns. Examples of corner cases include startDate versus endDate and currenciesAccepted versus priceCurrency.
The selection and labeling phase results in 46,790 tables annotated for the CTA task and 48,379 tables annotated for the CPA task which cover 17 Schema.org classes such as Person, Event or JobPosting. The selected table columns include three data types: textual values, numerical values and DateTime values. All CPA tables include a main (subject) column in the first column position used in pairs with other columns for the CPA task. 91 CTA labels are used to annotate 162,351 columns in the CTA tables such as Organization, Date and Ofer , and 176 CPA labels are used to annotate 174,998 main column/column pairs in the CPA tables like price, datePublished and productID. Detailed statistics about the number of tables per Schema.org class as well as number of columns per label are provided on the SOTAB project page2. We split the CTA and CPA tables with a ratio of 80:10:10 into fixed training, validation and test sets by using multi-label stratification to include examples of all labels in every set. We further split the training set into a smaller subset using the same method and provide the Small training set with the goal of comparing how methods perform when trained on less examples. Furthermore, we provide subsets of columns in the test set for specific annotation challenges. These subsets are created by grouping the columns in the test set by the challenges mentioned in Section 3: 2http://webdatacommons.org/structureddata/sotab/ test columns that have missing values (Test MV ), test columns that are corner cases (Test CC), test columns that include values in diferent formats ( Test FH) and test columns that are chosen randomly (Test RC). Statistics of all splits are given in Table 2.
We evaluate the dificulty of SOTAB using three supervised methods. The first is a simple
Random Forest classifier using TF-IDF-weighted words as features. The second is TURL [
This paper introduced the WDC SOTAB benchmark. The aim of SOTAB is to complement the set of publicly available table annotation benchmarks with a CTA and CPA benchmark covering various entity types of general interest, e.g. products, local business, job postings, and to provide training data from many independent data sources for these types in order to reflect the full heterogeneity of the values that are used to describe entities. The WDC SOTAB benchmark is available for public download on the project page. The code that was used for the creation of the benchmark is provided on github3. 3https://github.com/wbsg-uni-mannheim/wdc-sotab