While tables are a rich source of structured information, their automated use is oftentimes prevented by the inherent ambiguity contained within. Issues ranging from mere typos over inconsistent naming conventions to homonymy among values pose substantial barriers to exploiting this source of knowledge. Although the Semantic Web can alleviate many of these issues, the actual annotation process remains challenging. To foster new ideas and the improvement of existing approaches, the Semantic Web Challenge on Tabular Data to Knowledge Graph Matching (SemTab) since 2019 hosts yearly competitions allowing systems to present their current capabilities. Datasets of di erent origins and characteristics highlight the various challenges present in this area. In this paper, we report on the evolution of our system, \JenTab", during SemTab2021. We re-designed the system architecture, optimized individual modules, and developed various pipelines to target speci c challenges posed throughout the challenge. JenTab is among the top 5 systems in the rst two rounds of SemTab2021. The results demonstrate JenTab's exibility and its ability to quickly address new challenges.
Tables are an essential tool when it comes to structuring large amounts of information. Nevertheless, they are hardly machine-interpretable in their raw form and are thus hidden from many automated processes. The annotation of regular tables with concepts from the Semantic Web faces various challenges, including misspellings, abbreviations, and the general ambiguity of the free text.
Over time, di erent approaches have been developed to cope with these issues and provide a semantic layer on top of common tables. The Semantic Web Challenge on Tabular Data to Knowledge Graph Matching (SemTab)4 o ers a forum for state-of-theart systems to compare against one another and provides them with various datasets to ? Copyright 2021 for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0).
Country
Area
Capital
Country
Area
Capital
Country
Area
Capital Egypt
1,010,408 Germany 357,386
Cairo Berlin
Egypt 1,010,408
Cairo
Egypt 1,010,408
Cairo Germany 357,386
Berlin
Germany 357,386
Berlin https://www.wikidata.org/wiki/Q79 https://www.wikidata.org/wiki/Q183 https://www.wikidata.org/wiki/Q6256 https://www.wikidata.org/wiki/Q5119 (a) CEA (b) CTA
(c) CPA
challenge their capabilities. In its third year, it features a series of three rounds. Each
round consists of a variety of raw tables. Such tables to be annotated with concepts
either from Wikidata [
The annotation tasks themselves are split into three areas, namely Cell Entity Annotation (CEA), Column Type Annotation (CTA), and Column Property Annotation (CPA). Given a data table and a target Knowledge Graph (KG), CEA links a cell to an entity within the KG (cf. Figure 1a). CTA is the task of assigning a semantic type (e.g., a class) to a column (cf. Figure 1b). Finally, CPA assigns a suitable semantic relation (predicate) from the KG to individual column pairs (cf. Figure 1c).
In this paper, we present the evolution of JenTab as a reaction to the new challenges introduced by SemTab2021. We have adopted the system to cover multiple Knowledge Bases (KBs) and explored various pipeline con gurations based on the issues encountered. Moreover, we have enhanced the preprocessing steps to handle domain-speci c datasets within the challenge. Finally, we optimized many components to improve the overall processing time.
The remainder of this paper is structured as follows. Section 2 summarizes the basic ideas and outlines JenTab's approach. Section 3 describes the new capabilities and changes made to address SemTab2021. Section 4 lists the challenges encountered in this year's datasets and the ways they are addressed in JenTab. Section 5 discusses our results for all three rounds of SemTab2021. Finally, Section 6 concludes the paper and gives an overview of our future work. 2
This section provides an overview of the general approach JenTab follows. We brie y
discuss the various table contexts exploited during the annotation process. Finally,
we outline the default con guration of our pipeline. For more details, kindly refer to
previous publications [
JenTab follows a Create, Filter, and Select (CFS) pattern: It starts with creating a large pool of candidates or possible solutions for each table component (cells, columns, and column pairs). Subsequently, this set of candidates is iteratively ltered using various characteristics of the di erent table contexts and the current state of the annotation process. For example, an already determined CTA-solution may be used to remove CEA-candidates which are not an instance of the selected column type. Finally, (a) Cell (b) Column (c) Row (d) Row-Column once the list of candidates is reduced, solutions for each task can be selected based, e.g., on their string similarity to the initial cell values.
As noted before, JenTab relies on di erent table contexts to reduce the number of considered candidates. These contexts are illustrated in Figure 2.
{ The Cell Context relies on nothing else but the given value of each cell (cf.
Figure 2a).
{ The Column Context aggregates all cells of a single column and assumes that they
represent the same characteristic of their respective tuples (cf. Figure 2b).
{ The Row Context combines all cells of a single row and considers them as di erent
aspects of the same entity (cf. Figure 2c)
{ The Row-Column Context combines both the Row and Column context and thus
represents all the direct information known for a single cell (cf. Figure 2d)
Figure 3 gives an abstract overview of JenTab's default pipeline, pipeline full.
We use 5 phases that each contains a series of modules. In the rst phase, we create
the initial set of candidates using only the cell context. The following component is a
series of ltering modules leveraging both row and column context. The third phase
represents our rst attempt at selecting solutions. Afterwards, we turn our eye to cells
without solutions and try to nd candidates by exploiting row and column contexts.
Finally, the last phase covers our last resort strategies for selecting a solution. For the
complete con guration of our default pipeline, kindly refer to previous publications
that provided a comprehensive overview of the system alongside a detailed evaluation
of this pipeline [
In this section, we discuss the current system architecture of JenTab, our newly developed annotation modules, and the various pipeline con gurations we have developed. 3.1
Manager - The central service. Assigns tables to be processed to the actual Runners and collects results, errors, and audit records.
Runner - Worker node. Handles the communication between Manager and Solver(s). Solver - Core service. Performs the preprocesing steps and executes the actual pipeline on an assigned table.
Wikidata Proxy and DBpedia Proxy - Encapsulate all interactions with the
individual KGs, e.g., Wikidata [
Generic Lookup - Holds a precomputed mapping to the target KG for the unique
cell-values of the entire dataset using Jaro Winkler distance [
Caching Server - Centralized caching server. Reduces the number of queries issued through the proxy services by caching responses already retrieved before.
In the following, we list the changes compared to the previous year's system [
Runner
Runner errors audit results
Manager
Runner
Clean Cells
Type Prediction Solver
Wikidata Proxy DBpedia Proxy Generic Lookup
cache
Caching Server
During SemTab2021 we revised some of the previously created modules (see Section
3.2 in [
Based on the original JenTab pipeline [
JenTab. For more details, we kindly refer to previous publications [
Zooey Deschanelle Zooey Dechanelle Sarah Mclauglain Zooey Deschamel Zooey Dishonal Sarah Mclauhclin
Sarah Mcclaucklen (a) Clean table values.
(b) Matched values in the noisy table In this section, we describe JenTab's evolution during SemTab2021's three rounds. For each round, we discuss the provided tables, required tasks, and the target KG followed by our attempts and the used pipelines to tackle these challenges. In the rst round, two datasets are given { each consisting of 180 tables. The tasks included only CEA and CTA - no CPA-solutions were required. Wikidata and DBpedia are the targets KGs.
The most crucial challenges about these datasets are ambiguity and noise. In
addition, tables comprised more rows than most datasets of the past years. The datasets
have been reported to be adapted versions of the Tough Tables dataset [
Initially, JenTab supported only Wikidata as a target KG as used and tested in
the SemTab2020. However, the design was intended to support various KGs from the
beginning. We have implemented DBpedia Proxy that encapsulates the DBpedia lookup
API6 and the SPARQL queries necessary. Thus, with a simple con guration, the core
Solver service can easily be changed to retrieve mappings from DBpedia instead of
Wikidata and vice versa. In our rst trial, we used the DBpedia spotlight [
Ambiguity is a real challenge in the given tables. Since CPA-solutions were not required in this round, no corresponding targets were given. However, we believe column types alone are not su cient as the only semantic context during the lter stages. Thus, we have added CPA-targets on our own and evaluated the impact during various runs with and without these CPA-targets. This resulted in pipeline no cpa as the rst variation of our core pipeline, pipeline full.
The massive amount of noise was another challenge during this round. For example, table 00E2H310 about actors contained more than 12K rows while overall just describing a set of 5 unique cells (actor names). Here, we have manually grouped the tables by similarity and solved only the cleanest ones using pipeline full. The resulting solutions were subsequently transferred to their noisy counterparts. This approach resulted in pipeline keyTables for the Wikidata dataset. 4.2
In the second round, two new datasets are given. On the one hand, HardTables contains 1750 tables similar to previous tasks. On the other hand, BioTables features 110 tables from the biomedical domain. Both datasets required all three tasks { CEA, CTA, and CPA { and were based on Wikidata as the target KG.
A unique characteristic of BioTables is its rather wide tables with an average number of columns and rows of 6 and 2500, respectively. Initially, our pipeline full was not able to cope with these characteristics and returned timeouts for many tables.
So in a rst attempt, we developed pipeline essential to cope with BioTables. It consists of the core parts of pipeline full but resorts to ltering by properties only once. As this yielded mixed results, we turned our eye again to the implementation of individual modules. After close inspection and some restructuring within the modules, we were able to improve the performance of multiple modules up to the point that allowed us to run pipeline full also on this dataset.
HardTables featured an increased level of ambiguity again. We even encountered examples of tables where we, as humans, could not determine proper solutions. However, with no particular angle on this kind of challenge, we employed our regular pipeline full once again. Throughout several experiments, it continuously provided the best results.
PrefixSearch).
4.3
In the third and nal round this year, three datasets from di erent domains were
given. For the general domain, HardTables contained 7207 tables with CEA, CTA, and
CPA tasks required based on Wikidata as the target KG. Another general domain
dataset is GitTables including 1101 tables with only the CTA task being required [
The structure of tables in HardTables changed compared to previous datasets. This time, most tables contained only a single object column and multiple numeric ones. This structure makes some of the modules in pipeline full obsolete as they try to infer subject candidates from other candidates found in the same row. Nevertheless, the row-context seemed more critical than the column-context in this dataset. Combined, these facts led to the development of pipeline numeric and subsequently pipeline conditional that uses the traditional pipeline full as a backup if the initial attempts fail. The new pipeline numeric emphasizes candidates that match their row over those that match other candidates in the same column. As evaluating the column-context rst is computationally cheaper, pipeline full used the columncontext before the row-context. However, within this dataset, a reversal of this order proved more e ective.
GitTables posed new challenges, in particular regarding the structure of tables and targets. Unlike other datasets where the subject column had always been the rst column in a table, here, it could be located anywhere within a table. In another deviation from past datasets, proper CTA solutions could also involve properties. So far, CTA-solutions were always given by classes whereas the corresponding properties were subject to CPA. Other issues surrounding GitTables were of technical nature: In particular, the sparsity of cells in some columns provided hard challenges to our pipelines. Some columns for which CTA were requested even contained no cell value at all. Consequently, we approached GitTables as follows. 1. Parse the given targets to identify a) subject-columns as a the rst column of a table that has a corresponding CTA-target, b) object-columns as all remaining columns having a CTA-target but not being a subject-column. 2. Reduce the CTA-targets to those subject-columns. We expect them to be annotated by semantic types. 3. Since we can solve neither CTA nor CPA without also tackling CEA, we arti cially added CEA-targets for the subject-columns. 4. We established the CPA-targets between subject-columns and the corresponding object-columns. Again, we expect them to be annotated by semantic properties. 5. We have selected pipeline essential to solve GitTables. 6. The nal targets' solutions are given by a combination of CTA and CPA solutions. Besides the default con gurations that retrieve types from DBpedia, we have supported schema.org as well. We have con gured the DBpedia Proxy to lter the retrieved types to those from schema.org only and discard all others. To the best of our knowledge, there are no other means to access schema.org mappings dynamically, e.g., using a standalone lookup API.
The domain-speci c dataset of BiodivTab also provided new challenges. In particular, there is only very limited context to disambiguate among possible candidates. Furthermore, we encountered the following challenges: { Most of the data are numerical, measured in research laboratories, and thus could not be mapped to the target KGs. { Abbreviations are extensively used for taxon names making it di cult to derive proper candidates. For example, C.glauca matches 48 plant species in Wikidata as of 14. October 2021. { Multiple entities contained within a single cell: A column containing a researcher's name may also include the research institute at the same time. { Values appear in condensed formats. We encountered species names represented in a structure like species:abc sub:xyz. { Chemical elements are commonly given by their corresponding symbols. It may be seen as a special case of the abbreviations mentioned above.
We devised a custom treatment for some of the issues encountered. We adapted our preprocessing to cope with special formats like species names. To mitigate the impact of abbreviations, we constructed dictionaries for both taxons as well as chemical elements based on Wikidata. We queried Wikidata with all instances of the taxon (?species wdt:P31 wd:Q16521) and programmatically fetched the corresponding labels. From these labels, we created abbreviated variations by using one or two characters from the rst word followed by the full second name. So, \Canna glauca", e.g., was converted into both \C.glauca" and \Ca.glauca". These variants were subsequently used as aliases for the respective Wikidata entities. The corresponding lookup was used with the highest priority during our initial creation of CEA-candidates. We have tackled the nested entities problem by trying to parse individual cells and only retaining the rst part. For example, David Eichenberg (University of Halle-Wittenberg) would be converted to only David Eichenberg. We used pipeline no cpa for two reasons: First, no CPAtargets were required. Second, we were not able to determine properties from Wikidata that would have connected subject candidates with the remaining values in their rows. 5
Table 1 shows an overview for the given datasets across all rounds. Only the HardTablestasks in R2 and R3 resemble the characteristics of the challenges provided in previous years. All other tasks provide fewer but substantially larger tables. This growth in size can largely be attributed to an increased number of rows, whereas the number of columns remained relatively constant. BiodivTab and GitTables are the exceptions here as they also feature a two to three times increase in the number of columns. Overall, this shifted the focus towards CEA and put less emphasis on both CTA and CPA tasks. Furthermore, some datasets did not require all tasks to be solved: R1 and the BiodivTab dataset of R3 required no CPA solutions, whereas the GitTables dataset of R3 included only CTA-targets. To leverage all of JenTab's modules, we created missing targets most of the time. The respective number of targets are provided within Table 1.
Spelling mistakes and arti cial noise are common challenges across SemTab2021's
datasets. A generic lookup is our primary strategy for tackling this crucial issue. This
lookup is created ahead of time. Due to the resources required for comparing cell
values against all labels (and aliases) within Wikidata or DBpedia For this, we extract
the unique values from all tables of a dataset and match those against the labels of
the respective KG using an optimized Jaro-Winkler Similarity implementation based
on [
Table 3 summarizes our results for R1 and R2 as reported by AICrowd8. R3 did not rely on AICrowd, and thus scores are unknown as of the time of writing. In the following, we discuss our results for each round. While R1 did not include CPA-targets as part of the challenge, we created such targets in order to use pipeline full. This approach improved the results considerably. Further improvements were achieved by using pipeline keyTables as described before. The mixed results indicate that JenTab is still challenged by the amount of noise introduced in these datasets. The 2T dataset
includes a vast number of arti cially created misspellings that severely impact the
performance of JenTab. We have tackled the ambiguity, especially in the R2 HardTables
using pipeline full. Our highest scores are achieved at CPA task. Thanks to our fuzzy
matching technique [
In this paper, we have reported on the updates to JenTab as part of the 2021-edition
of Semantic Web Challenge on Tabular Data to Knowledge Graph Matching. We
discussed the changes to the system architecture. We further created a variety of pipelines,
each catering to the speci c requirements of the individual datasets of the challenge.
JenTab remains a top-5 system among participants. Our code is publicly available [
We see various areas for further improvement. First, the binary decision of whether to keep candidates or remove them should be replaced by a scoring system that emphasizes well-supported candidates but maintains other options. A further challenge to address is the lack of targets for speci c tasks either voluntarily or by lack of corresponding structure in the target KG. As witnessed in particular by the GitTables dataset, JenTab struggles in such cases. Finally, we continuously strive to improve the performance of JenTab in particular to reduce the number of timeouts received from the public endpoints of KGs.
The authors thank the Carl Zeiss Foundation for the nancial support of the project \A Virtual Werkstatt for Digitization in the Sciences (P5)" within the scope of the program line \Breakthroughs: Exploring Intelligent Systems" for \Digitization - explore the basics, use applications". We would further like to thank Muhammad Abbady and Sarah Boning for the fruitful discussions throughout the challenge. Last but not least, we would thank Birgitta Konig-Ries her guidance and continuous feedback.