Tables are a ubiquitous source of structured information. However, their use in automated pipelines is severely a ected by conicts in naming and issues like missing entries or spelling mistakes. The Semantic Web has proven itself a valuable tool in dealing with such issues, allowing the fusion of data from heterogeneous sources. Its usage requires the annotation of table elements like cells and columns with entities from existing knowledge graphs. Automating this semantic annotation, especially for noisy tabular data, remains a challenge, though. JenTab is a modular system to map table contents onto large knowledge graphs like Wikidata. It starts by creating an initial pool of candidates for possible annotations. Over multiple iterations context information is then used to eliminate candidates until, eventually, a single annotation is identi ed as the best match. Based on the SemTab2020 dataset, this paper presents various experiments to evaluate the performance of JenTab. This includes a detailed analysis of individual components and of the impact di erent approaches. Further, we evaluate JenTab against other systems and demonstrate its e ectiveness in table annotation tasks.
Although a considerable amount of data is published in tabular form, oftentimes,
the information contained is hardly accessible to automated processes. Causes range
from issues like misspellings and partial omissions to the ambiguity introduced by using
di erent naming schemes, languages, or abbreviations. The Semantic Web promises to
overcome the ambiguities but requires annotation with semantic entities and relations.
The process of annotating a tabular dataset to a given Knowledge Graph (KG) is
called Semantic Table Annotation (STA). The objective is to map individual table
elements to their counterparts from the KG as illustrated in Figure 1 (naming according
to [
Copyright c 2021 for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0). 1,010,408 357,386 wd:Q79 ("Egypt") wd:Q183 ("Germany")
Cairo Berlin
Egypt Germany
JenTab is a toolkit to annotate large corpora of tables. It follows a general pattern of
Create, Filter and Select (CFS): First, for each annotation, initial candidates are
generated using appropriate lookup techniques (Create). Subsequently, the available context
is used in multiple iterations to narrow down these sets of candidates as much as
possible (Filter). Finally, if multiple candidates remain, a solution is chosen among them
(Select). We provide several modules for each of these steps. Di erent combinations allow
to ne-tune the annotation process by considering both the modules' performance
characteristics and their impact on the generated solutions. The contributions of our paper
are as follows. All experiments are based on the large corpus provided by Semantic Web
Challenge on Tabular Data to Knowledge Graph Matching (SemTab2020) [
{ We demonstrate the e ectiveness of JenTab relying only on public lookup services. { We provide a detailed evaluation of the impact individual modules have on the candidate generation. { We perform three experiments exploring di erent CTA-strategies that vary the mode of determining cells' types and hence the column annotation. { We compare JenTab's performance to other top contenders of the SemTab2020.
The remainder of this paper is structured as follows. Section 2 gives an overview of the related work. Section 3 describes our pipeline. Section 4 explains the dataset, encountered challenges, and the metrics used in our evaluation. Section 5 discusses our experiments and results. Section 6 concludes the paper and shows future directions. 2
We start by brie y reviewing benchmark datasets and motivate the selection of the
SemTab2020 dataset for our evaluation. We then summarize existing approaches to
match tabular data to KGs. While both semi-automatic and full-automatic approaches
have been proposed, we will focus our attention on later ones. This is in line with the
assumptions in this paper and the conditions posed by the SemTab challenges.
Benchmarks. In the past, various benchmarks have been proposed and used for STA
tasks. Manually annotated corpora like T2Dv27 or the ones used in [
6 We use the pre xes wd: and wdt: for http://www.wikidata.org/entity/ and
http://www.wikidata.org/prop/direct/ respectively.
7 http://webdatacommons.org/webtables/goldstandardV2.html
(2T) dataset [
Approaches. ColNet [
All these approaches rely on lookup services for their success. However, each of them
addresses only a single task from STA. Moreover, they can not cope with the frequent
changes of KGs since they rely on snapshots of the KG to train their respective models.
SemTab2019. In 2019, the SemTab challenge was initiated to bring together the
community of automatic systems for STA tasks. A four-round-dataset was released with
DBpedia [
A key success factor to those systems is the use of Wikidata and Wikipedia as
additional data sources. In this paper, we focus on exploiting only the target KG data
sources. Therefore, we try to maximize the bene t from a given cell value and minimize
our reliance on di erent data sources, which leads to a more straightforward system.
SemTab2020. The second edition of the challenge in 2020 changed the target KG
to Wikidata. MTab4Wikidata [
revisions. Cell annotation candidates are generated using this index and a
one-editdistance algorithm. Disambiguation is done via pairwise lookups for all pairs of entities
within the same row. bbw [
To our knowledge, none of the these systems provided a detailed study on various solutions for STA tasks, backward compatibility across rounds, or a time analysis. 3
Our system's modules can be classi ed into one of the following three phases, which together form a Create, Filter and Select (CFS) pattern. During the Create-phase, candidates are retrieved for each requested annotation. In the Filter-phase, the surrounding context is used to reduce the number of candidates. Eventually, in the Select-phase, the nal annotations are chosen among the remaining candidates. The individual modules for the same task di er in their treatment of the textual input and the context used. This causes not only di erences in the accuracy of their results but also a ects their performance characteristics. In the following, we explain the necessary preprocessing steps and describe the developed modules for each phase. 3.1
Before the actual pipeline, each table is subjected to a preprocessing phase consisting of three steps: The rst step aims at normalizing the cells' content. First, we attempt to x any encoding issues using ftfy11. Further, we remove special characters like parentheses or slashes. Finally, we use regular expressions to detect missing spaces like in \1stGlobal Opinion Leader's Summit ". In addition to the initial values, the normalized ones are stored as a cell's \clean value". In the second step, we use regular expressions to determine the datatype of each column. While our system distinguishes more datatypes, we aggregate to those having direct equivalents in KGs, i.e. OBJECT, QUANTITY, DATE, and STRING. Cells in OBJECT-columns correspond to entities of the KG, while the others represent literals. In the nal step, we apply type-based cleaning. In general, it attempts to extract the relevant parts of a cell value for QUANTITY and DATE columns. For example, it splits the numeric value from a possibly existing unit in QUANTITY cells. Similarly, redundant values like \10/12/2020 (10 Dec 2020)" are reduced to \10/12/2020 ".
(a) Cell
(b) Column
(c) Row
(d) Row-Column
Tabular data o ers di erent dimensions of context that can be used to either generate
annotation candidates (Create-phase) or remove highly improbable ones (Filter-Phase).
Figure 2 illustrates those visually. The Cell Context is the most basic one, outlined in
Figure 2a. Here, nothing but an individual cell's content is available. We can then
de ne a Column context as shown in Figure 2b. It is based on the premise that all cells
within a column represent the same characteristic of the corresponding tuples. For
the annotation process, this can be exploited insofar that all cells of a column share
the same class from the KG. Annotations for cells in OBJECT-columns have further a
common class as required by the CTA task. Similarly, the assumption that each row
refers to one tuple leads to the Row Context of Figure 2c. Annotation candidates for
the subject cell, i.e., a cell holding the identi er for the respective tuple/row, have
to be connected to their counterparts in all other cells within the same row. Finally,
all contexts can be subsumed in the Row-Column Context as given by Figure 2d. It
combines the last two assumptions representing the most exhaustive context. In the
following, we summarize our modules. For a detailed description kindly refer to [
With candidates available for individual cells, another set of modules can be used to derive candidates for the CTA and CPA tasks.
{ CTA collects the parent classes from all CEA-candidates for a particular column and uses them as CTA-candidates for that column. { CPA retrieves all properties for CEA-candidates of subject cells and compares those to the values of the row. While object-properties are matched against the candidate lists, literal-properties use a mix of exact and fuzzy matching. DATE-values are matched based on the date part omitting any additional time information. Di erent datetime-formats are supported.
STRING-values are split into sets of tokens. Pairs with an overlap of at least 50% are considered a match.
QUANTITY-values are compared using a 10% tolerance, as given in Equation 1.
M atch = (true;
if j1
f alse; otherwise
vvaalluuee21 j < 0:1
(1)
Filtering Candidates The previous modules generate lists of candidates for each task.
Next, lter-modules remove unlikely candidates based of di erent contexts.
{ CTA support (Column Context ) removes CTA-candidates that do not apply to
at least a minimum number of cells in that column.
{ CEA unmatched properties (Row Context ) removes CEA-candidates that are
not part of any candidate for a CPA-matching.
{ CEA by property support (Row Context ) rst counts CPA-matches for
subjectcells' CEA-candidates. All but the ones scoring highest are then removed.
{ CEA by string distance (Cell Context ) excludes all CEA-candidates whose label
is not within a certain range wrt. their Levenshtein distance [
CTA-candidate, i.e., the one with the most instances in the KG.
{ CPA by Majority applies a majority voting on the remaining CPA-candidates. 3.3
12 The lter modules applied before might have removed all CEA-candidates. Runner
Runner errors audit results
Manager
Runner
Clean Cells
Type Prediction Approach
Lookup (Wikidata) cache Generic Strategy
cache Endpoint (Wikidata) cache
Autocorrect results correspond to annotations of tasks, audit data that allows assessing the impact of individual modules, and possibly a list of any errors thrown during the processing.
The Manager's dashboard contains information about the following, the current state of the overall system, i.e., processed versus not yet tables, besides, data about connected Runners and errors are thrown (if any). It also gives an estimate of the remaining time needed. Finally, once the processing has nished, all gathered annotations can also be accessed from this central point. The Runner coordinates a single table's processing at a time through a series of calls to di erent services. Tables are rst passed through the preprocessing services of Clean Cells and Type Prediction. Afterwards, the core pipeline is executed via the Approach service. Approach depends on the following four services. Lookup and Endpoint are proxies to the respective KG lookup and SPARQL endpoint services respectively. Moreover, the computationally expensive Generic Strategy, in the CEA lookup, see Subsection 3.2, is wrapped in a separate service. These three services include caching for their results. The nal dependency is given by the Autocorrect service, which tries to x the spellings mistakes in cells.
The chosen architecture has several advantages. First, using caches for computationally expensive tasks or external dependencies increases the overall system performance. Furthermore, it reduces the pressure on downstream systems, which is especially important when public third-party services are used. Second, when the target KG is to be substituted, all necessary changes like adjusting SPARQL queries are concentrated within just two locations: the corresponding lookup and endpoint services. Third, the distributed design allows scaling well with respect to the number of tables to be annotated. Any increase in the number of tables can be mitigated by adding new Runners to cope with the workload. Finally, the implementation allows reusable, and self encapsulated pieces of code. For example, Runner can deal with any other Approach implementation, and Autocorrect can be used by any other Approach. 4
We base the evaluation of our approach on the corpus provided by the Semantic Web
Challenge on Tabular Data to Knowledge Graph Matching (SemTab2020) [
CEA Label Lookup
CTA CTA-Support
CPA CEA by Unmatched Properties
CEA by String Distance
CEA by Column CEA by String Similarity CTA by Direct Parents CTA by Popularity
CEA by Row and Column
CEA by Row
CTA CTA-Support
CPA CEA by Unmatched Properties
CEA by Subject CEA by String Similarity 2 3 7 8
CEA by Property Support CEA by String Similarity
CEA by Column
CTA by LCS CPA by Majority CEA by Column
CPA CEA by Unmatched Properties
CEA by String Similarity 4 5 6 The order of modules used in the evaluation is outlined in Figure 4. The modules are arranged into several groups. Some groups are only executed if the preceding group had any e ects on the current candidate pool. Similarly, the di erent approaches for creating CEA-candidates skip cells that already have candidates at the time.
Group 1 represents the most direct approach. As its modules use only a few interdependencies, queries are rather simple and can be executed quickly. Still, it accounts for a substantial share of candidates and thus forms the basis for subsequent groups.
For cells that so far did not receive any CEA-candidates, Group 2 is a rst attempt to compensate by expanding the considered scope. Here, CEA by Row and Column precedes CEA by Row. Using more context information, i.e., the Column Context, returned results are of higher quality compared to CEA by Row. It will fail, though, when the current list of corresponding CTA-candidates do not yet contain the correct solution. In such cases, CEA by Row can ll in the gaps. If any of the two modules resulted in new CEA-candidates, the corresponding modules for CTA and CPA candidate creation will be repeated in Group 3 .
Group 4 attempts to select annotations for the rst time. A prior lter step again uses the Row Context to retain only the CEA-candidates with the highest support within their respective tuples. Afterwards, annotations are selected from the candidate pool available at this point. It yields solutions for the majority of annotation-tasks but may leave some gaps on occasion.
The next two groups represent our last e orts to generate new candidates using stronger assumptions. Group 5 assumes that we were already able to determine the correct CTA-annotation for the respective column and then uses all corresponding instances as CEA-candidates. Similarly, Group 7 assumes that the CEA-annotation subject cell is already determined and creates candidates from all connected entities. Country
Inception (LITERAL)
Area (LITERAL)
Label (LITERAL)
Capital (IRI) Groups 6 , and 8 are used to validate those candidates and possibly select further annotations to ll in the gaps.
Group 9 makes a last-ditch e ort for cells that could not be annotated so far. As no other module was able to nd a proper solution, this group will reconsider all CEA-candidates that were dropped at some point. Using this pool, it attempts to ll the remaining gaps in annotations. 4.2
We use the SemTab2020 dataset [
Figure 5 illustrates the challenges present in the dataset. a missing or not
descriptive table metadata, like column headers. b spelling mistakes. c ambiguity in cell
values. For example, UK has (Ukrainian (Q8798), United Kingdom (Q145), University
of Kentucky (Q1360303) and more) as corresponding entities in Wikidata. d missing
spaces, causing tokenizers to perform poorly. e inconsistent format of date and time
values. f nested pieces of information in Quantity elds, interfere in the corresponding
CPA tasks. g redundant columns. h encoding issues. i seemingly random noise in
the data. Berlin would be expected in the context of the given example. j missing
values including nulls, empty strings or special characters like (?, -, {) to the same
e ect. k tables of excessive length.
Besides the datasets, SemTab2020 also provides a framework to evaluate tabular data to
knowledge graph matching systems [
P = jcorrect annotationsj ; R = jcorrect annotationsj ; F 1 = 2 jannotated cellsj jtarget cellsj
P P + R
R (2)
However, these default metrics fall short for the CTA task. Here, there is not always
a clear-cut distinction between right and wrong. Some solutions might be acceptable but
do not represent the most precise class to annotate a column. Taking the last column
of Figure 5 as an example, the best annotation for the last column would likely be
capital (Q5119) (assuming \Tubingen" is noise here). Nevertheless, an annotation city
(Q515) is also correct, but just not as precise. To account for such correct but imprecise
solutions, an adapted metric called cscore is advised as shown in Equation 3 [
cscore( ) = AP =
P cscore( ) jannotated cellsj 81; > > ><0:8d( ); if >0:7d( ); if > >:0; ; AR = if is in GT; otherwise
P cscore( ) jtarget cellsj is an ancestor of the GT; is a descendant of the GT; ; AF 1 = 2
AP AR AP + AR (3) (4) 5
In this section, we discuss our ndings regarding the overall system. We start with preprocessing assessment, \Type Prediction" step which is responsible for determining a column's datatype, see Subsection 3.1. Figure 6 shows the confusion matrix of this step with 99% accuracy. We used the ground truth for CEA and CPA tasks to query Wikidata for their types; such values represent the actual datatypes, the predicted values are our system results.
Spelling mistakes are a crucial problem that we have tackled by using \Generic
Strategy", see Subsection 3.2. The e ectiveness of this is illustrated in Table 2: Almost
99% of unique labels were covered in the rst three rounds. However, this is reduced
to 97% in the last round. Our pre-computed lookups are publicly available [
OBJECT 0.99 0.01 0.00 0.00 l a u t c A
Our modular approach enables us to exchange individual components or provide backup solutions if the existing ones failed in speci c situations. By this means, we have established three di erent experiments to explore the e ect of changing cells' types retrieval. These three modes include: First, \P31" includes only direct parents using instance of (P31 ). We have used a majority vote to select a column type. Second, 2 Hops, includes \P31" with one additional parent via subclass of (P279 ). Finally, Multi Hops, creates a more general tree of parents following subclass of (P279 ) relations.
We have implemented ve strategies for an initial CEA candidates creation, see Subsection 3.2. Figure 7a shows how much each strategy is used. This underlines the need for various strategies to capture a wide range of useful information inside each cell. The shown distribution also re ects our chosen order of methods. For example, \Generic Strategy" is our rst priority, thus used most of the time. On the other hand, \Autocorrect" is has the lowest priority and is used as a means of the last resort. CEA selection phase involves two methods. Figure 7b demonstrates the use of each of them: our dominant select approach is \String Similarity", it is used by 38% more than the \Column Similarity". Finally, Figure 8a describes the distribution of CTA selection methods during the \P31" setting. While, Figure 8b represents the used methods in \2 Hops" mode, where LCS is the dominant selection strategy. Let's compare \Majority Vote" with the LCS methods in the two settings. The former successfully nds more solutions than the latter, which yields less reliance on backup strategies or tiebreakers. The same exclusive execution concept in CEA selection is also applied in CTA selection methods. The dominant method, e.g., LCS in \2 Hops" mode, is invoked more frequently due to its highest order. Other backup strategies try to solve the remaining columns if other methods failed to nd a solution for them.
Table 3 reports our results for the four rounds given the three execution modes.
In the rst three rounds, we achieved a coverage of more than 98.8% for the three
tasks. In the fourth round, CEA task, the coverage is dramatically a ected by the
selected mode. \P31" has achieved the highest coverage by 99.39%, while \Multi Hops"
reached only 81.83%. F 1 Score in CEA, CTA and CPA tasks is greater than 0.967,
0.945 and 0.963 receptively. These scores where obtained through the publicly available
evaluator code13 on our solution les [
R4
R4
Tough Tables (2T) dataset [
Table 5 shows the time consumption for all four rounds with the number of used
runners for each mode setting of the CTA task. Close inspection revealed that the
execution time is largely dominated by the responses of Wikidata servers and thus beyond
our control. Execution was time-scoped, i.e. an upper limit for the time per table was
set. This allowed the system to converge faster compared to the initial
implementation [
The results show that for most tables \P31" mode is the most e cient fastest approach. However, for the 2T dataset a more sophisticated approach is needed. Here, the \2 Hops" appraoch yields better results. The \Multiple Hops" strategy can not surpass any of the other strategies no matter the setting. In terms of both performance and results it delivers inferior results and should thus not be used.
A reoccurring source of issues was the dynamic nature of Wikidata. Users enter new data, delete existing claims, or adjust the information contained. On several occasions, we investigated missing mappings of our approach only to nd that the respective entity in Wikidata had changed. The challenge and ground truth were created at one point in time, so using the live system will leave some mappings unrecoverable. Moreover, we are limited by the fair-use policies of the Wikidata Endpoint service. Another limitation a ects the \CEA by Column" module. Some classes like human (Q5) have a large number of instances. Here, queries to retrieve those instances oftentimes fail with timeouts, which limits the module to reasonably speci c classes. 6
In this paper, we presented an extensive evaluation of our toolkit for Semantic Table
Annotation, \JenTab". Based purely on the publicly available endpoints of Wikidata,
its modular architecture allows to exploit various strategies and easily adjust the
processing pipeline. \JenTab" is publicly available [
We see multiple di erent areas for further improvement. First, certain components currently require substantial resources, either due to the number of computations necessary like the Generic Lookup or the lacking performance of the SPARQL endpoint. While we can address the latter by rewriting queries or re-designing the approach, the former o ers plenty of opportunities to accelerate the system.
The authors thank the Carl Zeiss Foundation for the nancial support of the project \A Virtual Werkstatt for Digitization in the Sciences (P5)" within the scope of the program line \Breakthroughs: Exploring Intelligent Systems" for \Digitization - explore the basics, use applications". We would further like to thank K. Opasjumruskit, S. Samuel, and F. Zander for the fruitful discussions throughout the challenge. Last but not least, we thank B. Konig-Ries and J. Denzler for their guidance and feedback. 15 https://github.com/fusion-jena/JenTab