A large number of HTML Tables on the Web contain relational data which can be used to augment knowledge bases such as DBpedia, Yago, or Wikidata. A large part of this data is time-dependent, i.e., the correctness of a fact depends on a specific temporal scope. In order to use this data for knowledge base augmentation, we need temporal meta-information. Existing methods rely on timestamps within the table itself or its context as temporal meta-information. Yet, the relationship between these timestamps and data within a table is often unclear. Additionally, timestamps are rather sparse, and there are many web tables for which no timestamps exist. Knowledge-Based-Trust (KBT) uses the overlap with ground-truth to estimate the trustworthiness of a dataset. This paper introduces TimedKBT, which overcomes the dependence on sparse and possibly misinterpreted timestamps by propagating temporal meta-information from a knowledge base to web table data using KBT. It also derives a trust score that estimates the correctness of the data and the assigned temporal meta-information. We evaluate Timed-KBT on the use case of fusing data from a large corpus of web tables for filling missing facts in a knowledge base. Our evaluation shows that Timed-KBT yields an increase in F0.25-Measure of 19.01 % when compared to KBT and 9.44 % when compared to a method that relies solely on timestamps extracted from the table and its context.
Besides free text, information on the Web might be represented in the form of
HTML tables that contain relational data, referred to as web tables [
Many web tables contain time-dependent data, in which a fact is only valid
given a certain temporal scope. There is large potential and growing interest
in utilizing time-dependent web data [
Time-aware fusion strategies require temporal meta-information. We define
temporal meta-information as the overall presence of temporal scopes, which are
time annotations of certain facts or values. Existing works estimate temporal
meta-information by utilizing timestamps [
Our task is therefore to estimate missing temporal meta-information using other sources than the timestamps within a webpage.
This paper introduces Timed-KBT, an approach that estimates missing
temporal meta-information using Knowledge-Based-Trust (KBT) [
We evaluate Timed-KBT on the use case of data fusion using a large corpus of web tables. As a knowledge base to be augmented we use a subset of Wikidata that contains facts about countries, cities and athletes. We further extended the subset with various time-dependent datasets. We find that Timed-KBT is able to estimate missing temporal meta-information with enough quality to improve data fusion results. By using timestamps as a restriction for Timed-KBT we are furthermore able to derive a precision-orientated time-aware fusion method1.
The next section provides a motivating example and describes the overall use case at hand. Section 3 frames this research within related work. In Section 4 we describe our fusion methodology and Timed-KBT itself, while our experimental setup, is described in Section 5. The results are discussed in Section 6. Section 7 is our conclusion. 1 The methods presented in this paper are implemented as part of the publicly available T2K Framework: http://dws.informatik.uni-mannheim.de/en/research/T2K. entity
Germany population: 2007 82,266,372 continent: Europe non-time-dependent attribute (reference type) fact time-dependent attribute (numeric type) 2008 2009 2010 81,902,307 81,776,930
temporal scope
We aim to augment a temporal knowledge base using web table data [
A slot refers to a missing fact in the knowledge base. In this work we aim to use web table data for targeted slot filling, where we try to fill specific slots within a series of facts of an existing time-dependent triple. This means that the temporal scopes of the slots are previously known and provided as targets for fusion strategies. Fusion strategies are tasked to fill these target slots by using matched values. We define matched values as values extracted from web tables and matched to the knowledge base as described in the following example.
Figure 2 shows an example web table with three time-dependent columns: one leader and two population columns. The leader column corresponds to the year 2017, the first population column to 2015, a date not found on the webpage, i.e. lacking a timestamp, and the second to 1990. The figure shows that timestamps on the webpage are not explicitly associated with data, not all temporal scopes are described by timestamps and timestamps unrelated to the data exist. As such, 1990 is not assumed to be the temporal scope of the 5th column.
Rows in the table are matched to entities in the knowledge base, while
columns are matched to attributes [
As a result, for a given target slot of the attribute population with the target year 1990, population numbers from the 4th and 5th columns are both taken as candidate values for fusion. The task of fusion methods is then to assign matched values to temporal scopes before using them to fill target slots.
Page Title: Country Data 2017
…
The following table provides information about those five countries, including capital, the national day, the current leader and current
population. In comparison we provide population numbers from the year 1990
Country Capital [
© 2014 – FactsFactsFacts.com Our work is related to three research areas: (1) utilization of ground-truth to estimate the quality of web data, (2) methods for consolidating time-dependent web data, and (3) time-aware fusion for web table data.
Utilizing a ground-truth for fusion is an approach employed by many fusion
methods [
For time-aware consolidation methods for web data, there is a comprehensive
exploration by Dong et al. [
In regards to time-aware fusion, Dong et al. [
Finally we will introduce two works that provide time-aware fusion for web
table data. The InfoGather+ system [
We evaluate the quality of estimated temporal meta-information for the use
case of data fusion. For this, we implement and compare five fusion strategies:
Two baselines strategies, Voting and KBT, a time-aware strategy from previous
research [
The underlying fusion methodology consists of four steps:
Scoring: All fusion strategies provide scores for each matched candidate value given the temporal scope of its target slot. The fusion strategies are essentially scoring functions that influence the fusion process solely by scoring the individual matched values. Scores are provided from a range 0.0 to 1.0.
Filtering: Based on their scores, values are filtered using a learned threshold. The filtering influences a possible precision/recall trade-o↵. Depending on the quality of the fusion strategy’s score, filtering could lead to a favorable increase in precision or an unfavorable decrease in recall. We deal with this tradeo↵ by optimizing thresholds for a weighted F -Measure. We learn thresholds in 0.05 steps from 0.0 to 1.0 and per fusion strategy and class-attribute combination.
Grouping: For a given target slot, we cluster all unfiltered equal values into groups. Numeric-type values are considered equal if they have a normalized similarity of at least 0.98, while reference-type values must refer to the same entity. If multiple values from the same source, i.e. the same pay-level domain, are present in one group, only the value with the highest score is kept. We sum the scores of all values in a group to calculate a group score.
Selection: For every target slot we select the group with the highest summed score and extract from the group a value that is chosen as the resulting fused fact of that target slot. For numeric values, where not all values in a group are exactly equal, we use the median as the extracted value. 4.2
Voting is a common baseline strategy [
KBT is a fusion strategy based on Knowledge-Based-Trust [
KBT(column) = # values in column with correct overlap # values in column with overlap (1) 4.3
Timed-KBT assigns explicit temporal scopes to web table data by exploiting its overlap with a temporal knowledge base. It is based on the assumption that neighboring values, e.g. within one column, share a common temporal scope. The idea is to use the knowledge base to detect this scope for overlapping values, and propagate the scope to the neighboring non-overlapping values.
To generate missing temporal scopes we first find temporal scope t that maximizes the KBTt score of a column. The KBTt score is computed by only using values from the knowledge base that are annotated with the given temporal scope t. We assign t to the to the web table column, while the KBTt score itself is then used as the fusion score of the matched values. In the table in Figure 2, Timed-KBT will e.g. be able to assign di↵erent scopes to the population columns as the temporal scope t that maximizes KBTt will likely di↵er per column.
KBTt(column) = # values in column with correct overlap given scope t # values in column with overlap given scope t tcolumn = argmax KBTt(column)
t2 T
Timed-KBT(column) = KBTtcolumn (column) We implement two Timed-KBT-based fusion strategies. In the first, TKBT, T is a set of temporal scopes derived from the knowledge base. In the second, TKBT-Restricted, we restrict T to temporal scopes that exist as timestamps (2) (3) (4) extracted from the table and its context. For the second approach, this would mean in the specific case of the table in Figure 2, that the first population column cannot be assigned a scope of 2015.
Neighboring Scope Estimation: As we assign only one temporal scope to a table column, its values are only used for the fusion of slots with that assigned scope. Assuming that temporal scopes are years and that facts of certain attributes do not completely change yearly, it would make sense to allow values that were assigned one scope, to be used to fuse facts for neighboring scopes. Given for example Figure 2 and that the first population column was assigned the scope 2015, its values can be used as candidates for slots with scope 2014, with an adapted score computed as estimatedScore = (neighboringScore
di↵ ⇥ ↵ 0 di↵ <= maxDi↵ di↵ > maxDi↵ (5) , where neighboringScore equals the assigned score of the column, di↵ equals the absolute di↵erence between the assigned year and the target year, while maxDi↵ equals the maximum di↵erence allowed. This maximum di↵erence is learned per class-attribute combination from 0 to 10. We therefore define ↵ to be 0.1 = 1/10. 5
In this section we describe the knowledge base, the web table data and how we measure fusion performance. 5.1
As the knowledge base to be augmented in our evaluation we use a subset of
the temporal knowledge base Wikidata [
Table 1 provides an overview of the classes, entities, attributes and facts in the knowledge base. Classes are categories of entities and their corresponding attributes. The table also shows by class from which sources datasets were used to complement Wikidata. We acquired data from the sources either by manually written crawlers and extractors, or through data dumps. 5.2
For our experiments we use the Web Data Commons Web Table Corpus from 20153, which was extracted from the July 2015 Common Crawl. The original 2 The resulting knowledge base is publicly available as the Time-Dependent-Ground
Truth dataset: http://webdatacommons.org/timeddata/
3 http://webdatacommons.org/webtables/#toc2
corpus contains 1.78 billion HTML pages, whereas the web table corpus consists
of 90 million relational HTML tables [
We extracted timestamps using HeidelTime4 [
Most timestamps are found in the context of the table, which could mean that they have no explicit relation to the data in the table. Timestamps extracted from cells of the same row could similarly describe an unrelated date attribute, e.g. the ‘National day’ column in Figure 2. Timestamps in table captions and column headers, which are likely to be the most relevant, are sadly also the least present. Presence also di↵ers by class: For Country and City we find many in the column header, while for NFL Athlete we find more in cells of the same row. 5.3
To test our fusion methods we make use of the Local-Closed-World-Assumption
(LCWA), where we assume that facts present in the knowledge base are correct
and can be used to determine whether fused facts are correct. The LCWA has
been used and empirically examined by research with a similar task [
We use the F -Measure [
F = (1 + 2) ⇥ 2 P⇥ rPecriesciiosni o⇥nR+eRcaelclall (6)
As the knowledge base is used for both, learning and testing, we split the data four times, each time placing approximately 25 % of the data in the testing set, and the remainder in the learning set. To replicate the use-case of targeted slot filling, where some missing slots within a series are to be filled, we split by series of timed facts, so that some timed facts of a triple are in the testing set, while the remaining are used for learning. To ensure that the temporal scopes of removed facts are well distributed, we randomize how each series is split.
Within this paper we define temporal scopes as years. Nonetheless, it is possible that in web tables we will also find attributes which are more frequently updated and would require more fine-grained temporal scopes. 4 https://github.com/HeidelTime/heideltime In this section we will present and discuss the overall results of the implemented fusion strategies and discuss the e↵ect of the neighborhood scope estimation. Table 4 shows the average performance by fusion strategy. We can first of all see that KBT outperforms Voting by a large margin for both F1 and F0.25. Additionally KBT has the highest recall for F0.25 and among the highest for F1, which means that any strategy that outperforms KBT, does so by increasing precision.
For F1 the di↵erence between KBT and TT-Weighting is minimal. For F0.25,
there is an increase in the F-Measure and a larger increase in precision from KBT
to TT-Weighting. This shows that the scores computed by TT-Weighting are
relevant to the fusion precision, but also that they are only e↵ective when a drop in
recall is acceptable. The results could indicate that some timestamps are relevant
to the data in the table and that timestamp locations have certain relationships
with attributes, which is the main assumption behind TT-Weighting [
Both Timed-KBT-based approaches show an increase in performance when compared to other methods for both F1 and F0.25. TKBT for F1 even has the highest recall. Through this we can infer that a knowledge base can successfully be used to generate temporal meta-information for web table data.
TKBT-Restricted outperforms TKBT for F0.25. While its increase in precision comes at the cost of recall, the decline happens at a favorable rate. TKBT is unable to yield a higher precision for F0.25, e.g. by increasing the threshold, without a performance drop, whereas the precision increase for TKBT-Restricted is large enough to compensate for the drop in recall. This shows that timestamps from the tables and their context can be relevant to the data and that TKBT-Restricted is able to use them e↵ectively.
Strategies that use timestamps, i.e. TKBT-Restricted and TT-Weighting, generally speaking come at a large cost to recall. This could show that timestamps in web tables are too sparse for a high recall fusion strategy. From Table 5 we can see that incorporating neighborhood estimation into the Timed-KBT-based strategies had a large positive e↵ect on fusion performance. The relative increase was more than 40 % for F1 and 20 % for F0.25 for both strategies. A rather unexpected result is that neighborhood estimation increases precision in addition to recall. The reason for the increase in precision is likely that matched values of neighboring temporal scopes with a high score can outweigh low-scoring, and probably incorrect, values assigned to the target scope. 7
In this work we introduced Timed-KBT, an approach that exploits the temporal meta-information in a knowledge base to generate missing temporal metainformation for web table data. We test Timed-KBT using a large web table corpus and an extended subset of Wikidata as a knowledge base for slot filling, a knowledge base augmentation task that makes use of fusion methods.
We find that Timed-KBT is able to assign useful explicit temporal scopes to web table data. We also find that using scores estimated by Timed-KBT for fusing time-dependent web table data yields a performance increases when compared to other fusion methods. We then utilized timestamps extracted from the web tables and their contexts as a restriction for candidate temporal scopes used by Timed-KBT. This approach yields a higher performance in regards to precision, and therefore a possibly more favorable performance for knowledge base augmentation. We conclude that timestamps in the table and its context are useful for a precision-oriented time-aware fusion strategy. Finally we show that data with assigned explicit temporal scopes is highly useful for estimating facts with neighboring temporal scopes.
Overall we demonstrate that a temporal knowledge base can be used to estimate missing temporal meta-information for web table data. We also show that with Timed-KBT, we are able to perform knowledge base augmentation from web table data for current and historic facts, instead of just for facts limited to one point in time. Our findings enable the utilization of time-dependent web data even when that data lacks temporal meta-information.