The Web provides a plethora of structured data, such as semantic annotations in web pages, data from HTML tables, datasets from open data portals, or linked data from the Linked Open Data Cloud. For many use cases, it is necessary to integrate such web data with existing local datasets. This integration entails schema matching, identity resolution, as well as data fusion. As an alternative to using a combination of partial or ad hoc solutions, this poster presents the Web Data Integration Framework (WInte.r ), which supports end-to-end data integration by providing algorithms and building blocks for data pre-processing, schema matching, and identity resolution, as well as data fusion. While being fully usable out-of-the box, the framework is highly customisable and allows for the composition of sophisticated integration architectures such as T2K Match, which is used to match millions of web tables against DBpedia. A second use case for which WInte.r was employed is the task of stitching (combining) web tables from the same web site into larger tables as a preprocessing step before matching. The WInte.r framework is written in Java and is available as open source under the Apache 2.0 license.
Many web-based systems need to combine data from various sources, such as semantic annotations in web pages, data from HTML tables, datasets from open data portals, or linked data from the Linked Open Data Cloud. These data often do not use the same schema and miss explicit links between their entities. Such heterogeneous data sources must hence be integrated before they can be used for any further use case.
While there are tools available to solve speci c data integration problems,
such as schema matching, identity resolution or data fusion, using these tools
in order to realise a complete data integration work ow can be cumbersome
and requires researchers to spend quite some time on writing glue code. Also,
scalability is often a problem as many tools are designed for use cases with
only a few datasets and not for web-scale scenarios where thousands of data
sources need to be integrated. As an alternative, this poster presents the Web
Data Integration (WInte.r ) Framework. The WInte.r framework supports
endto-end data integration processes by providing algorithms and building blocks
for web data pre-processing, schema matching, identity resolution, as well as
data fusion. As an example, the integration of millions of tables collected from
web pages with the DBpedia knowledge base has been implemented using the
WInte.r framework in the T2K Match [
The WInte.r framework is fully usable out-of-the box with implementations for all mentioned data integration tasks and provides highly customisable functionality that allows for the composition of sophisticated integration architectures. This allows researchers to focus on the actual task instead of starting from scratch and repeatedly implementing the same functionality in di erent projects. The WInte.r framework is written in Java and is available as open source under the Apache 2.0 license.2 2
Functionality The WInte.r framework covers all steps of the data integration process, including data loading, pre-processing, schema matching, identity resolution, and data fusion. This section gives an overview of the methods that are provided for each of these steps. Figure 1 shows the components that are involved in the integration process. The WInte.r data sets act as an interchangeable data management and processing component for the rest of the framework. They implement essential processing operations such as iteration, ltering, transformations, aggregations and joins. By exchanging the data set implementation, a user can, for example, switch between single-threaded or parallel execution.
Data Loading. WInte.r provides readers for standard data formats such as CSV, XML and RDF. In addition, WInte.r implements a specialized JSON format for representing tabular data from the Web together with meta-information about the origin and context of the data, as used by the Web Data Commons (WDC) web tables corpora.3 Speci cally, the context represents data such as the original URL, page and section headings and the surrounding text from the web page from which the tables were extracted.
Pre-processing. During pre-processing, data is prepared for the methods that are applied later on in the integration process. WInte.r provides specialized preprocessing methods for data with missing schema information, such as: data type detection, unit of measurement normalization, header detection, and entity name detection (In cases where no explicit \rdfs:label" property is available, the entity name detection nds the property that most likely contains the entity names). 2.2
The matching components are used for schema matching and identity resolution. First, an optional blocking step generates candidate pairs of records to reduce the total number of comparisons. Then, a matching rule evaluates each candidate pair and decides whether or not to create a correspondence. Matching rules are speci ed by the user and can, for example, calculate the weighted sum of di erent similarity values and apply a threshold. Alternatively, matching rules can be learned using supervised machine learning. Given a labelled training set, the framework will apply a user-speci ed machine learning algorithm from the WEKA machine learning library to learn the matching rule.
Schema Matching. Schema matching methods nd attributes in two schemata that have the same meaning. WInte.r provides three pre-implemented schema matching algorithms which either rely on attribute labels (label-based schema matching) or data values (instance-based schema matching), or exploit an existing mapping of records (duplicate-based schema matching) in order to nd attribute correspondences.
Identity Resolution. Identity resolution methods (also known as data matching or record linkage methods) identify records that describe the same real-world entity. The pre-implemented identity resolution methods can be applied to a single dataset for duplicate detection or to multiple datasets in order to nd recordlevel correspondences. WInte.r provides the following pre-implemented methods: standard blocking by single/multiple blocking key(s), Sorted-Neighbourhood Method, token-based identity resolution and rule-based identity resolution. 2.3
Data fusion methods combine the data from multiple sources into a single,
consolidated dataset. For this task, they rely on the schema- and record-level
correspondences that were discovered in the previous steps of the integration process.
However, di erent sources may provide con icting data values. WInte.r provides
a mechanism to resolve such data con icts (i.e., deciding which value to
include in the nal dataset) by applying di erent con ict resolution functions for
strings, numbers, lists of values, data type independent functions, and functions
that consider metadata such as creation time or dataset speci c trust scores.
A user can specify a fusion strategy that speci es how data con icts should be
handled for each property.
This section gives an overview of existing work based on the WInte.r framework.
Integration of Large Numbers of Data Sources: Augmenting the DBpedia
Knowledge base with Web Table Data. Many web sites provide data in the form of
HTML tables. Data from these tables can be used to ll missing values in large
cross-domain knowledge bases such as DBpedia. An example of how pre-de ned
building blocks from the WInte.r framework are combined into an advanced,
use-case speci c integration method is the T2K Match algorithm [
5 https://github.com/olehmberg/WebTableStitching 6 http://ds4dm.de/en/