As a part of LOD2 project we developed several extensions for Google Refine - a simple, yet very powerful open-source tool for working with messy data, to make LOD a first-class citizen in this tool, which is currently tightly coupled with Freebase and has no support for DBpedia. LODGrefine is a version of Google Refine with integrated extensions developed by Zemanta and DERI, adding support for reconciliation with DBpedia, export to RDF, augmentation of data with columns from DBpedia and extraction of entities from full text. Use of LODGrefine will be demonstrated in three use cases.
Data cleansing and linking are very important steps in the life cycle of linked
data [
Unfortunately, these steps are not always trivial for an average user, e.g.
a statistician working with statistical government data; on the contrary, they
pose a problem even for more skilled researchers working in the field of
semantic web. If data is available online this doesn’t necessary mean it is ready to be
used in semantic applications. In most cases such assumptions are wrong; it is
very likely that data has to be cleaned, because everyone can publish data on
the Web. Taking care of quality of Web data is still one of the main challenges
for Semantic Web applications [
A good and publicly available cleansing/linking tool should at least be able
to: assist user in detecting inconsistent data, quickly perform transformations
on a relatively large amount of data, export cleansed data into different formats,
be relatively simple to use, and be available for different operating systems.
Fortunately, there is one open-source (BSD licensed) solution available, which
meets all the criteria mentioned above and even more. It was created especially
for dealing with messy data, it is modular based and extendable, it works on
all three major operating systems and it already provides functionalities to
reconcile data against Freebase. This tool is Google Refine (GR) [
GR provides means to reconcile and extend data with data from Freebase, but not from DBpedia. By providing a LOD-friendy version of this tool (LODGrefine) supporting DBpedia we’ve made an important step towards making LOD a first-class citizen in this powerful, yet easy to use tool. LODGrefine has preserved all of the GR’s cleansing and reconciliation functionalities and added new ones to make it even more useful for Semantic Web community. 2
GR is currently one of most powerful and user-friendly open-source tools for cleansing and linking data with Freebase. Support for faceted browsing and good filtering possibilities are its main assets, it works fast even when dealing with large amounts of data and it has a built-in support for Google Refine Expression Language (GREL), a special scripting language, which is easy to learn and use to transform data. The most important features of GR are the reconciling and extending data.
It is a server-client web application intended to run locally by one user. Instead of using a database to store imported data, it uses memory data-store, which is built up-front and optimized for GR operations. Its data cleansing and reconciliation abilities are tightly integrated with Freebase (Fig. 1) and making it support a different triplestore offering a SPARQL endpoint, e.g DBpedia, was not possible without implementation. 2.1
Due to the modular nature of GR architecture it was not required to change
the code of GR itself to make it LOD-enabled. We implemented extensions
with additional functionalities. Maali and Cyganiak, researchers at Digital
Enterprise Research Institute already developed RDF Refine extension [
Extensions (dbpedia-extension) developed by Zemanta complements
functionalities of RDF Refine with ability to extend reconciled data with new
columns based on data from DBpedia. It also supports extraction of entities
from unstructured text using Zemanta API [
Both extensions are free to use, their code is shared under the BSD License
on Github (RDF Refine1 [
To simplify the process of obtaining and installing LOD extensions we decided to integrate them into the latest version of GR ( 2.5-r2407) and name the LODenabled version LODGrefine. Although GR itself does not need any special installation, it is enough to unpack it and run it, the location of extensions depends on the operating system and it is more convenient, especially for first time users, if extensions are already integrated in the tool. Furthermore, we created a Debian package for LODGrefine, which will be integrated into LOD2 Stack, a stack of tools for managing the life-cycle of Lined Data.
LODGrefine is available under Apache License 2.0 and can be freely
downloaded either in binary format or as source code [
For demonstration we prepared three use cases – examples of how LODGrefine can be used to clean data from different sources and domains and how to transform it into Linked Data. 3.1
100 best novels In first example we will demonstrate how to convert data from a website first to a LODGrefine project, reconcile it, augment it additional columns from DBpedia and then export it as Linked Data.
In this example we will transform a list of 100 best novels from Modern library web page3. The list contains two rows for each novel - first row contains the title and the second one the author, but we need data in columns - one column for title and one for author. Fortunately, LODGrefine has an option to import line based text files and it can read text from clipboard. With some minor changes of default settings our data is imported in columns in few seconds instead of minutes or even hours. With GREL functions we convert titles from uppercase to titlecase and remove ’by ’ preceding authors names.
Next step is reconciling author names with DBpedia using RDF extension to entity type dbo:Person4. After reconciliation data is ready to be extended with has abstract property from DBpedia using Zemanta extension (Fig. 2).
The last step of converting online list of novels into Linked Data is configuring RDF schema alignment skeleton, with which we specify how RDF data will be generated (Fig. 3). At any time we can preview the Turtle representation of generated RDF data to see whether schema we defined produced expected results. After the schema has been configured, data can be exported into one of the RDF representations supported by LODGrefine - RDF/XML or Turtle (fig. 4).
In this example we demonstrated how easy it can be to transform data from a website into Linked Data using LODGrefine. We also demonstrated its most important functionalities. The Comprehensive Knowledge Archive Network (CKAN) is well known system for storage and distribution of data. It is widely used for storing government data and national registers as well as Data Hub, the community-run catalogue of useful sets of data on the Internet 5. CKAN data is especially interesting for Linked Open Data community, but currently not all CKAN datasets in the Data Hub are provided as RDF (either SPARQL endpoint or RDF dump). A lot of datasets are provided as files with comma separated values (CSV), as Excel files or XML. Our goal is to show how it can be relatively easy transformed into triplets using LODGrefine in a similar way as described in previous example with novels.
In the last example we will demonstrate the full power of LODGrefine by using it to clean and filter links extracted from blog posts to obtain links, that could be considered as descriptions for entities.
Many times bloggers include links into their blog posts to point the reader to Wikipedia or any other web page, that can be considered as information resource (e.g. Google Maps, Crunchbase6 - free database of technology companies, Amazon). Ideally, links that could be considered as entity candidates, have non-empty anchor text (entity surface form) and href attribute set to external URL, which directs to a page containing a description of concept/person/object (disambiguation page) mentioned in anchor text.
Unfortunately, blog posts also contain even more links that can be considered as noise or even spam, e.g. links with anchor text without semantic value (e.g. here, this, Read more). We used LODGrefine faceted browsing and filtering abilities to quickly identify patterns of occurring anchor texts or target links, which could be considered either as entity candidates or noise. We used GREL7 expressions to simply extract features from columns containing anchor texts and target URL, e.g. number of words in anchor texts, flag whether first word in anchor text is capitalized or not, domain part of the target URL, path level of the target URL and more.
When applying faceted browsing on a large number of rows it is not always possible to display all unique values. LODGrefine offers the ability to display all
refine/wiki/GRELFunctions different values by choice count, which can be further used in mathematical expressions. For example, if only a few anchor texts appear 100x more frequently than the rest of the anchor texts, it is difficult to filter out anchor texts with occurrences between 20 and 35. In this case it is better to use logarithmic scale. It is worth mentioning, that filtering in LODGrefine works really fast even for 100 000 rows, where some other tools might start having problems.
After filtering we reconciled entity candidates against DBpedia and/or Freebase to link them to existing entities and then exported entity candidates in Turtle representation. 4
LOD-enabled version of Google Refine is one of the best open-source tools for cleaning and linking. With the examples we demonstrated its versatility and powerfulness for transforming tabular data to Linked Data for different problem domains. 5
This work was supported by a grant from European Union’s 7th Framework Programme (2007-2013) provided for the project LOD2 (GA no. 257943).