Linked Open Data (LOD) has emerged as one of the largest collections of interlinked datasets on the web. In order to bene t from this mine of data, one needs to access to descriptive information about each dataset (or metadata). This information can be used to delay data entropy, enhance dataset discovery, exploration and reuse as well as helping data portal administrators in detecting and eliminating spam. However, such metadata information is currently very limited to a few data portals where they are usually provided manually, thus being often incomplete and inconsistent in terms of quality. To address these issues, we propose a scalable automatic approach for extracting, validating, correcting and generating descriptive linked dataset pro les. This approach applies several techniques in order to check the validity of the metadata provided and to generate descriptive and statistical information for a particular dataset or for an entire data portal.
From 12 datasets cataloged in 2007, the Linked Open Data cloud has grown to
nearly 1000 datasets containing more than 82 billion triples3 [
Dataset discovery can be done through public data portals like Datahub.io
and publicdata.eu or private ones like quandl.com and enigma.io. Private
portals harness manually curated data from various sources and expose them
to users either freely or through paid plans. Similarly, in some public data
portals, administrators manually review datasets information, validate, correct and
attach suitable metadata information. This information is mainly in the form
of prede ned tags such as media, geography, life sciences for organization and
clustering purposes. However, the diversity of those datasets makes it harder
to classify them in a xed number of prede ned tags that can be subjectively
3 http://datahub.io/dataset?tags=lod
assigned without capturing the essence and breadth of the dataset [
There are several Data Management Systems (DMS) that power public data portals. CKAN4 is the world's leading open-source data portal platform powering web sites like DataHub, Europe's Public Data and the U.S Government's open data. Modeled on CKAN, DKAN5 is a standalone Drupal distribution that is used in various public data portals as well. Socrata6 helps public sector organizations improve data-driven decision making by providing a set of solutions including an open data portal. In addition to these tradition data portals, there is a set of tools that allow exposing data directly as RESTful APIs like thedatatank.com.
Metadata provisioning is one of the Linked Data publishing best practices
mentioned in [
Data pro ling is the process of creating descriptive information and collect
statistics about that data. It is a cardinal activity when facing an unfamiliar
dataset [
{ Metadata pro ling: Provides general information on the dataset (dataset description, release and update dates), legal information (license information, openness), practical information (access points, data dumps), etc. { Statistical pro ling: Provides statistical information about data types and patterns in the dataset (e.g. properties distribution, number of entities and RDF triples). { Topical pro ling: Provides descriptive knowledge on the dataset content and structure. This can be in form of tags and categories used to facilitate search and reuse.
In this work, we address the challenges of automatic validation and generation of descriptive dataset pro le. This paper proposes Roomba, an extensible framework consisting of a processing pipeline that combines techniques for data portals identi cation, datasets crawling and a set of pluggable modules combining several pro ling tasks. The framework validates the provided dataset metadata against an aggregated standard set of information. Metadata elds are automatically corrected when possible (e.g. adding a missing license URL reference). Moreover, a report describing all the issues highlighting those that cannot be automatically xed is created to be sent by email to the dataset's maintainer. There exist various statistical and topical pro ling tools for both relational and Linked Data. The architecture of the framework allows to easily add them as additional pro ling tasks. However, in this paper, we focus on the task of dataset metadata pro ling. We validate our framework against a manually created set of pro les and manually check its accuracy by examining the results of running it on various CKAN-based data portals.
The remainder of the paper is structured as follows. In Section 2, we review relevant related work. In Section 3, we describe our proposed framework's architecture and components that validate and generate dataset pro les. In Section 4, we evaluate the framework and we nally conclude and outline some future work in Section 5. 2
Data Catalog Vocabulary (DCAT) [
Metadata pro ling: The Project Open Data Dashboard9 tracks and measures how US government web sites implement the Open Data principles to understand the progress and current status of their public data listings. A validator analyzes machine readable les: e.g. JSON les for automated metrics like the resolved URLs, HTTP status and content-type. However, deep schema information about the metadata is missing like description, license information or tags. Similarly on the LOD cloud, the Datahub LOD Validator10 gives an overview of Linked Data sources cataloged on the Datahub. It o ers a step-bystep validator guidance to check a dataset completeness level for inclusion in the LOD cloud. The results are divided into four di erent compliance levels from basic to reviewed and included in the LOD cloud. Although it is an excellent tool to monitor LOD compliance, it still lacks the ability to give detailed insights about the completeness of the metadata and overview on the state of the entire LOD cloud group and it is very speci c to the LOD cloud group rules and regulations.
Statistical pro ling: Calculating statistical information on datasets is vital
to applications dealing with query optimization and answering, data cleansing,
schema induction and data mining [
Topical Pro ling: Topical and categorical information facilitates dataset
search and reuse. Topical pro ling focuses on content-wise analysis at the
instances and ontological levels. GERBIL [
work that provides machine processable output allowing e cient querying. In
addition, there exist several entity annotation tools and frameworks [
Although the above mentioned tools are able to provide various types of information about a dataset, there exists no approach that aggregates this information and is extensible to combine additional pro ling tasks. To the best of our knowledge, this is the rst e ort towards extensible automatic validation and generation of descriptive dataset pro les. 3
In this section, we provide an overview of Roomba's architecture and the processing steps for validating and generating dataset pro les. Figure 1 shows the main steps which are the following: (i) data portal identi cation; (ii) metadata extraction; (iii) instance and resource extraction; (iv) pro le validation (v) pro le and report generation.
Roomba is built as a Command Line Interface (CLI) application using Node.js. Instructions on installing and running the framework are available on its public Github repository11. The various steps are explained in detail below. 3.1
Roomba should be extensible to any data portal that exposes its functionalities via an external accessible API. Since every portal ca have its own data model, identifying the software powering data portals is a vital rst step. We rely on several Web scraping techniques in the identi cation process which includes a combination of the following: { URL inspection: Various CKAN based portals are hosted on subdomains of the http://ckan.net. For example, CKAN Brazil (http://br.ckan. net). Checking the existence of certain URL patterns can detect such cases. { Meta tags inspection: The <meta> tag provides metadata about the HTML document. They are used to specify page description, keywords, author, etc. Inspecting the content attribute can indicate the type of the data portal. We use CSS selectors to check the existence of these meta tags. An example of a query selector is meta[content*=``ckan''] (all meta tags with 11 https://github.com/ahmadassaf/opendata-checker the attribute content containing the string CKAN ). This selector can identify CKAN portals whereas the meta[content*=``Drupal''] can identify DKAN portals. { Document Object Model (DOM) inspection: Similar to the meta tags inspection, we check the existence of certain DOM elements or properties. For example, CKAN powered portals will have DOM elements with class names like ckan-icon or ckan-footer-logo. A CSS selector like .ckan-icon will be able to check if a DOM element with the class name ckan-icon exists. The list of elements and properties to inspect is stored in a separate con gurable object for each portal. This allows the addition and removal of elements as deemed necessary.
The identi cation process for each portal can be easily customized by overriding the default function. Moreover, adding or removing steps from the identi cation process can be easily con gured.
After those preliminary checks, we query one of the portal's API endpoints. For example, DataHub is identi ed as CKAN, so we will query the API endpoint on http://datahub.io/api/action/package\_list. A successful request will list the names of the site's datasets, whereas a failing request will signal a possible failure of the identi cation process. 3.2
Data portals expose a set of information about each dataset as metadata. The model used varies across portals. However, a standard model should contain information about the dataset's title, description, maintainer email, update and creation date, etc. We divided the metadata information into the following types:
General information: General information about the dataset. e.g., title, description, ID, etc. This general information is manually lled by the dataset owner. In addition to that, tags and group information is required for classi cation and enhancing dataset discoverability. This information can be entered manually or inferred modules plugged into the topical pro ler.
Access information: Information about accessing and using the dataset. This includes the dataset URL, license information i.e., license title and URL and information about the dataset's resources. Each resource has as well a set of attached metadata e.g., resource name, URL, format, size.
Ownership information: Information about the ownership of the dataset. e.g., organization details, maintainer details, author. The existence of this information is important to identify the authority on which the generated report and the newly corrected pro le will be sent to.
Provenance information: Temporal and historical information on the dataset and its resources. For example, creation and update dates, version information, version, etc. Most of this information can be automatically lled and tracked.
Building a standard metadata model is not the scope of this paper, and since we focus on CKAN-based portals, we validate the extracted metadata against the CKAN standard model12.
After identifying the underlying portal software, we perform iterative queries to the API in order to fetch datasets metadata and persist them in a le-based cache system. Depending on the portal software, we can issue speci c extraction jobs. For example, in CKAN-based portals, we are able to crawl and extract the metadata of a speci c dataset, all the datasets in a speci c group (e.g. LOD cloud) or all the datasets in the portal. 3.3
From the extracted metadata we are able to identify all the resources associated with that dataset. They can have various types like a SPARQL endpoint, API, le, visualization, etc. However, before extracting the resource instance(s) we perform the following steps: { Resource metadata validation and enrichment: Check the resource attached metadata values. Similar to the dataset metadata, each resource should include information about its mimetype, name, description, format, valid de-referenceable URL, size, type and provenance. The validation process issues an HTTP request to the resource and automatically lls up various missing information when possible, like the mimetype and size by extracting them from the HTTP response header. However, missing elds like name and description that needs manual input are marked as missing and will appear in the generated summary report. { Format validation: Validate speci c resource formats against a linter or a validator. For example, node-csv13 for CSV les and n314 to validate N3 and Turtle RDF serializations. 12 http://demo.ckan.org/api/3/action/package\_show?id=adur\_district\ _spending 13 https://github.com/wdavidw/node-csv 14 https://github.com/RubenVerborgh/N3.js
Considering that certain datasets contain large amounts of resources and the
limited computation power of some machines on which the framework might
run on, a sampler module can be introduced to execute various sample-based
strategies detailed as they were found to generate accurate results even with
comparably small sample size of 10%. These strategies introduced in [
However, the sampler is not restricted only to these strategies. Strategies
like those introduced in [
A dataset pro le should include descriptive information about the data examined. In our framework, we have identi ed three main categories of pro ling information. However, the extensibility of our framework allows for additional pro ling techniques to be plugged in easily (i.e. a quality pro ling module reecting the dataset quality). In this paper, we focus on the task of metadata pro ling.
Metadata validation process identi es missing information and the ability to automatically correct them. Each set of metadata (general, access, ownership and provenance) is validated and corrected automatically when possible. Each pro ler task has a set of metadata elds to check against. The validation process check if each eld is de ned and if the value assigned is valid.
There exist many special validation steps for various elds. For example, the email addresses and urls should be validated to ensure that the value entered is syntactically correct. In addition to that, for urls, we issue an HTTP HEAD request in order to check if that URL is reachable. We also use the information contained in a valid content-header response to extract, compare and correct some resources metadata values like mimetype and size.
From our experiments, we found out that datasets' license information is noisy. The license names if found are not standardized. For example, Creative Commons CCZero can be also CC0 or CCZero. Moreover,the license URI if found and if de-referenceable can point to di erent reference knowledge bases e.g., http://opendefinition.org. To overcome this issue, we have manually created a mapping le standardizing the set of possible license names and the reference knowledge base15. In addition, we have also used the open source and 15 https://github.com/ahmadassaf/opendata-checker/blob/master/util/ licenseMappings.json knowledge license information16 to normalize the license information and add extra metadata like the domain, maintainer and open data conformance. f g , f g " l i c e n s e i d " : [ "ODC PDDL 1.0" ] , " disambiguations " : [ "Open Data Commons Public Domain Dedication and License (PDDL) " ] " l i c e n s e i d " : [ "CC BY SA 4.0" , "CC BY SA 3.0" ] , " disambiguations " : [ " cc by sa " , "CC BY SA" , " Creative Commons A t t r i b u t i o n Share Alike " ]
Listing 1.1. License mapping le sample 3.5
The validation process highlights the missing information and presents them in a human readable report. The report can be automatically sent to the dataset maintainer email if exists in the metadata. In addition to the generated report, the enhanced pro les are represented in JSON using the CKAN data model and are publicly available17.
Data portal administrators need an overall knowledge of the portal datasets and their properties. Our framework has the ability to generate numerous reports of all the datasets by passing formatted queries. There are two main sets of aggregation tasks that can be run: { Aggregating meta- eld values: Passing a string that corresponds to a valid eld in the metadata. The eld can be at like license title (aggregates all the license titles used in the portal or in a speci c group) or nested like resource>resource type (aggregates all the resources types for all the datasets). Such reports are important to have an overview of the possible values used for each metadata eld. { Aggregating key:object meta- eld values: Passing two meta- eld values separated by a colon : e.g., resources>resource type:resources>name. These reports are important as you can aggregate the information needed when also having the set of values associated to it printed.
For example, the meta- eld value query resource>resource type run against the LODCloud group will result in an array containing [f ile; api; documentation:::] values. These are all the resource types used to describe all the datasets of the group. However, to be able to know also what are the datasets containing resources corresponding to each type, we issue a key:object meta- eld query resource>resource type:name. The result will be a JSON object having the resource type as the key and an array of corresponding datasets titles that has a resource of that type. 16 https://github.com/okfn/licenses 17 https://github.com/ahmadassaf/opendata-checker/tree/master/results =======================================================================
Metadata Report ======================================================================= group i n f o r m a t i o n i s m i s s i n g . Check o r g a n i z a t i o n i n f o r m a t i o n as they can be mixed sometimes o r g a n i z a t i o n i m a g e u r l f i e l d e x i s t s but t h e r e i s no value d e f i n e d =======================================================================
Tag S t a t i s t i c s =======================================================================
There i s a t o t a l o f : 21 [ undefined ] v o c a b u l a r y i d f i e l d s 100.00% =======================================================================
L i c e n s e Report =======================================================================
L i c e n s e i n f o r m a t i o n has been normalized ! =======================================================================
Resource S t a t i s t i c s ======================================================================= There i s a t o t a l o f : 10 [ m i s s i n g ] url type f i e l d s 100.00% There i s a t o t a l o f : 9 [ m i s s i n g ] c r e a t e d f i e l d s 90.00% There i s a t o t a l o f : 10 [ undefined ] c a c h e l a s t u p d a t e d f i e l d s 100.00% There i s a t o t a l o f : 10 [ undefined ] s i z e f i e l d s 100.00% There i s a t o t a l o f : 10 [ undefined ] hash f i e l d s 100.00% There i s a t o t a l o f : 10 [ undefined ] mimetype inner f i e l d s 100.00% There i s a t o t a l o f : 7 [ undefined ] mimetype f i e l d s 70.00% There i s a t o t a l o f : 10 [ undefined ] c a c h e u r l f i e l d s 100.00% There i s a t o t a l o f : 6 [ undefined ] name f i e l d s 60.00% There i s a t o t a l o f : 9 [ undefined ] w e b s t o r e u r l f i e l d s 90.00% There i s a t o t a l o f : 9 [ undefined ] l a s t m o d i f i e d f i e l d s 90.00% There i s one [ undefined ] format f i e l d 10.00% =======================================================================
Resource C o n n e c t i v i t y I s s u e s ======================================================================= There are 2 c o n n e c t i v i t y i s s u e s with the f o l l o w i n g URLs :
n u r l f http : / / dbpedia . org / void / Dataset g =======================================================================
Un Reachable URLs Types ======================================================================= There are : 1 unreachable URLs o f type [ f i l e ]
Listing 1.2. Excerpt of the DBpedia validation report
In this section, we provide the experiments and evaluation of the proposed framework. All the experiments are reproducible by our tool and their results are available in its Github repository. A CKAN dataset metadata describes four main sections in addition to the core dataset's properties. These sections are: { Resources: The distributable parts containing the actual raw data. They can come in various formats (JSON, XML, RDF, etc.) and can be downloaded or accessed directly (REST API, SPARQL endpoint). { Tags: Provide descriptive knowledge on the dataset content and structure.
They are used mainly to facilitate search and reuse. { Groups: A dataset can belong to one or more group that share common semantics. A group can be seen as a cluster or a curation of datasets based on shared categories or themes. { Organizations: A dataset can belong to one or more organization controlled by a set of users. Organizations are di erent from groups as they are not constructed by shared semantics or properties, but solely on their association to a speci c administration party.
Each of these sections contains a set of metadata corresponding to one or more type (general, access, ownership and provenance). For example, a dataset resource will have general information such as the resource name, access information such as the resource url and provenance information such as creation date. The framework generates a report aggregating all the problems in all these sections, xing eld values when possible. Errors can be the result of missing metadata elds, unde ned eld values or eld value errors (e.g. unreachable URL or incorrect email addresses). 4.1
We ran our tool on two CKAN-based data portals. The rst one is datahub.io
targeting speci cally the LOD cloud group. The current state of the LOD cloud
report [
We ran the Roomba instance and resource extractors in order to cache the metadata les for these datasets locally and ran the validation process. The experiments were executed on a 2.6 Ghz Intel Core i7 processor with 16GB of DDR3 memory machine. The approximate execution time alongside the summary of the datasets' properties are presented in table 1.
In our evaluation, we focused on two aspects: i)pro ling correctness which manually assesses the validity of the errors generated in the report, and ii)pro ling 18 http://data.amsterdamopendata.nl/ completeness which assesses if the pro lers cover all the errors in the datasets metadata. 4.2
To measure pro le correctness, we need to make sure that the issues reported by Roomba are valid on the dataset, group and portal levels.
On the dataset level, we choose three datasets from both the LOD Cloud and the Amsterdam data portal. The datasets details are shown in table 2.
To measure the pro ling correctness on the groups level, we selected four groups from the Amsterdam data portal containing a total of 25 datasets. The choice was made to cover groups in various domains that contain a moderate number of datasets that can be checked manually (between 3-9 datasets). Table 3 summarizes the groups chosen for the evaluation.
After running Roomba and examining the results on the selected datasets and groups, we found out that our framework provides 100% correct results on the individual dataset level and on the aggregation level over groups. Since our portal level aggregation is extended from the group aggregation, we can infer that the portal level aggregation also produces complete correct pro les. However, the lack of a standard way to create and manage collections of datasets was the source of some errors when comparing the results from these two portals. For example, in Datahub, we noticed that all the datasets groups information were missing, while in the Amsterdam Open Data portal, all the organisation information was missing. Although the error detection is correct, the overlap in the usage of group and organization can give a false indication about the metadata quality. 4.3
We analyzed the completeness of our framework by manually constructing a set of pro les that act as a golden standard. These pro les cover the range of uncommon problems that can occur in a certain dataset19. These errors are: { Incorrect mimetype or size for resources; { Invalid number of tags or resources de ned; { Check if the license information can be normalized via the license id or the license title as well as the normalization result; { Syntactically invalid author email or maintainer email.
After running our framework at each of these pro les, we measured the completeness and correctness of the results. We found out that our framework covers indeed all the metadata problems that can be found in a CKAN standard model correctly. 5
In this paper, we proposed a scalable automatic approach for extracting, validating, correcting and generating descriptive linked dataset pro les. This approach applies several techniques in order to check the validity of the metadata provided and to generate descriptive and statistical information for a particular dataset or for an entire data portal. Based on our experiments running the tool on the LOD cloud, we discovered that the general state of the datasets needs attention as most of them lack informative access information and their resources su er low availability. These two metrics are of high importance for enterprises looking to integrate and use external linked data.
It has been noticed that the issues surrounding metadata quality a ect directly dataset search as data portals rely on such information to power their search index. We noted the need for tools that are able to identify various issues in this metadata and correct them automatically. We evaluated our framework manually against two prominent data portals and proved that we can automatically scale the validation of datasets metadata pro les completely and correctly.
As part of our future work, we plan to introduce work ows that will be able to correct the rest of the metadata either automatically or through intuitive manually-driven interfaces. We also plan to integrate statistical and topical pro lers to be able to generate full comprehensive pro les. We also intend to suggest a ranked standard metadata model that will help generate more accurate and scored metadata quality pro les. We also plan to run this tool on various CKAN-based data portals, schedule periodic reports to monitor the evolvement of datasets metadata. Finally, at some stage, we plan to extend this tool for other data portal types like DKAN and Socrata. 19 https://github.com/ahmadassaf/opendata-checker/tree/master/test
This research has been partially funded by the European Union's 7th Framework Programme via the project Apps4EU (GA No. 325090).