As big data is expected to contribute largely to economic growth, scalability of solutions becomes apparent for deployment by organisations. It requires automatic collection and processing of large, heterogeneous data sets of a variety of resources, dealing with various aspects like improving quality, fusion and linking data sets to provide homogeneous data to data analytics algorithms that are able to detect patterns or anomalies. This paper introduces two components in a data sharing architecture for big data. It presents the state of the art as basis for a research agenda. Aspects like ransport, storage, and processing are important for big data, but not addressed by this paper.
Big – and open data are mentioned as the most important technology trends, contributing growth to our society and
economy (Brynjolfsson, 2012),
An analysis of data sharing platforms is already done
First of all, the data value chain and resource orientation are discussed and secondly the architectural components are presented for identifying research questions. When applicable, available software components that partly the required functionality is presented.
Copyright © 2015 by the paper’s authors. Copying permitted only for private and academic purposes.
This section introduces the data value chain
This section briefly introduces the data value chain and discusses its functionality. In order to create a data collection
architecture as a framework for research questions, a more elaborate and detailed model is required. Using expert
interviews, case interviews, and workshops as primary input – combined with desk research
According to the data value chain, data governance and security is part of the core of analytics. However, data
governance with interventions like Identification and Authentication is a requirement of a data owner to stay in
control of its particular data
This section briefly presents two design principles. The first is to apply semantic web standards like Ontology Web
Language (OWL) and Resource Description Format (RDF) for both modelling knowledge and linking data sets by
constructing networked ontologies of those sets, thus creating Linked Data
The second design principle is to consider every thing, person, and organization as a data resource from an
Information - and Communication Technology (ICT) perspective. A package, a container, a truck, a truck driver, and
a carrier are for instance all considered as a ‘resource’ in logistics. Each of these resources can have some type of
autonomy represented by its goals and capabilities
Resources are autonomous, which imposes data sharing restrictions from a legal - like privacy or liability or a
commercial perspective. Since data is currently also considered as an asset
The previous section introduced the data value chain and two design principles, resulting in requirements to ICT components. This section introduces two ICT components that can be further decomposed, namely a a registry component and a data processing engine. Both components have to interface with each other and can be deployed in various ways, e.g. a data-sharing platform or peer-to-peer solutions. Deployment aspects will not be discussed; requirements to functionality of both components are further discussed. By identifying functionality, research questions can be formulated.
In the past, various forms of registries have been specified, e.g. Uniform Description Discovery and Integration
(UDDI) for web services
Semantics of a resource, that we will express as ontology (see before), can be specified at different levels, e.g. per
API, per data set of a resource, or a common information model of various APIs or data sets
As we have seen, not only semantics and technical interface with communication protocols and Uniform Resource
Locator (URL) are important, but also the annotation of resource and its data in terms of data quality
In open data communities, platforms like CKAN or WSO2 API manager are applied as registries, whereas the
functionality of these platforms differs
Queries can also be expressed in semantics using for instance SPARQL. Such a query might retrieve one or more URLs of data sets or resources with a required annotation or to decompose the query into queries directly to two or more resources (query federation). The latter might result in a particular sequencing in which these resources should be queried, also including functionality like combining data sets of different resources to one or validating consistency across data sets of different resources. Such sequencing is represented by a (data) workflow with particular data manipulation functions (see next section). A data collection strategy might provide rules that govern the data workflow. These rules are for further research.
The objective of a data collection engine is to homogenise data of heterogeneous data sets, potentially link or fuse
those data sets, and improve data quality (see before). Data collection utilizes one or more of the three mechanisms
mentioned before
Data semantics and – annotation, expressing data quality and other aspects of both a data set and data resource,
determine data processing. More particular, the following functionality is required to automatically collect and
process large (structured) data sets:
- Data workflow engine. A data collection strategy based on data set annotation governs collection and
preprocessing. A data workflow engine provides such a control by for instance collecting and comparing data of two
or more resources for validating (in)consistencies. Such an engine can also support the various ways of data
collection
This section introduces two main research questions that can be further specified. These two main research questions that will be further detailed, are: 1. Resource profiles to find and collect (large) data sets in these adaptive complex systems. 2. Seamless interoperability for creating adaptive complex systems.
Besides these basic research questions, there are also systems architecture questions, like the interface between a registry and a data collection engine. Furthermore, individual components need to interface to construct a solution that can be deployed. We will not address these questions in this paper; they are very relevant for actual deployment.
In the previous sections, we have argued that it is difficult to find useful, annotated data due to all types of reasons.
We have also argued on the importance of data set annotation for data processing. Lack of semantics is one of the
reasons that we will address in section 4.2. We have also introduced some definitions in a loosely way, e.g. resource,
data set, and instance of a data set. Proper definitions are required to be able to develop software, like a resource is
the owner or keeper of one or more data sets, where each data set has a known semantics and many instances.
This section identifies the following research questions that build upon these definitions:
- Profile of a data set of a resource. There are not yet profiles specified that express available instances of a data set
in a known semantic model andprovide technical details as to how instances can be collected. Such a profile should
also include the value instances of a data setcan provide expressed in known semantics (e.g. a temperature in degrees
Celsius) or more complex ones like utilization of water depths for vessels. An example of profiles can be found in
A resource can have one or more data sets. In some occasions, a profile is already standardised by a community,
e.g. based on a common semantic model with its supporting technical solutions. In those occasions, that profile acts
as a template implemented by more than one resource. Its semantics is probably expressed by an ontology that can be
matched or linked to existing ontologies
Various sources have stressed the importance of seamless interoperability, e.g.
Seamless interoperability can be expressed at different levels
Semantic transformation addresses is required if a profile of a data set is going to be implemented by a data
resource or at data collection. Semantic – and syntactic behaviour is required for adaptive complex systems like
smart cities and logistics in which various application domains have to be interoperable and where the constellation
of participating resources changes dynamically. These systems have a network (see for instance
We have constructed an architecture with two basic components supporting to support a big data value chain and
used this architecture to identify research questions. Besides the basic research questions of profiling and seamless
interoperability, there needs to be further research towards deployment of an architecture that utilizes distributed
transport, processing, and storage of large data sets. For instance, to prevent unnecessary data sharing across the
Internet, it might be feasible to analyse data sets locally by transporting a data collection engine and an analytics tool
to a resource and provide the results to another data collection engine. These aspects, covered by for instance
Software Defined Networks
Another research issue is to apply semantics in data analytics. Current algorithms are agnostics of data semantics;
they just require homogeneous data sets to detect patterns and anomalies based on statistic or neuristic algorithms
Although this is not explicitly mentioned in our paper, the focus has been primarily on structured data sets
Available at: bigdatavalue.eu [Accessed 22 March 2015].