Providing quality-aware techniques for reusing data available on the Web is a major concern for today's organizations. High quality data that o ers higher added-value to the stakeholders is called smart data. Smart data can be obtained by combining data coming from diverse data sources on the Web such as Web APIs, SPARQL endpoints, Web pages and so on. Generating smart data involves complex data processing tasks, typically realized manually or in a static way in current organizations, with the help of statically con gured work ows. In addition, despite the recent advances in this eld, transfering large amounts of data to be processed still remains a tedious task due to unreliable transfer conditions or transfer rate/latency problems. In this paper, we propose an adaptive architecture to generate smart data, and focus on a solution to handle volume diversity during data processing. Our approach aims at maintaining good response time performance upon user request. It relies on the use of RESTful resources and remote code execution over temporary data storage where business data is cached. Each resource involved in data processing accesses the storage to process data on-site.
During the last few years, governments, companies and organizations have opened
their databases and information systems to the world across the Web, thanks to
initiatives such as the open data project [
This smart use of data has caught the interest of the community as a natural
development. The objective of smart data [
In this paper, we present our adaptive architecture and propose a solution, through the use of adaptive data access strategies and remote code execution on temporary data storage unit resources, to handle latency and architecture issues that appear when processing data volume diversity. Our architecture optimizes and adapts work ows to handle the variety of data sources involved in the query. In this approach, we present this resource oriented architecture and explain our solution for reducing latency in resource oriented architecture.
This paper is organized as follows. Section 2 presents related approaches to handle volume diversity in data sources in RESTful architectures. Section 3 presents our resource oriented architecture and our solution to minimize data exchange. Section 4 gives an evaluation of our prototype in terms of responsiveness and shows how it responds to user requests with acceptable timings. Section 5 discusses our results and provides guidelines for future work. 2
Over the past few years, big data has generated a lot of interest from researchers
and industrials, answering to four challenges, known as the four Vs: Volume,
Variety, Velocity and Veracity. De ning our smart data architecture [
Devresse et Al. [
In order to handle our smart data challenges, we envision a resource-oriented
architecture, generating adaptive work ows at runtime to adapt to data source
characteristics. We rely on the data source models presented in our previous
work [
Then we de ne an resource oriented architecture to manage these adaptive work ows and the resources involved. 3.1
In our approach, we de ne di erent necessary steps to complete the data
aggregation process and produce smart data. The main steps are extraction from
data source and transformation into a pivot format, semantic annotation of
the extracted data set ([
In our architecture, upon query request, an adapted work ow is generated, involving resources and services, to handle the required processing tasks. This work ow is executed, our data bus handles data exchanges, generating HTTP requests and retrieving responses from resources. We identi ed the following data transfers during the process: extraction from data sources, data transfer between core resources, and data download by the user. In this context, data extraction and download are required, but transfer between resources can be avoided or modi ed.
Based on this observation, we modify our architecture to decrease the volume of exchange data between resources. We design our API to only manipulate queries and metadata (data models, etc.). Computing this metadata, remote processing codes are generated in order to complete the process managed by this resource. These processes are handled close to data, minimizing execution time by lowering latency and network time. 3.3
In order to temporarily store data, our proposal is based on temporary data storage units, generated at each user request. Storage units act as le hosting services, they are generated at runtime, for each new user request, for a limited amount of time and contains the data and metadata for this request. They are provided with an engine capable of processing remote processing tasks generated by resources. Storage units are erased after a certain amount time, which has been xed to a default value of 24 hours. This delay is customizable for each request. Time counter is reset each time a user reissues the query or reaccesses the storage. Storage units are accessible as RESTful resources, for management purpose or to retrieve query responses when data processing is over. When the processing tasks are over, the storage URI is given to the user, so that he could download the data sets answering to his request. 3.4
In order to handle the di erent tasks required to complete the smart data process, we provide the storages with a functional engine capable of executing remote processing tasks directly above the data instances. These codes are generated by the tasks resources and transfered instead of data.
We rely on functional languages to de ne processing tasks, since our data sets
are stored as tabular data sets (lists or dictionary in JSON or JSON-LD[
In this paper, we focus our work on handling data volume diversity in our architecture for smart data management.
In order to evaluate the scalability of our architecture, we demonstrate the evolution of performance, evaluating request response time when answering to a set of complex semantic queries over multiple data sources. We vary the number of data sources and measure response times. 4.1
We focus our work on the enrichment and reusability of data, and based our models and implementation on the data handled by a communication company, which has a need for an adaptive system able to automatically re ne and combine data coming from their internal information system and enrich it with help from open Web data sources. This solution has for objectives to study the impact of campaign broadcasts over a list of customers and to provide decision support tools for future broadcasts. This scenario describes the following data sources, presenting several characteristics, speci c to our scenario:
Source 1 is a linked service giving access to our company small business data set. Data that come from this source are subject to privacy constraints. Source 2 is a SQL endpoint to a database that contains millions of tuples with a low update frequency. Source 3 is another SQL endpoint to a database that contains customer daily activities, updated regularly. Source 4 is a RSS stream that contains user update requests (removal requests from customers, request form architecture's user). Other sources are represented by a set of Web sources: FOAF and vcard ontologies that help to annotate data, as well as a Dbpedia SPARQL endpoint. Relying on the scenario presented above, we create two request, involving di erent concepts subgraphs. We populate our scenario with a set of data sources, covering the di erent subgraphs.
PREFIX a l : <http : // r e s t f u l . a l a b s . i o / concepts /0.1/> PREFIX xsd : <http : //www. w3 . org /TR/xmlschema 2/> SELECT ? e m a i l v a l u e ? campaign WHERE f ? e m a i l a a l : e m a i l ; a l : h a s e m a i l v a l u e ? e m a i l v a l u e ; a l : b l a c k l i s t s t a t u s ? s t a t u s . ? c l i c a a l : c l i c ; a l : c l i c e m a i l ? e m a i l ; a l : c l i c d a t e ? date .
FILTER ( ? s t a t u s != 1 && ? date >= " 1 4 1 1 4 7 7 4 5 0 " ^^ xsd : date )
PREFIX a l : <http : // r e s t f u l . a l a b s . i o / concepts /0.1/> SELECT ? e m a i l v a l u e WHERE f ? e m a i l a a l : e m a i l ; a l : h a s e m a i l v a l u e ? e m a i l v a l u e ; a l : b l a c k l i s t s t a t u s ? s t a t u s .
FILTER ( ? s t a t u s != 1) g
Query 1.1 involves four subgraphs, implying data sources with di erent characteristics such as high volume (scenario's big database) and privacy sensitive information (scenario linked service). This query also presents user speci c lters. Query 1.2 involves only a few number of concepts, and present less data manipulations. Tests are performed on a double core 2.3 GHz machine, with 4 GB of RAM. Restful resources and architecture are implemented through PHP frameworks, Zend and Slim2 and hosted on scenario company servers (Apache). 4.2
We evaluate our architecture response time to query Q1, and Q2 respectively, when the number of data sources increases. We compared response time evolution for the same queries and data, with and without our optimisation process.
As can be seen in Fig. 2 and Fig. 3, the volume of data increase with the number of data source, and the classical approach response time su er, due to HTTP transfers. In parallel, optimized approach response time increase linearly, but stays acceptable standing under the treshold of 4 seconds. In this case, transfer and network latency consumes time exponentially. Query Q2 involves less concepts, and as a result less data manipulations, but our architecture still su er from a very high latency due to data transfer. When the number of data source exceed 24, the classical approach is unable to provide a response in an acceptable delay. This graph clearly show that our architecture can adapt to large volumes of data, using our optimisation technique. 2 See http://www.slimframework.com/ ) 10 s ( e m i .t 5 p s e R 0 8 ) (s 6 e it 4 m . p se 2 R 0
16 sources 20 sources 24 sources In this paper, we describe a resource-oriented architecture that relies on adaptive data processing strategies to optimize data exchanges between resources that process data. We focus on the latency problems that appear when dealing with diverse data volumes especially when transferring data between the di erent architecture components. All along the smart data construction process, we rely on a temporary data storage where business data is stored. Our architecture handles the communication between the di erent resources, by transferring metadata about the query and the data storage URI. Resources generate adapted remote processing codes, which are forwarded to storage units and executed on-site by the data storage engine. Therefore, data manipulation is performed on-site, and the data does not ow through the architecture. By reducing latency due to data transfers, we alleviate our decentralized and distributed architecture from the burden of data tranfer, and improve the responsiveness of data-driven resource work ows. We support our implementation with a set of evaluations and tests through our use case scenario. As future work, we envision to investigate the appearance of data uncertainty in data aggregating approach, relying on probabilistic integration techniques.