Conventional data analytics platforms are not adequate to be applied in the management of emergency situations. The 3V the usually characterize big data (volume, variety, velocity) along with the issue of integrating information coming from heterogeneous networks require the development of new systems. In this paper we provide the design of a data analytics platform that we are developing around the concept of event, that is simple or complex data stream gathered from physical and social sensors that are encapsulated with contextual information (space, time, thematics).
Nowadays we are witnesses of the proliferations of different sensor devices able to
produce heterogeneous types of data (textual, visual, audio, and other rich multimedia
formats) that can be profitable used for detecting, handling and advising people of the
verification of emergency events such as disasters due to natural phenomena (like
flooding, storming, extreme temperatures etc.). Beside the physical sensors, able to
detect data about physical phenomena (like temperature, humidity, wind, rain, pressure,
level of see water), there is a proliferation of social sensors able to collect data from
people (like twitter data, traffic information, train or flight schedule) [
elaborated during the emergency event. Therefore, scalable and efficient solutions should be devised that can be applied on-line. ETL (extract, transform and load) solutions, usually applied off-line, need to be revised and applied on-line for feeding the DW (data warehouse) with fresh and timely data. Finally, an user-friendly environment has been conceived for properly helping the user in the different phases of the analysis processes (sensor discovery, feeding, and knowledge inference).
In this paper we propose a novel OLAP Analytics Platform at the level of event for analyzing multi-dimensional relations between multiple data streams based on event attributes (spatial, temporal and thematic attributes). The output would be spatial-temporal-thematic aggregates of multiple data streams (e.g., physical and social sensing data streams) representing complex events (e.g., social response to natural disaster) organized at different spatio-temporal granularities..
In Section 2 the new architecture will be illustrated and compared with respect to conventional Data Analytic Platform. Section 3 describes the event data model tailored for handling multidimensional complex events according to a multigranular spatio-temporal-thematic (STT) data model. Section 4 starts with discussing the description of the services that should be developed for the Event ETL, for then describing the user interface developed for creating a workflow based on the combinations of different data streams. Finally, we provide a description of the current efforts and detail for developing the Event OLAP interface, which aim is to support users in analyzing data coming from different sensors and detect possible correlation and significant events. 2
In conventional OLAP system there is a distinction between ETL operations and OLAP
operations. The former are used for feeding the DW, whereas the latter are used to query
the DW once the cube has been defined. Moreover, feeding the DW is considered a
preprocessing phase that is taken off-line with the purpose to solve many issues (extracting,
cleaning and riconciliate heterogeneous data and denormalize data) in order to make
somehow simpler the execution of the OLAP operations. Stream DW have been also
proposed for handling information produced in streams and approaches based on
sliding windows have been introduced to manage continuous data [
Recently the term “active” or “real-time” DW [
Copyright © 2014 for the individual papers by the papers' authors. Copying permitted for private and academic purposes. This volume is published and copyrighted by its editors.
First of all, we need to point out the adoption of a different data model. Indeed,
conventional DWs adopt the relational model for the representation of the information
on the storage. This is not adequate in the Event warehouse because of the heterogeneity
of the data formats used and also the fact that information is stored at different
granularities (ranging from simple raw values of temperature to structured and complex data
like the objects extracted from a picture, or the mood inferred from tweet data with their
level of trustiness). Second, conventional systems process stored data, whereas in the
Event warehouse streams of data should be processed and collected and analyses
through temporal windows. These streams can be stored in the DW or only aggregated
information are stored. Orthogonally, contextual information can be associated with the
stream by means of machine learning algorithms and exploited for extracting
knowledge. For what concern the implementation of the architecture and of the required
operations, we are considering the use of cloud-based architectures because we need to
guarantee high throughput and a low end-to-end latency from the system, despite
possible fluctuations in the workload. We are currently evaluating several architectures like
Apache S4 [
In order to represent events, we consider the set of basic and structured (like list and
records) types whose values are represented thorough a JSON-like notation. These
types allow to represent different aggregation of values without imposing the
restrictions of the relational model and so be able to handle a great variety of datatypes.
Moreover, we consider spatial and temporal types at different granularities. Temporal
granularities include seconds, minutes, days with the usual meaning adopted in the
Gregorian calendar, whereas, meters, kilometers, feet, yards, provinces and countries are
examples of spatial granularities. Different granularities provide different partitions of
their domains because of the diverse relationships that can exist among granularities,
depending on the inclusion and the overlapping of granules [
For example, temporal granularity seconds is finer than minutes, and granularity months is finer than years. Likewise, spatial granularity provinces is finer than regions. Thematics are also considered for associating to a given value a semantic annotation. For example, the annotation HR,MR,LR can be associated to the real number representing the degree (high, medium, low) of rain that is precipitated in a given area. Several thematics can be identified depending on the context where the datum is acquired or processed. Thematics can indeed be inferred by machine learning algorithms. Relying on the concepts of temporal and spatial granularities, we present the concept of event, that is a value associated with a spatial object at a given time according to given thematics. Therefore, an event is a value represented at a given spatio-temporal granularity for which thematic information is added. Relying on the concept of events, we can characterize an event stream that a source can produce. A source can be sensor (either physical or virtual) or a service (for example by aggregating sensors/services streams). Proc. of Third International Workshop on Cultures of Participation in the Digital Age - CoPDA 2015 Madrid (Spain), May 26th, 2015 (published at http://ceur-ws.org).
Copyright © 2014 for the individual papers by the papers' authors. Copying permitted for private and academic purposes. This volume is published and copyrighted by its editors.
At the bottom of our architecture, a middleware provides description of the available sensors and offers several services including service discovery, service monitoring, execution control, and service message exchanges. These services are exploited for executing Event ETL operators over a programmable network (like SDN) efficiently and effectively (e.g., filter by in-network data processing). Currently we developed three kinds of operators that are relevant in our context: conversion, merging and connecting operators. Conversion operators are used for changing the spatio-temporal-thematics granularity of an event stream. Merging operators are used for merging events at the same spatio-temporal granularity in a single data structure. Connecting operators are used to link a data source to another by applying a SQL joint-like operator. Following, for reasons of space the paper does not describe in detail the implementation of these operators focusing only on the graphical visual interface using which the user can identify the sensors and apply the operators for converting, merging and combining their data stream. Finally, we provide the description of the Event OLAP system used for analyzing the data and for extracting knowledge. 4.1
ETL operations have been proposed in different contexts depending on the kinds of
data to handle (structured and semi-structured). In [
In our context, we designed a Web environment where the users have the possibility to drag and drop different sensor data sources and visually apply on them a set of operations. This application offers an engine and graphical environment for data transformation and mashup. As depicted in Fig. 1 (section A), this mash-up consists of a user interface that contextually displays icons of data sources or operations in order to link, filter or merge data coming from different sensors. It is based on the idea of providing a visual workflow generator for letting the end user creating aggregation, filtering, and porting of data originated by sources. An advanced use of such visual paradigm allows the end users to have an online generation of sample data coming from the data sources dragged-and-dropped on the canvas, or as result of the operations carried out on them (see Fig. 1 section B). This End User development strategy offers a solution able to gather information from across the net and trigger specific filters and operations. This solution enables a very simple and easy to learn solution based on the definition of Proc. of Third International Workshop on Cultures of Participation in the Digital Age - CoPDA 2015 Madrid (Spain), May 26th, 2015 (published at http://ceur-ws.org).
Copyright © 2014 for the individual papers by the papers' authors. Copying permitted for private and academic purposes. This volume is published and copyrighted by its editors.
sources to use for collecting data and on the possibility to apply converting, merging and connecting operations in a visual way.
Future activates aim at developing machine-learning algorithms for exploiting the possible operations that users can apply to a set of data sources or for providing them useful predictions of what can happen by integrating the selected data. 4.2
Traditional OLAP front-ends, designed primarily to support routine reporting and
analysis and offering visualization merely for expressive presentation of the data [
Event OLAP aims at providing a much more powerful data analysis environment, accessible through an intuitive user interface for quickly analyzing STT events. Event OLAP delivers a web-based interactive environment that allows non-technical users but experts of the analysis domain, to explore data in real-time by slicing and dicing, pivoting, filtering, and summarizing them in an intuitive way. As depicted in Fig. 2, it is based on a 3D map-based visualization through which carried out analysis and monitoring of trajectories (that is, spatio-temporal movements) of data streams that are part of an event (e.g. a thunderstorm). The aim of this environment is to provide users with a visualization of the data streams coming from different sensors in order to capture possible visual correlations among data. The visual perception of these correlations can be used for integrating correlations that can be detected in automatic way by applying Proc. of Third International Workshop on Cultures of Participation in the Digital Age - CoPDA 2015 Madrid (Spain), May 26th, 2015 (published at http://ceur-ws.org).
Copyright © 2014 for the individual papers by the papers' authors. Copying permitted for private and academic purposes. This volume is published and copyrighted by its editors.
data mining techniques on the data streams. Visual display of events’ trajectories and collection of movement statistics can pro-vide useful indicators about how an event stream can influence another event streams. Moreover, current efforts are addressed to endow the web application with the possible to modify the workflow designed by using the Visual Event ETL in order to visualize the related changes in real time on the on the Event OLAP system.
Other activities aim at extending this web application with a location intelligence visualization strategy to identify patterns and trends by seeing and analyzing data in a map view with spatial analysis tools such as thematic maps and spatial statistics. This location intelligence service will help to find data by using spatial relationships to filter relevant data. Moreover, a temporal condition of this location intelligence service is applied for providing spatio-temporal clusters, simulation and visualization, map animation and movement tracking. To finish, in order to take into account the social aspect of the events collected by the Event OLAP, it will be endowed with a set of functionalities for creating a social network of users that will promote the creation of communities around each event. This social component exploiting ad-hoc computing techniques is able to study social network dynamics and to promote crowd-sourcing analysis for new and meaningful uses of data. Proc. of Third International Workshop on Cultures of Participation in the Digital Age - CoPDA 2015 Madrid (Spain), May 26th, 2015 (published at http://ceur-ws.org).
Copyright © 2014 for the individual papers by the papers' authors. Copying permitted for private and academic purposes. This volume is published and copyrighted by its editors. In this paper we have presented the architecture of an Event OLAP system specifically conceived for the management of emergency situation. We have detailed some peculiarity of the system and compare it with conventional OLAP systems and current research efforts. We are currently working on the formal specification of the needed operations at the ETL, DW and OLAP levels. Moreover, we are choosing the Cloud architecture in which the different services will be implemented and tested. For the testing we will exploit the collection of event data made available by NICT. 6