-This paper presents IoT-Hub a new scalable, elastic, efficient, and portable Internet of Things (IoT) data-platform based on microservices for monitoring and analysing largescale sensor data in real-time. IoT-Hub allows us to collect, process, and store large amounts of data from multiple sensors in distributed locations-which could be deployed as a backend for Virtual Research Environments (VRE) or Science Gateways. In the proposed data-platform, all required software, which involves a variety of state-of-the-art open-source middleware, is packed into containers and deployed in a cloud environment. As a result, the engineering and computational time and costs for deployment and execution is significantly reduced.
The emergence of the Internet of Things (IoT) is introducing
a new era to the realm of computing and technology [
Virtual Research Gateways (VREs), also known as Science
Gateways [
In this paper, we present IoT-Hub, an integrated, comprehensive, elastic, and portable data-platform based on microservices. IoT-Hub combines the benefits of several wellknown data-frameworks with Docker containers. The current implementation of IoT-Hub includes a service-pipeline composed by Apache Kafka, Apache Spark, Elasticsearch, and Kibana middleware that enables automated gathering, preprocessing, storing, and visualization of IoT streams in a scalable, efficient, and robust manner. IoT-Hub acts as a backend for VREs to run stream-based applications, deploying cloud resources upon request. It reduces the engineering time and effort (and possible human errors) required by scientists or VRE administrators to build such complex systems. Our hypothesis is that if we provide scientific communities with portable and elastic platforms to interrogate the IoT data, it will speed up scientific discoveries.
We have demonstrated the feasibility of IoT-Hub via a
real use case application, which processes sensor data from
the British Geological Survey (BGS) environmental baseline
programme [
This paper is structured as follows. Section II presents background. Section III discusses IoT-Hub features. Section IV presents the use case for testing the platform. We conclude with a summary of achievements and outline future work.
VREs can be defined as community-development set of
tools, applications, and data that is integrated via a portal or a
suite of applications, usually in a graphical user interface, that
is further customized to meet the needs of a specific
community [
VERCE [
MoSGrid [
CyberSKA [
EFFORT [
Having a closer look to the technologies, tools, systems, and computing resources that are very often behind VREs’ backends, we can categorize them as follows:
puting resources to perform high performance
computations (including processors, memory, disk, and operating
system) [
tems characterized by heterogeneous networked computers called to offer data processing facilities. This includes high-throughput computing and cloud computing;
tems enacting the definition and execution of
scientific workflows consisting of a list of tasks and
operations, the dependencies between the interconnected
tasks, control-flow structures, and the data resources to
be processed [
workbenches enabling scientists to execute analytic tasks.
Such platforms tend to provide their users with
implementations of algorithms and (statistical) methods for the
analytics tasks [
These classes of solutions and approaches are not isolated, rather they are expected to rely on each other to provide VREs end users with easy to use, efficient, and effective data processing facilities, e.g., SWMS rely on distributed computing infrastructures to actually execute their constituent tasks.
The proposed framework (IoT-Hub) can be used as a VRE backend, where scientists can simply inject and execute their processing analyses (via VRE fronted) without putting effort in operating the enabling technology. In order to meet this goal, we have leveraged Docker containers, which allows us to have an elastic computational environment based in loosely coupled services, which are immediately portable. Docker handles the packaging and execution of a container so that it works identically across different machines, while exposing the necessary interfaces for networking ports, volumes, and so forth, allowing other users to reconstruct an equivalent computational environment. Therefore, IoT-Hub can be deployed on demand (as-a-service) reducing engineering time and computational cost.
The explosive increase in the number of devices connected
to the IoT and the exponential increase in data consumption
only reflect how the growth of big data perfectly overlaps with
that of IoT [
The number of IoT devices: With growth forecasted in the number of connected “things” and expected to reach billions world-wide, there will be masses of devices which may be a data source, and which may be subject to third party control;
of IoT devices and manufacturers it is reasonable to assume there will be many occasions where IoT devices malfunction in various ways; Update frequency: Though some devices will produce data reports at a low frequency there may be substantial quantities of data streaming from more sophisticated Internet connected things.
Therefore, IoT-Hub has been designed to collect data from a wide range of different IoT devices, geographically distributed that stream data at different frequency ratios, and could yield malfunction behaviors intermittently.
Our proposed solution provides an IoT data-platform, which
support an ecosystem of third party application developers
(e.g.domain scientists and data scientists) to explore data
given the described challenges. IoT-Hub offers a degree of
flexibility making use of Docker and Docker-compose tools
for deploying services on demand in virtualized environments,
such as cloud systems. Previous works have targeted similar
environments, such as [
With recent advances in cloud computing, virtualization,
containerization, continuous integration, and the DevOps
movement, deploying software solutions today is very
different from even just a few years ago. Today’s distributed
applications are built as a set of independently deployable
microservices distributed over clusters of commodity
hardware. The microservice term – also known as the microservice
architecture – refers to a new architectural style that structures
an application as a collection of loosely coupled services,
which implement business capabilities. So, we have built
IoT-Hub following microservice principles [
The question that arises now is, which components should be used in IoT-Hub to develop a high performance platform to efficiently analyze IoT big data. For answering this question, we developed a prototype platform in an elastic cloud environment, and Falcon1, Apache Kafka2, Apache Spark3, Elasticsearch4, Kibana5 and Docker6 have been initially selected (see Table I). This selection could be easily extended in the future for including additional data-frameworks, such as Cassandra, Apache Flink, and Jupyter Notebooks. Reliable, high-performance Python web framework for building large-scale app backends and microservices. It encourages the REST architectural style with minimal external dependencies, while remaining highly effective.
Distributed streaming platform that allows for publishing and subscribing to streams of records (topics) in a faulttolerant way and process streams of records as they occur.
Fast and general engine for large-scale data processing. Among other features, it allows writing streaming jobs the same way as writing batch jobs. It supports Java, Scala and Python.
A distributed, RESTful search and analytics engine for performing and combining many types of searches structured, unstructured, geo or metric.
An open source data visualization plugin for Elasticsearch. It provides visualization capabilities on top of the content indexed on an Elasticsearch cluster. It also supports remote I/O A lightweight, stand-alone, executable package of a piece of software that includes everything needed to run it: code, runtime, system tools, system libraries, settings.
Version 2.0 0.10.2.0 2.2.0 6.2.2 (oss) 6.2.2 (oss) 1.13.1
2https://kafka.apache.org/ 3https://spark.apache.org/ 4https://www.elastic.co/ 5https://www.elastic.co/products/kibana 6https://www.docker.com/ Events / Data Streams
Data Processing/ Data Cleaning
Data Storage
Data Analysis / Visualizations Location 1 SA SB SC Location 2 SA SB SC Location 3 SA SB SC
Falcon data generation
Data Producers (Domain Scientists)
Apache Kafka
Apache Spark
ElasticSearch
Kibana
Cloud Infrastructure data preparation and storage
applications
Data Engenieers
Scientists (Domain Scientists /
Data Scientists)
Fig. 1: IoT-Hub: Data-platform for gathering, quality checking, storing, and visualizing environmental sensors streams.
IoT-Hub integrates several middleware based on the microservices architecture. Apache Kafka provides the mechanism for ingesting real-time data streams and making them available to downstream consumers in a parallel and faulttolerant manner. Data in Apache Kafka is organized into topics that are split into partitions for parallelism. A topic can be viewed as an infinite stream where data is retained for a configurable amount of time. Producers are applications that publish stream of records to one or more topics. In our case,
which are subscribed to one or more topics for parsing their content, all of which is done in near real-time.
Spark Streaming API enables scalable, high-throughput, fault-tolerant stream processing of live data streams. Data can be ingested from many sources (e.g. Apache Kafka, Flume, Twitter), but in this current version of IoT-Hub we limited it to Apache Kafka. Spark Streaming API can be used for processing the ingested data using complex algorithms composed of high-level functions like map, reduce, join, and window. The processed data can be published to yet another Kafka topic for further consumption or it can be stored as results in HDFS, databases, or dashboards. In this work, we have selected Elasticsearch as temporary storage system. One of the reasons for this choice is elasticsearch-hadoop provides native integration between Elasticsearch and Apache Spark, in the form of an RDD (Resilient Distributed Dataset).
Kibana offers interactive visualizations (e.g. histograms, line graphs, pie charts, sunbursts, etc.) and advanced time series analysis on Elasticsearch data, by leveraging the full aggregation capabilities of Elasticsearch.
In order to have a portable, scalable, and elastic dataplatform, we created a Docker cluster for each of the previous middleware and connected them via docker-compose, since it allows us to setup and run multi-container environments. Figure 1 shows a visual description of the IoT-Hub components, and the interactions from different roles (e.g. data producers, data architects, data scientists, domain scientists) that we anticipate. All Dockerfiles and docker-compose file used to generate the IoT-Hub are available freely online in a GitHub repository7 for allowing reproducibility and share our approach among the scientific community.
For our experiments, we have used the NSF-Chameleon cloud8, using a CentOS7 image with 42 CPUS for deploying our hub. Note that the proposed framework could be deployed to any other Cloud system.
IV. CASTE STUDY: ENVIRONMENTAL BASELINE
MONITORING PROGRAMME
To demonstrate the feasibility of IoT-Hub, we have used
the Environmental Baseline Monitoring programme [
Groundwater quality: The sensors installed in the boreholes provide real-time measurements of water-quality parameters: water level, temperature, pH, conductivity, and dissolved gases (O2, CH4, CO2, Rn).
Seismicity: The monitoring of background seismicity has involved installation of a network of seismic stations in the vicinity of the proposed shale gas wells. Real time seismic data are being collected from the array of stations to help characterize current levels of seismic activity. The information captured in near real-time includes: station code, station name, seismic data (for a single channel), and timestamp.
Air composition: The monitoring equipment measures concentrations of ozone (O3), particulate matter (PM1, PM2.5, PM4, and PM10), nitrogen oxides (NO, NO2 and NOx), methane (CH4), non-methane hydrocarbons (NHMCs), hydrogen sulphide (H2S) and carbon dioxide (CO2) as well as capturing meteorological information.
Lancashire - Groundwater sensors- boreholes from 1 to 5
Fig. 2: Groundwater sensors from the areas of Lancashire and Vale of Pickering. In total, we have 9 boreholes (sensors are attached to boreholes) placed at geographically distributed locations.
We have initially focused on Groundwater quality sensor data. However, very little work has to be done in IoT-Hub to enable support to other sensors. This is discussed in Section V. These groundwater sensors are attached to boreholes, which are called emb1, emb2, : : : emb10. Figure2 shows the locations of these boreholes. For simplicity, we have selected emb2, emb3, and emb4 boreholes, but IoT-Hub supports any number of boreholes and sensors.
IoT-Hub collects in simulated real-time the water-quality
parameters described before, from sensors attached to the
selected boreholes (marked as 2, 3, and 4 in Figure 2).
Since we did not have direct access to these sensors, but
access to yearly compressed files instead, monthly datasets
were downloaded locally [
Furthermore, IoT-Hub includes Kibana for exploring the data visually (see Figure 3 as an example). the values are within the predefined range, all the data is stored into the corresponding fields and the normal label is stored in the qc field. Otherwise, the data is stored and annotated with the anomaly label instead. Note that Apache Kafka offers a decoupling and buffering of the input streams. Therefore producer-consumer need not to know about each other. From multiple sources producers can write data to any topic in kafka, and several consumers can be subscribed to the same topic making each of them different processing analysis. Since kafka persists data to disc, consumers can be shut down for performing changes and when they are restarted, they will retrieve all data from the time they were offline.
Listing 1: Elasticsearch index for storing groundwater values. i n d e x : e m b t e s t t y p e : emb f i e l d s : s e n s o r i d > t y p e : t e x t , d e s c r i p t i o n : I d o f t h e s e n s o r d a t e > t y p e : d a t e , d e s c r i p t i o n : Date (UTC) t i m e >t y p e : d a t e , d e s c r i p t i o n : Time (UTC) s e c > t y p e : i n t e g e r , d e s c r i p . : M i c r o s i e m e n s p e r c e n t i m e r ph > t y p e : f l o a t , d e s c r i p t i o n : PH w a t e r l e v e l > t y p e : f l o a t , d e s c r i p t i o n : Water l e v e l aOD w at er t em p > t y p e : f l o a t , d e s c r i p t i o n : Water t e m p e r a t u r e t d g > t y p e : i n t e g e r , d e s c r i p t i o n : T o t a l d i s s o l v e d g a s qc > t y p e : t e x t , d e s c r i p t i o n : Q u a l i t y c o n t r o l ( S m) u / c S S E )900 /Smc700 CE(C400 200
3) Both globalized and localized AD in evolving time series.
The criteria address two main problems in the AD domain. First, the monitoring of the network for the detection of anomalies without a-priori knowledge. Any selected method should be able to perform well both in global and local AD. Detecting localized anomalies can be utilized as a warning system to prevent device failures. As a consequence, this delivers more consistent up-time, improved data quality, and thereby results in a more robust system. Second, the monitoring of global trends that change over time and detect any unusual seasonal patterns. This can help in improving the understanding about the nature of seasonal patterns or changes in research environments.
Due to the aforementioned reasons, the algorithm selected
is Twitter’s Seasonal Hybrid Extreme Studentized Deviate
(SH-ESD) AD algorithm [
The only algorithm parameter that was pre-configured was the maximum anomaly (set to 0.2), m, that controls the algorithm’s upper bound of suspected anomalies. The algorithm ran for each of the different measurements. For brevity, in Figure 4 only the most interesting AD case is shown. In this case, the first (left) and the second run (right) of the algorithm after 15 minutes are shown. The first figure shows normal deviation and no anomalies are detected. In the second run, the algorithm picks up the anomalies that show a small spike before the sensor stops provides readings of value zero. Fig. 3: Kibana screenshot for filtering the normal values, and counting the different values of the water level parameter. 845.2 845 ) cm844.7 / S u (C844.5 E S 844.2 SEC AD
An anomaly detection (AD) algorithm has been implemented to periodically interrogate the data that is automatically collected, processed, and stored by IoT-Hub. The AD algorithm is called every 15 minutes to review all the data transmitted in that period. The IoT-Hub presents new challenges in an AD context due to its continuous data streaming.
Therefore, we have established a three criteria that an algorithm must fulfill in IoT-Hub: 1) Robustness in seasonal changes (i.e., weather patterns).
Fig. 4: IoT-Hub: AD results in emb3, Lancashire sensor for February 2017. First run (left), second run (right). Detected anomalies are shown as turquoise dots.
The performance of the method is promising in a real world scenario for AD. However, as can be seen by the high number of anomalies there is always the caveat of too many false positives, which is something to be avoided in real world scenarios. The parameter selection has also to be evaluated periodically. Careful consideration of algorithms that fulfill the above criteria with a focus on precision rather than recall have to be chosen in order to be able to deliver robust results.
In the future, we plan to create a warning system (e.g., via a VRE fronted) that makes use of IoT-Hub for running the described AD analysis to interpret sensors in the field. This system will send out alerts to domain scientists subscribed to these alerts, as well as to those in charge of deploying and maintaining the sensors in the field.
In this paper, we have presented IoT-Hub, which delivers specialized data-framework to exploit the IoT in a scalable, efficient, and robust manner reducing the engineering time and computational cost. We have demonstrated the feasibility of the proposed solution by using the Environmental
are collected, preprocessed, and stored in real-time using different microservices middleware. To test the capacity of IoT-Hub for running complex data-analytics tasks, we have implemented an anomaly detection algorithm, which queries data from Elasticsearch and detects the anomalies of each of the water-quality parameters. All the middleware that forms IoT-Hub has been containerized, which enables flexible and agile development, and deployment in cloud-based infrastructures.
The current version of IoT-Hub has been pre-configured for working with groundwater sensors. To extend this work to other sensors, it only would require to: (1) create a new Kafka topic to produce and consume new datasets; (2) modify the Spark-Streaming application to consume and check the data from this new topic; and (3) create a new Elasticsearch index.
One of the main uses of IoT-Hub could be to act as a backend for VREs or Scientific Gateways for running dataintensive applications and deploying cloud resources upon request.
As future work, we plan to include more middleware in IoT-Hub, such as Cassandra database (for high performance operations and handling massive datasets), an RDF repository (to store and harvest RDF data), SparQL Endpoint (to query a knowledge base via the SPARQL language), a job submission system (to submit applications to distributed computing infrastructures), and Jupyter Notebook (to offer an interactive computational environment).
Additionally, we also plan to create a warning system that
makes use of IoT-Hub for running the described AD analysis
to interpret sensors in the field, and leverage IoT-Hub
capabilities to process near real-time logs from scientific workflow
executions [
Acknowledgments. This work was carried out when the lead author was with the British Geological Survey. It was funded under the Scottish Informatics and Computer Science Alliance with the Postdoctoral and Early Career Researcher Exchanges fellowship, partially funded by DOE under Contract DESC0012636, “ Panorama - Predictive Modeling and Diagnostic Monitoring of Extreme Science Workflows”, and by “ DARE -Delivering Agile Research Excellence on