Detecting and monitoring radiation level is one of the critical duties for governments and researchers because of the high threats it oppose to humans. It was challenging in the past century to have a centralized radiation monitoring system until the rise of IoT (Internet of Things). Radiation level is measured using wireless sensors that outputs data which are transferred to a back-end server that monitors radiation and alerts when high radiation levels are detected, the server also stores the data for further analysis. The traditional data warehousing systems cannot handle this type of data any more due to (1) data collection speed, (2) rapid data growth, and (3) data diversity. With the rise of Big Data notion, new technologies are developed to handle data with similar characteristics. In this paper, we proposed RaDEn a scalable and fault-tolerant radiation data engineering system that relies on Big Data technologies such as Hadoop, Kafka, Spark, and Hive. The system is responsible of (1) reading data from sensors and other sources, (2) monitor the radiation level in real-time, (3) storing the data, and (4) providing on-demand data retrieval to users. In addition, we have implemented our system and conducted experiments in a real case scenario in collaboration with the department of environmental radiation control at the Lebanese Atomic Energy Commission (LAECCNRS).
Radiation pollution is a critical concern due to its detrimental impact on living beings and environment. There are different types of radiation stemming from various radioactive materials and natural resources [1]. The higher level of these radiations specifically the gamma radiation causes severe damage to human health [2]. Therefore, controlling radiation level is critically important. In order to do so, monitoring radiation sources is an indispensable task. The advent IoT (Internet of Things) specifically, sensors have paved the foundation of building smart ecosystems that enable collecting radiation data, processing, and analyzing radiation level in real-time [3]. Radiation sensors collect and transmit data via communication network such as telecommunication network, Wi-Fi, and Internet to the computational engine for measuring radiation levels. Radiation monitoring sensors records data continuously; in consequence, massive volume data can be generated in a high speed. Conventional data engineering technologies such as data warehouse are not adequate to handle this type of data.
Several data engineering technologies have been
proposed in literature such as [5],[6],[7],[8],[9],[10],[11],
[12],[13],[14],[15],[16] and many others. These solutions
aims engineering radiation pollution data. However, existing
solutions have several limitations that we summarized as
follows: (
In this paper, we have proposed a solution called RaDEn, which is a scalable and fault-tolerant system for radiation data engineering that relies mainly on new data technologies that are able to handle massive volume of data generated in high speed. RaDEn has the ability to read data from different sources, monitor radiation level in real-time, storing data in a scalable repository that provides ondemand data retrieval to users for further analysis.
The remainder of this paper is organized as follows. In Section 2, we briefly introduce our solution called RaDEn. The development of RaDEn will be detailed in Section 3. Section 4 demonstrates RaDEn. We conclude our work in Section 5.
RaDEn is a scalable platform developed for radiation data engineering. It allows fetching massive volume of data from different sources. RaDEn enables user collection different types of data including such as structured databases, data streams and flat files. RaDEn has a radiation data lake which stores data a scalable cluster, process then with advanced techniques and visualize data using the best fit methods.
RaDEn adopted both realtime and batch style philosophies for collecting and processing data. The hybrid enables users to perform both realtime and batch style operations. The data streaming from sensors can be collected by the users in realtime and files can be ingested in storage as batch style. The processing and can be done the same ways. Besides these major operations it somewhat performs pre-processing tasks such data transformation and loading data into scalable data lake. Visualization is real-time meaning that the streams can be visualized with minimal latency and the can be done by files data.
RaDEn is built on cluster computing and parallel computing paradigm. In addition, it adopts the notion of Big Data. The cluster computing guides the solution adopt technologies that foster scalability whereas the parallel computing provides computation models for designing parallel operations using suitable programming model such as functional programming model.
RaDEn is built-on multi-layered architecture. Figure 1 shows the architecture of RaDEn. The figure shows that the RaDEn system can collect data from any number of sources. RaDEn consist of six layers which are explained in the following:
Data Sources: Data sources layer consists of data streams sent from sensors installed in cities and mountains, relational databases where archive data are stored and flat files that can be exported from any old echo system
Data ingestion layer: This layer is responsible of reading data from different sources and delivering them to data processing or data storage layer. This layer must ensure scalability and fault tolerance and must read a huge amount of data from different sources in real-time and batch mode. Data can be stored into Data storage layer directly or it can be sent to Data processing layer to be processed in real-time.
Data Storage: This layer is responsible of storing data. It relies heavily on HDFS, which is a distributed file system that ensures high scalability and fault-tolerance. On the Top of HDFS we have to use warehousing technology to define and configure the metadata of the data stored in HDFS and let the users be able to perform easy data retrieval operations.
Data Processing Layer: This layer is responsible of processing data and notifying the end-user when a high level radiation is detected. The nature of the data sources requires a distributed data processing platform that ensures a high scalability and fault tolerance.
Data visualization: This layer is to visualize the results of data processing and is responsible of bringing the end-user in action by drawing real-time graphs that show radiation level timeline.
Coordination layer: This layer is responsible of making It is a service that runs in background and has the ability to connect with any technology used.
We have built a 4-node Hadoop cluster. To build this system we have first deployed four virtual machines where we have installed Ubuntu1 16.04 LTS as operating system and Hadoop 3.1.0 for data storage. We have configured the first virtual machine to act as the master node (name node) and the others to act as slaves (only stores data).
On the master node, we have also installed the data ingestion, processing and visualization tools. As a programming language, We have used python2 because it is more powerful than other languages in data science domain due to the presence of many specialized libraries.
For data ingestion, we have installed Apache Kafka 3 ,
Apache Flume4, and Apache Sqoop5. We used Apache Kafka
as the main data ingestion tools, because: (
We have configured Apache Flume agents to read from Kafka topics consumers. The Flume agent is responsible of storing data from Kafka to HDFS. In addition, we used Apache Sqoop to read data from relational databases and store it into HDFS. Reading archive data from relational databases is not one of the main goals of the system, but it is an added value to allow user to migrate their old data from traditional warehousing system. In addition, we have used Apache Hive7 to define the metadata of the file stored in HDFS to make the data retrieval process more easily using SQL-Like languages such as HiveQL and Spark-SQL. 1 https://ubuntu.com 2 https://python.org 3 https://kafka.apache.org 4 http://fume.apache.org 5 http://sqoop.apache.org 6 https://zookeeper.apache.org 7 https://hive.apache.org
For Data processing, our solution relies heavily on
Apache Spark 8 for these main reasons: (
We have created a single node Apache Spark cluster as the first phase of deployment and we can add other slave nodes when it is required. In addition, we have use pandas9 python library because it provides many classes and functions that makes handling data easier.
For Data visualization, we used matplotlib 10 library which is a Python 2D plotting library which produces publication quality figures in a variety of hardcopy formats and interactive environments across platforms. It also allows drawing real-time graphs.
The data processing and visualization is done by a python
script, where we have implemented the radiation alert
system (designed based on the Lebanese Atomic Energy
Commission requirements) that work as the following: (
In this section, we demonstrate RaDEn. For our demonstration we used a radiation dataset supplied by the department of environmental radiation control at the Lebanese Atomic Energy Commission (LAEC-CNRS).
The dataset was provided by the LAEC-CNRS, is in form of flat files, because accessing the sensors or the web server (relational database) was not made due to confidentiality issues. The dataset contains the data collected from 2015-08-01 to 2016-08-01 from a testing sensor that was installed in Beirut. It contains information related to radiation such as: radiation level, temperature, rain level, Sensor battery power, data collection time and external battery power.
Before starting the process, the user must start the
following services: (
9 https://pandas.pydata.org
10 https://matplotlib.org
(Figure 4), (
To simulate data ingestion we have create a directory where we must copy all flat files, and we created a terminal script that creates a listener on this folder. When any file is inserted, the script loops over the lines and send them one by one to the Apache Kafka producer. Once the data is sent to the Kafka producer, the Apache Flume agent sent it directly to HDFS to a specific directory and the data is replicated on the three Hadoop data nodes. Figure 6 shows the data ingested into HDFS via Hadoop web interface. Also, figure 7 shows that the file stored in HDFS is replicated on 3 data nodes.
At the same time, the python script read the data from Kafka consumer using the PySpark library, the alarm script is applied, and the radiation level is visualized on a real-time graph using matplotlib library. In figure 8, we have shown sequential snapshots of the real-time graph that was visualized during the experiments and it shows the radiation level changes in function of date and time.
In addition, when an alert is raised it is shown in a message box where the alarm level is written in the title and the description is written in the body. In figure 9, we have showed the level 1 alert message box.
Based on the Flat files structure and for data retrieval purposes, we created an External Table using Apache Hive on the Directory location in HDFS. Then we created a view from this table to remove messy data such as duplicates rows and rows where dose rate is null.
After created external table on HDFS directory, we can search among the data imported to HDFS using Spark-SQL or HiveQL console. We have to write a query based on our requirements. As example, we need to retrieve all the data where the radiation level is higher than 50. First, we need to run the Spark SQL console using the spark-sql command or hive command to start Hive console, and we can use the following query:
SELECT * FROM vw_radiation WHERE dose_rate > 50; In this example, it shows that the number of rows returned is 10459 rows in 13.99 seconds as illustrated in Figure 10.
In this paper, we designed a solution called RaDEn that is able to handle a massive scale of data in real-time and batch style. It allow user to process the radiation data coming from different sources, predict any possible radiation problems and visualize the data in real-time. In addition, it gives the user the ability to query and retrieve the data using a simple SQL-Like language. In addition, we explained how we implemented this solution and we showed a real case scenario.
We tried to cover all the challenges we identified at the
beginning of our work but unfortunately, the
implementation we have made have some limitations due to
the following issues: (
A list of works is lined up to be done in future. More powerful tools such as bokeh11 and Kibana12 can be used to increase performance and more options to the end-user and are able to draw a huge number of real-time graphs at the same time. In addition, adding some user interface will make this this solution more powerful, because the current implementation is not user friendly; due to the lack of user interface, also it requires a good knowledge in SQL to be able to retrieve data from HDFS. In addition, the solution should be extended enhance the user the ability to visualize the results of queries on different types of graphs.
Furthermore, we stored data as flat files in HDFS, to improve the performance; we could create an automate job that run periodically and move new data to another HDFS location and convert it to Optimized Row Columnar (ORC) files which gives faster results. Also in future extension, distributed search engines such as Elasticsearch and Solr13 can be used for data retrieval process.