By 2025, it is estimated that there will be over 75 billion devices connected to the Internet, and it is expected that data volumes will continue to grow exponentially. As such, there will be an increasing need for e ective data management of data generated by sensors in the Internet of Things (IoT). In this paper, we propose a lightweight, generic data processing framework that may be easily customized for a diverse range of data-handling IoT applications. The sensor types required by an application are speci ed as metadata. Sensed data is persistently stored in a horizontally partitioned relational database across IoT nodes. Subsequently, it may be retrieved by posing SQL queries over the nodes, which may be historical or real-time. Alternatively, the sensed data may also be retrieved as a JSON document over a RESTful interface, thus potentially enabling easier integration with cloud and mobile platforms. We show an instantiation of the framework for applications in the climate monitoring and security domains.
The number of devices connected to the Internet is expected to grow
exponentially over the next few years. Indeed, according to some estimates, by the year
2025, there will be over 75 billion "things" connected to the Internet [
This is an important problem to address given the huge economic impact that
the IoT is expected to have in coming years [
Previous solutions to provide generic processing frameworks for IoT tend to
be limited. For example, IFTTT1 has limited expressibility with regards to the
data processing tasks that may be speci ed. Other approaches that have greater
expressibility are limited to the Wireless Sensor Network domain. For example,
the SNEE query processor [
This paper is structured as follows. Section 2 presents related work. A hardware design for a generic IoT node is described in Section 3, and the corresponding software architecture in Section 4. We present the middleware that we propose in Section 5. Section 6 showcases our data processing framework with an example deployment and application. Finally, Section 7 concludes. 2
In the area of technological tools such as technologies that are responsible for
making everyday tasks easier such as Paw Scout [
Another application designed for the development of the tasks of daily life
is Amazon Go [
In the eld of sensors, the company Libelium [
In the Internet of Things there are diverse aspects that may be considered; in particular, this work does not focus on security or scalability, since the main development of the work, will be to make available a general cluster that will be responsible for accepting any type of sensor or actuator and handle them with the bene ts of using a database-based system. As proof of concept, the decision was made to use a house and three monitoring points to be able to correctly populate the information and demonstrate that the entire system that is being planned to be assembled will work correctly.
According to Sethi et al. [
Taking into account some of the data ow structures surveyed by Razzaque et
al. [
Jensen et al. [
In the stage for hardware architecture planning, some applications are already
taken into account in the market and it is a matter of covering a good number of
these applications with a general design that could solve the need for the other
applications with a few minor modi cations [
In the most general design of the application we want to cover the broader aspects, starting with the Figure (Right side Fig. 1) by the simplest components that are going to be responsible for doing any type of reading, where the data will be stored and the IoT area can be ordered, which is going to take care of the fact that everything is interconnected, together with the neuralgic area of the design that is in charge of processing all the data and providing the user with something readable and with added value (Left side Fig. 1).
In the area of IoT you want to have complete access to the components by
means of the platform that best allows it like Raspberry Pi [
Regarding the node, the most general view is a platform with access to the internet, in which both the reading algorithms of the sensors and the logic can be executed in order to make decisions based on the data collected by the sensors and make the actuators have their e ect. Along with the ability to generate a database to store each of the sensing data correctly (see Fig. 2).
In the data center or the Warehouse [
To carry out the implementation of the design, it was decided to use the RaspBerry Pi platform in the nodes, due to its robustness and ease of work since it is based on Linux and its complementation with digital and analog readings through the connection pins with the which is already equipped, the set of sensors that were used to see di erent types of reading were binary data sensors such as the LM393 light sensor and the PIR HC-SR501 infrared motion sensor, for sensitive data this sensor is Humidity and temperature DHT11. The idea of representing all the connections of the physical layer exposed in Fig. 3 is to be able to explain the functionality that has been implemented in the operation of the middleware since based on these physical connections it they have to populate the tables of the local database of each of the nodes corresponding to the metadata.
At the same time there is an application server in each of the nodes which through the LAN connection is able to publish the data to make them readable, make them available to the central data, which is responsible for centralizing the information and make it accessible to the user can enter to consume it from any platform with internet access.
To make the connection of the nodes it is understood why the Raspberry Pi are so viable because they can receive several lines of information and at the same time to send information to put the actuators into operation. For the particular case of simulating an intelligent building, the connections were made as shown in Fig. 3, which connects with the applications through metadata tables in the database.
Fig. 3 presents the hardware design. The sensors connected for the prototype were (i) a LM393 light level sensor connected to GPIO.IN4 (ii) DHT11 humidity and temperature sensor connected to GPIO.IN17 and (iii) a PIR HC-SR501 infrared movement sensor connected to GPIO.IN23. 4
The construction of software components de nes its main structure in each IoT node, the synchronized data reading sequences of the connected hardware are based on interactions controlled by time cycles using metadata parameters, the information collected is articulated concerning the node and then stored in a generic data repository.
An information web service component is available in each node to expose functional data. Each component articulates two parts: one, an architecture application server based on HTTP 2.0 and another, a software service built with Rest JSON service speci cations and based on stored central data.
The data service is available for consumption, both generically by various types of software, since it uses software standard based on REST API services, and speci cally, in this case, through the implementation of a solution called Warehouse Node, to de ne the data centralization of each connected node in the architecture.
When developing a Warehouse architecture speci cation, as shown in Fig. 4 and Fig. 5, the data is centralized in a software design that consolidates the information for interpretation and analysis. So that actions can be taken on established segments of utility such as security or climate since it is part of the importance of the sensors established from the beginning of the architecture.
In order to elaborate the previous described architecture, in the rst instance the construction of a project in Python 3.5 is done, it starts by reading metadata data and it is articulated with the data capture of the connected sensors depending on the reading cycle time; then, each data reading is structured, including the sensing time, for storage in SQLite 3.0 database.
The operating system is prepared with a virtual Python environment to control the necessary components, including those of Web and Rest JSON architectures, the application server used is Flask, Twilio is used for the structure of Rest JSON entities. Then, the software code prepares the information to return as a key entity value the data stored in SQLite (see Fig. 6).
The REST structures enabled in each node are used by a Spring Core specication, and Spring Boot, to centralize the data; then, through speci c domain views, data is established as climate and Security. Thus, once consolidated data can be obtained, the information is articulated with a software component called Morris to graph the information. The source code used for the implementation of the prototype is freely downloadable from our Github respository2. 5
The implementation of Middleware for the proposed architecture consists of four hierarchically ordered elements: the rst, a generic software segment in charge of obtaining raw data modeled as a key-value; the second, a data modeler that structures data for repository storage called data warehouse; the third, domainspeci c views that give value to the consolidated information and, nally, the obtaining of data for analysis and decision making by means of graphs.
To obtain the data of each node, a property or con guration le is established rst to articulate the IP addresses of each node included in the architecture; then, an information reading of the nodes is made through the Rest service, available with the data that is captured and, nally, articulated with abstract software entities to structure and, again, stored in a central information repository.
Within the data modeling, the value of the information obtained is determined and structured with additional repository data, where the value of the information can be determined and the additional attributes that include node identi er and sensor locations.
In the speci c domain views, the data whose main value represents the usability of the sensors, the contrast with the location environment and the quality values to compare with generic values such as temperature and humidity are determined; the latter, depart from a central location of the node and can be compared with the temperature and humidity values of a city.
At the moment of graphing the information, it is consolidated by hours for the control of the data and the established reading dates, including the relevance 2 https://github.com/diegofnunezs/CoEduUtadeoIsTesisClusterIot of the spatial locations of the nodes; with this, a de nition for the presentation of data determined in dot, bar or pie charts is achieved.
Technically the warehousing node is established in the following way: rst, construction of code and parameterization les for reading data in the Rest speci cation; second, to de ne structures like generic software entities that allow the consolidation of information in a data container called warehouse; third, consolidated views of speci c domain that include centralized information, grouped by hour and including the location of each node and, nally, the preparation of SQL queries that group and transform the information to be visualized with graphing processes.
The warehouse database has main de nitions called metadata, these are basic con gurations for the generic use of the proposed architecture. Within the proposed con gurations, there are data read values for each node, sensors that each node can contain, and additional, connectivity con gurations of hardware components where the most important is the support of heterogeneous sensors. 6
During the general planning of an intelligent building which is able to maintain some variables of its environment controlled and monitored, in addition to keeping them centralized in the data warehouse, together with the nodes distributed and connected between them to the warehouse through the LAN connection, to test each of the components described above, a test area was established for the case study.
The case study is established to monitor di erent locations of a house based on Internet technology of things; thus, we de ne three main locations in a household: the rst, a room with movement, light, temperature and humidity sensors; the second, kitchen area with temperature and humidity sensors and, nally, a node located in the studio with all the architecture sensors. All the nodes were synchronized for reading in equal times and stored in local databases with generic data structures, which will be concentrated in a single database.
For a better organization of the sensors, it was decided to design an acrylic box with the Autodesk Inventor tool, which can be seen in the Fig. 8, which already implemented can be seen in the same image, with the design it was possible to adjust each one of the sensors in the upper part of the box to be able to make the measurements which require a wide and clear area so as not to interrupt with the correct readings of the sensors.
In Fig. 7 we can see the node that was in charge of the measurements from the study in the proof of concept.
As you can see in the Fig. 8 on the right is the 3D model of the design made in Autodesk Inventor and on the right is the execution of it made in acrylic, which was very useful for its transport and subsequent commissioning.
In order to make a correct test of the sensors in the house a week of measurements was taken, during this time the nodes with 4 sensors made a writing in the database of 4036 rows and in the node of the kitchen that only had 2 of the sensors, had at the end of the 2018 week records. With all the records and using all of them there are a few graphs so that the behavior of the sensors in the test area can be observed, the graphs are divided in temperature of 1 and 7 days, humidity 1 and 7 days and a graph that represents the number of times the sensor had a positive or negative response. In this way, with the graphs we can see, in detail, the variation of the data such as humidity and temperature; in addition, there is the Table ??, which is a list of information queries where criteria can be applied to link the various types of data. In them, you can group and delimit both with date attributes such as location and time and by speci c domain values: movement, light, humidity, and temperature.
In Fig. 9a the temperatures corresponding to the day May 26, 2019, can be observed where it is observed that in the whole house an average temperature of 21 Celsius was obtained, except for the living room around 13:00 and 15:00 of the day, when a temperature rises to 34 Celsius. Fig. 9b shows the temperatures for the week of May 25 from 9:00 until 1st of June at 9:00 in 2019 where it is observed that a temperature was obtained throughout the house average of 21 degrees during the rst ve days, and towards the end of the week a considerable increase of the average was obtained up to 24 degrees in the living room and the study, while in the kitchen the average temperature remained of 21 degrees all week. In Fig. 9c is noted as moisture was also quite stable except for measurements made between 13:00 and 15:00 corresponding to the high temperature at the time recorded. The humidity of the week recorded and observed in the Fig. 9d is quite stable throughout the week in the kitchen being around 60% humidity, as for the living room a fall happens every day around of the noon that corresponds to the temperature rise of the same and in the study a slight decrease is observed throughout the week., it is also observed that every day around midday the living room always had a strange increase in unsubstantiated temperature (a) One-day of test observed from Temperature sensor. (b) Seven-day test observed from the Temperature sensor.
(c) One-day test observed from the Humidity sensor. (d) Seven-day test observed from the Humidity sensor.
Fig. 9: Data collected during deployment trials.
In the Fig. 10 which represents the number of times that movement was recorded during the week in the living room, which is a social area in the house where it is seen that there were more movements than stillness. 7
In this paper, we have proposed a generic data management framework for the Internet of Things. We describe a generic hardware architecture for an IoT node and specify a concrete example of this implementation using the Raspberry Pi, a commonly used platform for prototyping IoT applications. We then describe a generic software architecture, in which queries can be posed using the SQLite database engine, via a RESTful interface. We envisage that such a proposal may be used for diverse IoT applications, and demonstrate an example three-node deployment in a smart home setting as a proof of concept.
Future work could usefully investigate the incorporation of semi-structured and unstructured data into the data management framework, as well as streaming multimedia data.