We present HARVEST, a complete end-to-end solution for smart agriculture monitoring. HARVEST is built using open-source systems specifically designed for big data applications to eficiently ingest and process real-world measurements collected in real-time from sensors located in diferent areas of Greece. The results from the analysis benefit not only the farmers but also additional roles in smart agriculture, such as data analysts and sensor infrastructure operators. HARVEST encapsulates advanced analytics to fill missing measurements, detect malfunctioning sensors and produce meaningful insights to support decision-making thus departing from common data warehousing deployments.
applications that move data from the sensors to the cloud enable more advanced analytics and benefit from efiThe agricultural industry nowadays aims transition to cient data storage [3]. In several cases, even though smart agriculture through the use of the Internet of the data are collected in the cloud, they are processed Things (IoT) and big data technologies. Smart agriculture separately for each user, which creates limitations regardapplications boost operational eficiency and productivity ing the employed analytics methods and is dificult to [1]. As shown in recent surveys, e.g., [1], smart agricul- compensate for the high costs to set up and manage a ture applications are typically divided into 4 categories, large-scale infrastructure. Therefore, taking advantage namely monitoring, tracking and tracing, smart precision of all the available data to more eficiently support the farming, and greenhouse production. We focus on the end-users’ decision making is the most attractive option. ifrst category that has received the highest attention.Its Data from custom sensors are pushed to the cloud, where core rationale is that important factors afecting farming they are processed collectively using ML and AI to learn and production, such as soil temperature and air humid- and understand the behavior of diferent crops in diferity, need to be measured by IoT sensors and transferred ent parts of the world. However, HARVEST, which is into a (typically) cloud-based infrastructure. Then, using our proposal, goes one step further, as it enables difermachine learning (ML) and other big data analytics tech- ent roles in smart agriculture, like farmers, data analysts, niques, useful information and insights are mined. The and network operators, to interact and benefit from this results of the analysis provide recommendations to the large-scale monitoring application. farmers about their farming and irrigation plans, which In this work, we introduce HARVEST, a complete soresult in optimized production while at the same time lution for smart agriculture monitoring that efectively minimizing labour costs. supports big data analytics. It is complete in the sense
Some of the monitoring applications do not use a cen- that it is an end-to-end system and serves the needs of all tral storage. They rather collect, process and visualize main stakeholders: farmers, sensor infrastructure operadata to a device that is connected with the sensors in tors and data analysts. In our scenario, we collect data the field. In these applications the processing takes place from the LoRa IoT Network of the American Farm School either on an edge device (like Raspberry Pi) or on a com- (AFS) of Thessaloniki, Greece, which produces over 70K modity machine, e.g., [2], which poses limitations due measurements per day and covers more than 1/4 of the toto lack of available compute resources. On the contrary, tal area of Greece. HARVEST can also detect outliers, fill missing values and extract useful insights, which is typiPErxopcleoerdaitniogns aofndthAen6athlyItnictsercnoa-ltoiocnaateldWwoirtkhshEoDpBoTn/ICBiDgTD2a0t2a3VJiosiunatl cally not included in data warehousing deployments. To Conference (March 28-31, 2023), Ioannina, GR achieve its objectives, HARVEST aims to fulfil the follow$ mavroudo@csd.auth.gr (I. Mavroudopoulos); ing functional requirements below: R1: Eficiently ingest tatoliop@csd.auth.gr (T. Toliopoulos); gounaria@csd.auth.gr and process big data, ensuring low latency and scalability. (A. Gounaris); kynigopo@csd.auth.gr (G. Kynigopoulos); R2: Inform infrastructure operators for anomalies and (aIn.dKraelafadsis).dws@gmail.com (A. Andreadis); ikalfa@afs.edu.gr malfunctioning sensors. R3: Provide easily understand© 2023 Copyright for this paper by its authors. Use permitted under Creative able analytical results and visualizations for the farmers CPWrEooUrckReshdoinpgs IhStpN:/c1e6u1r3-w-0s.o7r3g CCoEmmUoRns LWiceonsrekAstthribouptionP4r.0oIncteerenadtiionnagl s(CC(CBYE4U.0)R.-WS.org) to assist them in the day-to-day activities. R4: Equip data analysts with built-in functionalities for filling missing values and automated extraction of insights to facilitate data exploration.
Farmer Data Imputation Insights Visualization Analytics
Runtime Data Processing
Data Scientist
Of line Analytics The data used are collected in real time from the Ameri- Outlier can Farm School (AFS) of Thessaloniki’s LoRa IoT Net- Message Bus Detection work, which operates in many areas of Greece. The most Data Warehouse itmhepgoarttaenwtacyosm.Npoondeesntasreofsptheecinficeatllwyodreksiagrneetdhetonoopdeersaatned OpLeorRaator Storage CNoomtifpicoantieonnt Data Colector in open environments, like fields and farms, and each Notifications API node contains a variety of sensors. These sensors provide Gateway real-time measurements of environmental variables, such NLeotwRoark as humidity and temperature. In addition to the nodes, the AFS network also includes meteorological stations, Physical Layer Sensors which can capture additional measurements like solar radiation. At the time of the study, the AFS LoRa network Figure 1: System overview had 254 nodes and 50 meteorological stations, with an average number of measurements per day of 70K. measurement, such as the district name in which the sen
Gateways operate as routers between the nodes and sor operates, in order to provide more meaningful spatial the cloud, where the data are stored. The nodes uti- indexing later. The final measurements are ingested to lize LoRaWAN[4], a low-power network, to send pack- the Message Bus. ages to the gateways, while the gateways employ a high- Message Bus. This component enables information bandwidth network, i.e. WiFi, to transfer these packages lfow between the various components in HARVEST. Each to the cloud. LoRaWAN allows a gateway to receive pack- component of the Runtime Data Processing component ages from nodes that are installed in up to a 25-kilometer pulls data from the Message Bus, processes them, and distance. Therefore, with only 62 gateways, the network then writes them back to a diferent channel. In our coverage is estimated to be 36.6 million acres, which is implementation, we use Apache Kafka, an open-source more than 27% of the total area of Greece. distributed event streaming platform, widely used in the industry sector for high-performance data pipelines, 2.1. System Overview streaming analytics, and data integration (R1). Outlier Detection. Some of the data ingested by Fig. 1 depicts a high-level overview of the proposed infras- the Data Collector might have errors or inconsistencies, tructure, starting from the sensors and ending with the which, if left unattended, would have a negative impact analytics. In this section, we briefly discuss the various on any produced result. The Outlier Detection compocomponents of our solution. nent is implemented in Apache Flink and is responsible
LoRa Network sensors. The network consists of 13 for eliminating anomalies before the permanent stordiferent types of nodes, each with a unique collection age of the data in the Data Warehouse. We have impleof sensors. The most common type of node is the one mented two diferent approaches to anomaly detection. installed in the fields, which accounts for about 68% of The first one is a simple rule-based technique that detects the total nodes. The sensors embedded in this type of anomalous behaviors, like out of bounds values or inacnode measure the air pressure, humidity, and tempera- tive nodes, with the rules provided by domain-experts. ture, as well as the dew point, the soil temperature, and The other method uses a simple continuous outlier detecthe volumetric water content. All the measurements are tion technique to detect measurements that significantly transmitted from the sensors to the cloud through the deviate from previous measurements of the same node gateways in order to be stored. Then, they become avail- or from measurements made by similar sensors in other able through an API. nodes. This unsupervised technique allows HARVEST to
Data Collector. This module is implemented using a eliminate more complex anomalies that would have been simple Java software and is responsible for periodically missed by the naive rule-based method. All the detected retrieving new measurements from the AFS network API. anomalies are removed from the main stream and written Since the measurements come from diferent node types, into a diferent topic of the Message Bus in order to be a pre-processing step is required to standardize their handled by the Notification Component. structure. Moreover, additional fields are added to each Data Warehouse. After removing the outlying measurements, the data are forwarded to the Data Warehouse a very common phenomenon. Additionally, hardware (DW) via the Message Bus. DW is responsible for efi- damage and sensor malfunction, which can be caused ciently storing the multi-dimensional measurements and by extreme weather conditions, also produce large missenabling Online Analytical Processing (OLAP) opera- ing blocks of data. Therefore, even under continuous tions. Additionally, the stored data are then utilized by monitoring, the sensors often remain inactive for sevthe methods in the Analytics layer. In our implementa- eral hours, or even for days, until the engineer fixes tion, we have employed Apache Druid, an open-source the detected damage. In HARVEST, we employ data imreal-time database that automatically integrates with putation techniques to fill in the missing values. This Apache Kafka and provides fast and consistent queries at component reads and writes directly to the DW and is high concurrency (R1). For each measurement, except for implemented using Python. We have implemented and the value and the timestamp, we maintain the sensor id, evaluated the performance of a number of diferent methnode id, node type, minimum and maximum values that ods, e.g., methods that utilize the temporal dependencies are acceptable for this particular sensor type, and spatial and correlations between diferent sensors in the same information (longitude, latitude, municipality, prefecture, node to better approximate the missing values and methand district). This information allows aggregations at ods that take advantage of the structure of the network, various levels. utilizing the correlations between the same variables in
Notification Component. It is crucial for the AFS multiple nodes to enhance the accuracy of their predicLoRa operators to get notifications of any anomalous be- tions. Deep learning techniques are more accurate in havior. This allows them to detect and fix malfunctioning predicting the missing measurements when operating nodes and sensors or detect possible natural disasters, under the multi-node setting. However, since data impulike fires or floods. The Notification Component’s re- tation is a plug-in to the main system, in the future, we sponsibility is to inform the network operators of any can easily test additional techniques and re-evaluate our anomalies discovered by the Outlier Detection compo- choice. nent (R2). This component is implemented in Python. Up to this point, we have discussed the data ingestion The incidents are saved in a relational database to avoid along with preprocessing steps, like outlier detection and sending alerts for the same incident more than once, and data imputation, in order to enhance the quality of the notifications (in the form of emails and Viber messages) data. However, another important role of HARVEST is to are sent periodically. extract knowledge from the data to support the end-users
Analytics. A number of diferent analytics are applied in their daily activities. OLAP, enabled by the DW, can be on top of the ingested data. Also, a visualization tool that used to obtain useful information about the stored data, is easily configurable to match each user’s unique de- but it takes a lot of efort to manually pose queries and inmands has been employed (R3). In our implementation, terpret the results, particularly for non-expert users like we have used Grafana, an open source tool for creating farmers. Therefore, we need an automated way to obtain custom dashboards and that is easily integrated with both insights, where insights are interesting observations that DW and the database that stores the outliers. Next, we derive from aggregated data. Additionally, there should have the Ofline Analytics tool implemented in Python, be a way to rank the various insights with an appropriwhich extracts statistics (e.g., active nodes and average ate score function so that only the most interesting ones measurements per day) about the LoRa network in or- are kept. The extracted information will provide inforder to support future decision-making concerning the mative summaries of the data to non-expert users while infrastructure (R2,R4). Finally, on top of the DW, we also aiding data analysts in data exploration. To satisfy have implemented two advanced analytics methods, top- this requirement, we have integrated and extended the k insights extraction and data imputation. We discuss method described in [5] into our system. This method these methods in the following sections. Note that the extends OLAP tools’ simple aggregations by conducting structure of the proposed infrastructure enables the im- analysis operations on them (e.g., rank, diference), replementation of various other analytical methods, like sulting in the extraction of valuable information. This the prediction of a sensor failure, without interfering component requires the following input parameters. Parwith the rest of the system. In other words, the system is ticipating attributes: The user can define a subset of the modular and extensible by design. available attributes (e.g., the timestamp of the measurements and the region of the sensor that produced them) 2.2. Data Imputation and Insights in order to generate more meaningful insights. Measure column: The column on which aggregations will be conExtraction ducted on (e.g., temperature values). k: The number of Data transmitted over the network may contain missing top insights that will be extracted As the value of the values for various reasons. Lost packages due to com- parameter increases, the process time to calculate the munication errors between the node and the gateway is top-k insights increases; typically, a value between 50 The demonstration will present how the data are extracted from the network’s API in real-time, processed by the various components before stored in the data warehouse. Additionally, we will show how the diferent actors can benefit from HARVEST. End-users, e.g., farmers, can utilize the custom graphs in the Visualization comFigure 2: Left: average air pressure diference from the pre- ponent to monitor the crops in real-time and make better vious hour (point insight). Right: average temperature difer- day-to-day decisions, such as deciding when to water the ence through-out the day (shape insight). plants according to the measured soil humidity. Additionally, they can use the Insights UI to obtain interesting and 100 yields good results. Aggregator: The aggregation insights about the collected data without spending any function, which could be either SUM, COUNT or MEAN. time configuring the system and/or submitting queries . Extractor: Currently, in our system we support 4 types Data Analysts can also benefit from the proposed frameof extractor: (1) Previous Diference , which subtracts the work in multiple ways. First, insights can be extracted current value from the previous one (this applies to at- from both the Ofline analytics and Insights components, tributes that can be sorted, e.g. years), (2) Rank, which in order to provide some initial information about the simply ranks the values, (3) Percentage, which finds the data and the network structure. Next, using the OLAP percentage of each attribute and finally (4) Average Dif- tool provided by the DW, the analyst can eficiently subference, which subtracts the average aggregated value mit SQL queries over the multi-dimensional data. Finally, from the initial one. -depth: The number of extractors the HARVEST’s structure enables the implementation that will be used. A value of 1 indicates that only the and evaluation of additional components, like CEP-based aggregator will be used. The recommended value is 2 smart agriculture and predicting the remaining useful (aggregator + 1 extractor) as more extractors will raise lifetime of a sensor. We will demonstrate how such exthe complexity exponentially without providing better tensions can be realized. Operators of the LoRa Infrasresults. Insight types: HARVEST supports 5 insight types: tructure can continuously monitor the LoRa network in (1) Point insights mark an outstanding outlier, which is the Visualization component while also receiving realfar greater from the other set, (2) Shape insights show a time notifications for possible malfunctions. trend that is increasing or decreasing rapidly, (3) Attribution that shows outstanding percentage of an outlier, (4) Two Points, which reflects the diference of two points References from the others and finally (5) Last Point which refers to an outstanding outlier, which is very small compared to [1] V. K. Quy, N. V. Hau, D. V. Anh, N. M. Quy, N. T. Ban, the set [6]. Two examples are given in Fig. 2. S. Lanza, G. Randazzo, A. Muzirafuti, Iot-enabled
We built a user-friendly interface so that any actor can smart agriculture: Architecture, applications, and log in and quickly run a top-k insights query.1 Insights, challenges, Applied Sciences 12 (2022) 3396. similar to the Data Imputation component, is a plug- [2] G. Lee, M. Kim, K. Koroki, A. Ishimoto, S. H. in to the core system, i.e., it connects directly with the Sakamoto, S. Ieiri, Wireless ic tag based monitoring DW and does not interfere with any other component, system for individual pigs in pig farm, in: 1st Global making it easy to extend or modify. This component is Conf. on Life Sciences and Technologies (LifeTech), implemented in Javascript and is publicly available.2 IEEE, 2019, pp. 168–170. [3] W. Zou, W. Jing, G. Chen, Y. Lu, H. Song, A survey of big data analytics for smart forestry, IEEE Access 3. Deployment and Demonstration 7 (2019) 46621–46636. [4] S. Ali, T. Glass, B. Parr, J. Potgieter, F. Alam, Low cost The system has been running for over 20 months on a sensor with iot lorawan connectivity and machine Linux server with 16GB RAM, a 2.3GHz CPU with 8 cores, learning-based calibration for air pollution moniand 200GB of hard disk space. During that period, a to- toring, IEEE Transactions on Instrumentation and tal of 22 million measurements have been successfully Measurement 70 (2020) 1–11. ingested and stored. All the services were containerized [5] B. Tang, S. Han, M. L. Yiu, R. Ding, D. Zhang, Exto minimize the installation process and increase inter- tracting top-k insights from multi-dimensional data, operability. That is, if the AFS LoRa network expands in in: SIGMOD, 2017, p. 1509–1524. the future, the proposed system can be easily transferred [6] R. Ding, S. Han, Y. Xu, H. Zhang, D. Zhang, Quickto a new environment that can match the new demands. insights: Quick and automatic discovery of insights from multi-dimensional data, in: SIGMOD, 2019, p. 317–332.