The main goal of the research work is to solve a wide range of data problems by offering batch, iterative, and interactive computations in a unified architecture. The dissertation proposes an integrated architecture to manage a large amount of massively distributed data including spatial data. The implementation architecture has three major components: data preparation, data analytics, and data visualization. As a part of the implementation, a novel big spatial data analytics framework is developed to load, store, process, and query spatial and non-spatial data at scale. As proof of concept, spatial analytics applications are developed using agricultural reallife datasets.
In the mobile and Internet era, massive scale data is generated from
disparate sources with spatial components. Today’s users demand
high speed, scalable, sophisticated, economic, and accessible
solutions to perform relevant analytics on complex and distributed
data including spatial data. The conventional systems for data
management are becoming less capable to scale out extensively to
meet the current users’ demand due to limited computational power
and storage. The modern technologies like Big Data and Big Data
Analytics (BDA) have a huge potential to handle massive scale data
with high scalability and low latency. Though the modern big data
management tools such as Not only SQL (NoSQL) databases,
Hadoop [1], and Spark [
The research reviewed existing databases, frameworks, and architectures for spatial data management.
NoSQL databases such as Cassandra [
The Big data computational frameworks such as Spark and Hadoop
don’t offer native support for spatial data. A number of extended
systems have made important contributions to extend the
functionality of Hadoop/Spark engine for spatial data management.
These extension systems include parallel DB systems such as
Parallel Secondo, MapReduce systems such as ESRI Tools for
Hadoop [
Generally, big data architectures are designed and developed to
achieve a specific goal. Many big data architectures such as
Lambda [
There are many big spatial data frameworks have been developed on top of big data stack composed of Spark and Cassandra. The Cassandra-Solr-Spark framework has been developed by Datastax to enable spatial query processing on top of the big data stack. The framework provides SQL like query interface to perform spatial operations. However, it does not support join operations. It has also not been evaluated based on the performance metric. P. Shah et al. [25, 26] have developed a big data analytics framework including geospatial data. The spatial analytics applications in the agriculture domain have been developed using third-party GeoSpark libraries. The major drawback of the framework is data duplication. GeoSpark is a spatial extension for spark which can only access data available in HDFS or local disk.
The big data analytics architecture [26] is built and implemented on big data open source technologies for the enrichment of massive scale data including spatial data. The architecture is designed to provide scalable, flexible, extendible, and cost-effective solutions with available infrastructures and tools for agriculture. Architecture implementation has three components: data preparation, big spatial analytics, and data visualization. There are four types of user interaction with the architecture: 1) System developer, 2) Data scientist, 3) Domain expert, and 4) End users. The proposed architecture is shown in figure 1.
The main purpose of a big spatial data analytics framework [
The data preparation framework is implemented to fetch consistent and clean data from disparate sources and store into a persistent database. It provides two layers of data abstraction. First, it hides all physical data sources from the data repository. Second, it further unifies the data available in a data repository using various complex tools and techniques such as data fusion algorithms, schema mapping tools, and record linkage algorithms. The implementation architecture for data preparation framework is shown in figure 2. A NoSQL i.e. Cassandra based spatial data storage framework is built and implemented. The framework offers distributed and scalable APIs for spatial operations such as location search, proximity search, and KNN search.
The framework provides a convenient web-based REST interface to the end user. Cassandra performs fast data retrieval based on partition key and clustering key compared to Spark. The application architecture facilitates end users to execute ad-hoc queries on a suitable framework either Spark or Cassandra via a common user interface. The low latency queries are executed on Cassandra, whereas complex queries (e.g. aggregated and spatial queries) are executed on the Spark framework. The analytical results are explored to end users through visualization and REST interface. The performance of the framework is evaluated in terms of latency against the variable size of data. The performance of the framework is compared with the baseline technology, i.e. Cassandra for low latency queries.
Data visualization makes complex data more accessible, understandable, and usable. The implementation architecture provides a web-based user interface by developing analytical and visualization services through Restful ad-hoc APIs and interactive maps.
The Data visualization framework [26] is implemented to showcase the analytical results with dynamic layouts. A dashboard application is designed and implemented to depict the analytical results in the agriculture domain.
The challenges related to big data application development in agriculture is different in developed countries and developing countries. In developing countries, the major barriers for big data application development in agriculture are lack of tools, infrastructures, data standards, semantics, integrated data models, developers APIs, unified access points for public and private data, technical expertise, and finally the data.
The prototype applications in the agriculture domain are developed on top of the big data analytics architecture. Spatial and non-spatial data on weather, crop, and market are collected from different sources like meteorological departments, agriculture universities, and web portals. The summary of data collection is given in Table 1. The snapshots of the dashboard results are shown in figure 4 and 5.
The big data analytics architecture is designed and developed for massive scale data management including spatial data. The architecture is developed to address two big data challenges: Variety and Volume. The data preparation framework is designed with two levels of data abstraction. As a part of implementation, the REST interface is designed and implemented to fetch and collect data from different data sources with different formats such as PDF, spreadsheets, documents, web pages, and online services. The integration of data collected through the REST interface is the most critical module in data preparation framework. An algorithmic solution is to be devised to link a variety of data from diverse sources in aid of the unified search, query, and analysis. The core component of big data analytics architecture, i.e. big data analytics framework is implemented for spatial data management. The framework is to be extended by developing complex spatial operations like spatial join and kNN join. The spatial applications like spatial aggregation and spatial auto-correlation are to be developed on top of the framework. The complex applications in the agriculture domain are to be developed by identifying new data sources, formats, and data types. The real-life datasets including real-time and streaming data are to be collected and stored in a data repository to perform further analytics. The near real-time data analytics and visualization algorithms are to be devised to process real-time data like weather, disaster, etc. The analytical services like rainfall prediction, crop recommendation, crop price prediction, agro-inputs procurement, supply chain management, crop disease alert, fertilizer recommendations, etc. are to be implemented. These services can be used to generate customized and multilingual solutions in the form of weather-based crop calendar and alerts based on adverse events.
I would like to acknowledge and thank my Ph.D. thesis supervisor Dr. Sanjay Chaudhary, for his excellent guidance, constant encouragement, patience, care, and support. This work is a part of a research project on ‘Developing Data Analytics Architecture, Applications in Agriculture’, funded by NRDMS and NSDI, Department of Science and Technology, Government of India.
[1] Hadoop, Apache. "Hadoop." 2009-03-06. http://hadoop.
apache.org (2009). [11] Web site of
http://simin.me/projects/spatialspark/
Stack.
, [23] Sinnott, Richard O., Luca Morandini, and Siqi Wu. "SMASH: A cloud-based architecture for big data processing and visualization of traffic data." In Data Science and Data Intensive Systems (DSDIS), 2015 IEEE International Conference on, pp. 53-60. IEEE, 2015. [25] Shah, Purnima, Deepak Hiremath, and Sanjay Chaudhary. "Big data analytics architecture for agro advisory system." In High Performance Computing Workshops (HiPCW), 2016 IEEE 23rd International Conference on, pp. 43-49. IEEE, 2016. [26] Shah, Purnima, Deepak Hiremath, and Sanjay Chaudhary. "Towards development of spark based agricultural information system including geo-spatial data." In Big Data (Big Data), 2017 IEEE International Conference on, pp. 34763481. IEEE, 2017.