Big spatial data can be produced by observation or numerical simulation programs and correspond to points that represent a 3D soil cube area. However, errors in signal processing and modeling create some uncertainty, and thus a lack of accuracy in identifying geological or seismic phenomenons. To analyze uncertainty, the main solution is to compute a Probability Density Function (PDF) of each point in the spatial cube area, which can be very time consuming. In this paper, we analyze the problem and discuss the use of Spark to efficiently compute PDFs.
In this paper, we take advantage of Spark [
Figure 1 shows that the set of observation values at a point may have four distribution types, i.e. uniform (a), normal (b), exponential (c), and log-normal (d). The horizontal axis represents the values (V) and the vertical axis represents the frequency (F). The green bars represent the frequency of the observation values in value intervals and the red outline represents the calculated PDF. During the calculation of the PDF at a point, there may be some error between the distribution of the observation values and the calculated PDF. We denote this error by PDF error (or error for short in the rest of the paper). In order to precisely fit a PDF based on observation values, we need to reduce this error. Once we have the PDF of a point, we can calculate the QOI value that has the highest possibility, with which we can quantify the uncertainty each spatial data set.
Let DS be a set of spatial data sets, dk be a spatial data set in DS and N be the
number of points in a region. Each point px,y, where x and y are spatial dimensions in
the slice, has a set of values V = {v1, v2, ..., vn} while vk is the value corresponding to
the point px,y in dk 2 DS. Based on these notations, we define Equations 1 - 4, which
are based on the formulas in [
Pn i=1 vi n
x,y = s Pin=1(vi
µ)2 n
1 E =
P px,y2 slicei ex,y,i
N ex,y,i =
N X k=1 |
F reqk
N
Z min+(max min)⇤ Lk min+(max min)⇤ kL1
P DF (x)dx | (2) (3) (4)
We can now express the problem as follows: given a set of spatial data sets DS corresponding to the same spatial cube area C = {slice1, slice2, ..., slicej }, how to efficiently calculate the mean, standard deviation values and the PDF F at each point in slicei 2 C with a small average error E not higher than a predefined average error ".
The architecture (see Figure 2) has four layers, i.e. infrastructure, basic process to
compute PDFs, memory management and methods to compute PDFs. In the infrastructure
layer, the big spatial data is stored in NFS [
The basic processing of PDFs consists of data loading, from NFS to Spark RDDs, followed by PDF computation using Spark. The data loading process treats the data corresponding to a slice and pre-processes it in parallel, i.e. calculates statistical parameters of observation values of each point. Then, the PDF computation groups the data and calculates the PDFs and errors of all the points in a slice in parallel.
In order to efficiently compute PDFs, we use two memory management
techniques: data caching and window size adjustment. We use data caching to reduce disk
accesses with the Spark Cache operation and a memory-based file system [
We use two methods to compute PDFs efficiently, i.e. data grouping and ML prediction. The data grouping method groups the points with exactly the same mean and standard deviation values into a group, using the aggregation operation in Spark. Then, one point is selected for each group to compute the PDFs. The ML prediction method is based on a decision tree to predict the distribution type to compute PDFs for each point.
To validate our solution, we implement the two methods in a Spark cluster and performed extensive experiments using three big spatial data sets of from 235 GB to 2.4 TB, generated based on the seismic benchmark of the HPC4e project [HPC4E ]. The experimental results show that our solution is efficient and scales up very well compared with Baseline, i.e. brute-force method without using data grouping and ML prediction. Grouping outperforms Baseline by up to 92% (more than 10 times) without introducing extra error. ML can be up to 91% (more than 9 times) better than Baseline with very slight acceptable extra error (up to 0.017). The combination of Grouping and ML can be up to 97% (more than 33 times) better than Baseline with an acceptable extra error (up to 0.017).