The article is devoted to the analysis of the eficiency of the system of dynamic block access to data of ultra-large distributed ERS archives, implemented according to the technology developed at the Space Research Institute. The system under consideration is intended to prepare the required set of data blocks with given characteristics for their processing on a server cluster. Based on the analysis of the most significant factors afecting the performance of this system, a methodology was developed that makes it possible to estimate the preparation time of the required data set using the existing hardware infrastructure.
Many tasks related to the processing and visualization of spatial data can be most efectively solved using the block approach, which uses data on tiles, into which the entire area of interest is evenly divided. This approach is especially relevant for processing long time series of heterogeneous Earth remote sensing satellite data over large areas in order to obtain regular information products that allow analyzing the dynamics of changes in certain surface characteristics. Such products are currently widely used to solve a wide range of scientific and applied problems related to the monitoring of the natural environment and anthropogenic factors. In the framework of the described approach, the processing of satellite data on the area of interest is carried out in parallel on a set of servers, each of which is provided with a subset of tiles into which this area is divided. The main advantage of this mechanism is the ability to provide the required degree of parallelization of processing, which allows the most eficient use of available computing resources. This approach also makes it possible to efectively implement caching of data tiles in cases where they can be reused, which can significantly reduce the load on both computing resources and used network connections. It should be noted that the block approach is currently used by almost all spatial data visualization systems that provide mass access to such information.
The most common implementation of the block approach at present is the use of specially
organized archives of satellite data, which provide storage of various types of data in a single
spatial partition and projection. However, this option works well enough only in those cases
when it is necessary to operate with a relatively static and homogeneous set of data, for which
a single scheme for organizing the storage and presentation of data can be chosen. At the same
time, for the processing and analysis of data over large areas with the joint use of information
from diferent observation systems, such an approach in many cases is dificult to implement
and impractical. This is primarily due to the fact that the optimal schemes for organizing data
storage with diferent resolutions and observation schemes can difer significantly from each
other. It is important to note here that at present, to solve real problems, data from a large
number of various ERS satellite systems are used [
An alternative to the described implementation is the dynamic formation of data blocks
in such a spatial division and with such characteristics (spatial resolution, projection, set of
channels, etc.) that will be optimal for solving a specific task of processing or visualizing data.
This allows use of the main advantages of the block approach for solving specific data processing
problems based on the use of existing ultra-large satellite data archives. In this case, dynamic
block access can be provided both to data, the storage of which is implemented in the form of
certain regular partitions, and to data, the storage of which is not tied to such partitions. In
order to implement the above variant was developed a technology for dynamic block access to
the satellite data archives of IKI-Monitoring Shared Use Center [
The processing of large arrays of remote sensing satellite data, naturally, requires significant
computing resources, and also makes high demands on the network connections used in the
process of its implementation. Therefore, the task of optimizing the procedures for the dynamic
provision of data by blocks is one of the key tasks in the implementation of the above technology.
This article is devoted to a comprehensive analysis of the main factors afecting the performance
of a dynamic block access system to distributed satellite data archives. These factors include the
performance of processing servers and data storage systems, the speed of network connections,
the size and type of satellite data provided, etc. The article presents a technique that allows you
to quickly assess the resources required to provide a given set of data blocks with the desired
characteristics. It allows, at the stage of implementing a new type of processing, to estimate the
time spent on providing data blocks, estimate the resources required for this task and work out
an optimal solution based on the real performance of the existing infrastructure.
Within the framework of the developed technology of dynamic block access to data in satellite
data archives [
The execution time of a request to prepare a block of data depends on many factors, including the above characteristics of the files being prepared, as well as the type of initial data, their storage organization scheme, the speed of access to specific file servers, and the performance of network links. Establishing all dependencies without building a simplified model takes a lot of time and resources, which does not allow you to quickly estimate the required time for a new type of processing. To build such a model, the influence of the most significant of them was analyzed, and a number of simplified dependencies were established that can be used to estimate the preparation time for diferent samples of initial data. In particular, the times of data transmission over the network, as well as the time required for their compression, do not significantly depend on the type of source data and the organization of their storage. Also, within the framework of the developed model, the dependences of the preparation time of data blocks are used not on their size in geographic coordinates, but on the size in pixels of the image. It is important to note that the simplified model described below is approximate, and the estimates for the option chosen for its use can be refined if necessary. The time of data preparation for one channel of the satellite device was chosen as the main time to be modeled. Below is a formula for estimating this time as the sum of diferent dependencies that can be established experimentally:
T-channel(, store-type, result-projection, compression-type) =
where: is the linear size of the received data block in pixels, assuming that the blocks are square; store-type — a set of characteristics of source files, the key of which are their projection and the characteristic size of fragments; result-projection — projection of the received data; compression-type — used compression algorithm.
The first term in the T-base formula describes the base (main) time of formation of the data block, depending on its size, projection and fragmentation of the source data and the projection in which the results are to be obtained. It is essential that the dependence obtained for a specific type of data can be applied to other types of satellite data similar in terms of storage organization. The second (optional) term T-compression corresponds to the time required to compress the received data by block and depends only on the block size and the compression algorithm used. The third term T-get determines the time it takes to transfer the resulting file over the network, and depends only on the size of the data. Further in the article, the main methods for evaluating each of the dependencies considered above are briefly presented with examples of the dependencies obtained.
To obtain the basic dependence of the preparation time of data blocks on their size in pixels, it is necessary to obtain this time for various values of this size. It is important to note that the preparation time for a data block significantly depends on the speed of access to a particular file storage system, on which the initial data necessary for its formation are located. The diference in the performance of diferent storage systems can be associated with the peculiarities of their hardware implementation, the degree of workload and the bandwidth of network communication channels. Also, the time of formation of data blocks may depend on a number of other factors, such as the geographical coordinates of the requested area and the degree of its fullness with data. Therefore, it is necessary for each block size to obtain the average time value for all data blocks that must be provided for processing. Within the framework of the presented methodology, to estimate this time, the average value is taken over a set of randomly selected data blocks related to the desired sample. At the same time, for each block size, this sample is generated anew, which avoids caching of reading the original data files, which can significantly distort the results.
Figure 1 shows examples of basic dependencies obtained during the formation of blocks according to the data of the OLI_TIRS instrument installed on the Landsat series satellites for the period from 2013 to 2021 and a range of coordinates approximately corresponding to the territory of Russia. Dependencies are given for the case with the preservation of the UTM projection and for the case with the result being obtained in the geographic projection. The conducted studies have shown that when collecting suficient statistics, the obtained dependences are well approximated by polynomials of the second degree, which makes it possible, on the basis of a small number of measurements for diferent block sizes in pixels, to accurately estimate the preparation time for any such size. The fact that when the block size decreases, the preparation time does not tend to zero is associated with the presence of overhead costs for each launch of the data block generation procedure. It is important to note that a significant drop in the eficiency of disk read and write operations with a decrease in the amount of data with which they operate makes a significant contribution to almost all investigated dependencies.
In order to evaluate the eficiency of data generation procedures for diferent block sizes, it is possible to obtain the dependence of a fixed data area preparation time on the block size in pixels. An example of such a dependence is shown in Figure 2. It can be seen from it that the specific time decreases with an increase in the block size, but for large values of this size this efect practically disappears.
The dependence of the compression time on the block size can be obtained by subtracting the dependence obtained for preparation procedure without compression from the dependence of the preparation time for blocks using a given compression algorithm. An illustration of this approach is shown in Figure 3. It shows curves corresponding to the preparation time of data without compression, using diferent compression algorithms supported by gdal software, and curves corresponding to the desired T-compression dependence. For illustration, the fastest LZW compression algorithm and the slower and more eficient DEFLATE algorithm are selected. The required dependencies are shown in the graph with dashed lines. Note that the resulting dependences of the time spent on compressing blocks on their size are also well approximated by polynomials of the second degree.
The degree of data compression using each of the above algorithms was also evaluated. When compressing blocks that are completely filled with data, the compression percentage for the DEFLATE algorithm is stable at around 23%, and for the LZW algorithm, the file size even increases by about 3%. However, the use of the LZW algorithm may still be advisable when a significant part of the data blocks is not completely filled with data and previous versions of the gdal software are used, in which the storage of such data is implemented ineficiently.
Since the download time of data files using the standard HTTP protocol practically does not depend on their content, we can first establish the dependence of the download time of files on their size in megabytes, which is typical for the existing infrastructure. An example of such a dependence, well approximated by a linear function, is shown in Figure 4. The following graph shown in Figure 5 shows the corresponding dependence of the file transfer rate on its size. This speed tends to the maximum for the used communication channel as the file size grows.
Further, Figure 6 shows examples of the required dependences of the download time of data blocks on their size in pixels using various compression algorithms, which are approximated by polynomials of the second degree with coeficients calculated on the basis of the established linear dependence.
The article presents the main results of a comprehensive analysis of the efectiveness of a system of dynamic block access to data of ultra-large distributed archives of satellite data for processing them on a cluster of servers. Based on the analysis of the main factors afecting the eficiency of the mechanism for providing data blocks, a method was developed that allows you to quickly estimate the time spent on preparing a given set of initial data. The developed technique can also be used to select the optimal size of the data block and other characteristics of the data provided.
The work is performed in the frame of “Big data in space research: astrophysics, Solar system, geosphere” project (state reg. No. 0024-2019-0014).