The work is devoted to the development of a computing node for processing satellite and spatial data for earth sciences by the example of its implementation as part of the Information and Analytical Environment to support scientific research in geology of the Vernadsky State Geological Museum (SGM RAS). The prerequisites for the creation of such a computing node and the requirements for it to solve geological problems are given. An overview of cloud platforms for access to satellite and spatial data and its processing has been presented. Based on the overview a conceptual diagram of a computing node has been proposed and the list of modern technologies required for building it has been determined. The developed node provides tools for searching data from external cloud providers, processing them with various built-in and custom algorithms, as well as tools for visualizing the results. It is an independent web service, although it is part of the Computational and Analytical Geological Environment of SGM RAS and is integrated with its services. This allows a wide range of users to access data and processing algorithms provided by computing node, including integrating it into other information systems as a third-party application for processing satellite and spatial data.
The most of nowadays research in geosciences cannot be conducted without satellite or spatial data. Remote sensing with modern satellites provides information about Earth in many spectral ranges. Various physical parameters of the surface, ocean and atmosphere are calculated using this information. Satellite imagery is used in research in meteorology, oceanology, geology, and other earth sciences. Processed remote sensing data become crucial in industry, agriculture, territory management and other areas.
Problems and tasks that involve satellite imagery and spatial data processing require huge amount of storage space and computing power in terms of infrastructure, as well as a large number of special competencies in the field of geoinformatics in terms of qualification requirements. This fact along with the constant growth in the number of new satellites launched every year have led to the emergence of specialized cloud platforms and services to work with satellite imagery and spatial data. They reduce the cost of solving certain scientific and applied problems, providing access to data and giving tools and algorithms to process this data. Considering only scientific problems one can notice that they define a number of basic requirements for such cloud platforms and services: the possibility of open access for scientific purposes, simplicity and flexibility of use for various scientific problems, the possibility of implementing custom processing algorithms, and reproducibility of results.
Three main problems arise working with satellite and spatial data: 1) spatiotemporal search for satellite and spatial data and access to them; 2) interactive data visualization and analysis; 3) data processing with standard or user-defined algorithms.
Modern cloud solutions in considered field partially or completely solve these problems. The
problem of access to satellite imagery should be highlighted here. The development of satellite
remote sensing and forming of various initiatives to provide access to satellite data made it
possible to create distributed systems providing access to a wide variety of satellite and spatial
data, and researchers and other users got access to data directly through the web interfaces,
without the need to download them directly into local storage. Mentioned above problems of
working with satellite and spatial data and impact of modern cloud platforms to their solution
are discussed in the related work [
Information and Analytical Environment has been implemented to support scientific research
in geology (http://geologyscience.ru) created few years ago operates in Vernadsky State
Geological Museum of Russian Academy of Sciences (SGM RAS) providing a single entry point to various
databases and cloud processing services for solving scientific problems in geology [
In this paper, an approach to creating a computing node for processing satellite and spatial data in the ecosystem of the Information and Analytical Environment of the SGM RAS for solving geological problems is proposed to consider.
There are many cloud platforms and services to work with satellite and spatial data nowadays.
They difer in functionality, type of satellite and spatial information provided, thematic focus,
ease of use, cost. Many of the platforms provide an option to use them for research purposes, but
often only a part of the functionality provided. Examples of several platforms, their architecture,
and provided features are presented in the related work [
General-purpose platforms allow to solve a wide range of tasks by means of access to diferent satellite and spatial data, variety of visualization tools, custom processing algorithms and the mechanisms to share the results of work. The main general-purpose platforms are considered below.
Google Earth Engine (https://earthengine.google.com) is a cloud platform for analysis and visualization of geospatial data, free for the scientific purposes. The platform aggregates data from various satellite platforms and instruments. To work with this archive of satellite and spatial data, tools for visualization tools and development of processing algorithms are provided powered by the Google Cloud Platform.
Earth Observing System (https://eos.com) is a commercial cloud platform that provides access to a wide range of satellite data, including ultra-high resolution. The platform ofers a wide range of visualization and analytics tools, as well as a set of common algorithms for processing satellite data for various tasks. To use it in scientific purposes it is required to sign special agreement.
Planet (https://www.planet.com) is another commercial platform from Planet, which is distinguished by the fact that the company has its own constellation of Earth observation satellites, which makes it possible to track in more detail all changes in various parameters of the regions of interest. In addition to valuable satellite data, the platform has advanced visualization and change monitoring tools, as well as integration tools with various satellite data processing applications. It is possible to use the platform and its data for research and educational purposes.
Descartes Labs (https://www.descarteslabs.com) is a commercial cloud platform that aggregates and prepares satellite data for further analysis from various data providers. It has modern tools for visualization and data analysis, as well as special workspaces where, using the computing power and the programming interface of the platform in the Python programming language, users can build their own satellite data processing workflows to solve their problems. It also provides an opportunity for free use in for scientific purposes.
ArcGIS Online (https://www.esri.com/en-us/arcgis/products/arcgis-online/overview) is a commercial cloud platform from a well-known developer of geographic information systems (GIS), which is primarily a cloud GIS. Provides a wide range of tools for working with geospatial data (raster and vector), including analytics and a Python API. It is possible to upload your data, share the results of your work, use the results of the work of other users of the platform.
Astraea (https://astraea.earth) is a commercial cloud platform which has a well-structured satellite and spatial data processing workflow as a distinctive feature. 1. Automatic continuous delivery of satellite images of the region of interest from the required satellites. 2. Custom processing algorithms creation using the JupyterLab environment in the Python programming language. 3. Algorithms scaling to cloud computing nodes using Big Data approaches. 4. Creation of automated analytical tasks using low/no-code approaches.
The platform provides an opportunity for cooperation in solving research and scientific problems.
Sentinel Hub (https://www.sentinel-hub.com) is a commercial platform with great opportunities for research and solving scientific problems. It provides access to data from satellites of the Sentinel and Landsat constellations, as well as satellites Meris and Proba-V, has flexible visualization tools, and, like Google Earth Engine, allows users to implement their own scenarios for processing satellite data and provides a number of software interfaces for automated interaction with platform. It is also possible to use custom data.
Thematic platforms are aimed at solving a certain narrow range of tasks, as a rule, limited to a certain geographic region, providing the most relevant and thoroughly selected set of data, tools and algorithms for these tasks and region. Here are some examples of thematic platforms.
USGS Web Applications (https://www.usgs.gov/products/data-and-tools/web-application) is a collection of thematic web services from the US Geological Survey to solve a wide range of tasks in areas from geology to climate, mainly for the territory of USA. Web services use the USGS cloud data access platform and internal cloud infrastructure. All services and data access platform are free for scientific research.
NASA Earthdata Tools (https://earthdata.nasa.gov/earth-observation-data/tools) is a set of cloud-based tools from the US National Aerospace Agency, which provide the following functionality: data search and ordering, data preprocessing and filtering, geolocation and cartography, data visualization and analysis. Each of the services from this catalog is focused on solving a fixed range of tasks from certain domains. The services are also free and not limited just to the United States region.
Copernicus Marine Service (https://marine.copernicus.eu) is a cloud-based platform focused on research of the World Ocean, which is implemented as part of the European Union’s Copernicus Program. The platform ofers access to satellite and spatial data on the state of the ocean, as well as a range of tools for visualizing and analyzing various parameters of the ocean, thereby allowing to solve scientific and applied problems requiring this type of data. The platform is free and covers the entire World Ocean. There are similar thematic platforms within the Copernicus Program for climate, atmosphere and land studies.
Digital Earth Australia (https://www.ga.gov.au/dea/products) is a thematic cloud platform for monitoring various physical parameters in Australia. It combines satellite data and a set of thematic services that are focused on solving specific problems: from changing coastlines to determining freshwater reserves. Access to the platform is free.
Swiss Data Cube (https://www.swissdatacube.org) is a thematic cloud platform similar to the previous one, only focused on monitoring the territory of Switzerland.
Brazil Data Cube (http://brazildatacube.org) is a thematic cloud platform focused on monitoring the territory of Brazil.
The above review of cloud platforms for working with satellite and spatial data allows us to conclude that, to solve modern problems, such platforms, on the one hand, require the implementation of data access, processing and visualization capabilities, and on the other hand, simplicity and flexibility of use in modern thematic scientific tasks with the ability to define data sets and algorithms for their processing. At the same time, general-purpose platforms still have their own specifics and diferent goals, diferences in the set of services provided, and the possibilities of using them for research purposes. This explains the large number of such platforms.
Earth observation data are widely used in modern geology in a wide range of tasks: lineament analysis, minerals mapping, structural-tectonic analysis of deposits, monitoring of geodynamic processes, study of permafrost processes, study of the material composition of rocks, rational land-use, etc. Solving these problems requires a certain set of satellite and spatial data and algorithms for their processing, the composition of which is constantly changing with the development of methods for solving problems, also the problems themselves are changing. This defines certain requirements on tools for processing satellite and spatial data, which should combine the properties of both general-purpose and thematic platforms.
In terms of data access requirements, it is necessary to search and obtain data from various satellites and radiometers, as well as from spatial data catalogs. One solution is to create a data access gateway. Thus, the service for accessing satellite data of the Information and Analytical Environment of the SGM RAS (http://sputnik.geologyscience.ru) provides the search of satellite data from various providers: the US Geological Survey (http://usgs.gov), the Japan Aerospace Agency research (https://www.eorc.jaxa.jp), satellite center. Goddard (https://oceancolor.gsfc. nasa.gov), Scientific Center for Operational Monitoring of the Earth (http://www.ntsomz.ru), Center for Collective Use of Regional Satellite Environmental Monitoring of the Far Eastern Branch of the Russian Academy of Sciences (http://satellite.dvo.ru). The service provides access to the following types of satellite information: data from Landsat-7/8 satellites and Sentinel2A/2B satellites; satellite topography data STRM and ALOS; data from meteorological satellites Aqua and Terra; data from satellites EO-1, OrbView-3, as well as from the Russian satellite Kanopus-V.
Another option for providing access to data is to connect to satellite and spatial data catalogs that implement the modern rapidly developing Spatiotemporal Asset Catalog (STAC) speciifcation (https://stacspec.org). This specification regulates the rules for describing data and collections of spatiotemporal data to provide unified access to this data and navigation through it using a self-documented directory structure and data description. The main advantage of using this specification by data providers and their consumers is the uniformity of access to data without the need to change processing workflows and algorithms when adding new data types.
Modern work with satellite and spatial data takes place in web applications through interaction with a cartographic interface, into which the needed data is loaded, receiving from cloud platforms in real time. To provide such work with data, special storage formats and some preprocessing of real data are required to optimize delivery over the network, as well as render in the browser. For these purposes, the approaches ofered by the Open Geospatial Consortium (OGC) (https://ogcapi.ogc.org) have been used for a long time: Web Map Service, Web Feature Service, Web Coverage Service, etc. But more recently, the Cloud-Optimized Geotif (COG) standard (https://www.cogeo.org) has also begun to be used for these purposes. This standard adds to regular Geotif files the ability to store overview images as well as smaller chunks of the original image for quick access maintaining backward compatibility. The only prerequisite for accessing such data over the network is support for HTTP GET Range requests by both the client and the server.
The overview of thematic cloud platforms for working with satellite and spatial data shows
that an increasing number of such platforms are created using modern opensource software Open
Data Cube (ODC) (https://www.opendatacube.org) [
The modern landscape of platforms to work with satellite and spatial data, as well as a set of technological solutions and standards in the field of geoinformatics, make it possible to implement a similar approach when developing a computing node for processing satellite and spatial data within the ecosystem of the Information and Analytical Environment of SGM RAS. The schematic diagram of the computing node is shown in Figure 1. Computing node consists of the following components: • a cartographic web interface for user interaction with a computing node by searching for satellite and spatial data for the region and dates of interest, defining a processing algorithm, choosing a visualization and analysis methods for the result; • a subsystem for processing and dispatching requests, which provides interaction between the web application and the rest of the computing node; • a data access subsystem that provides interaction with STAC-catalogs of external cloud providers, a service for accessing satellite data of the SGM RAS, and contains service functions for working with the local ODC catalog, which is required for the ODC to work properly; • data processing subsystem, which is based on the ODC platform, providing tools for satellite imagery processing algorithms implementation; • a task queue, into which incoming requests for data processing are placed with an indication of the algorithm from the data processing subsystem, and which can track the status of tasks execution; • task executors that process tasks from the queue in a distributed manner; • local storage of data processing results in COG format and temporary files.
Calculations are performed in the Python environment using the ODC package and several auxiliary packages for working with geospatial data. The processing of incoming requests for data processing is performed by the FastAPI framework using the REST API implemented according to the OpenAPI standard (http://spec.openapis.org/oas/v3.0.3), using a queue of computational tasks based on the NoSQL Redis database.
Moreover, individual components of the system — the web interface, subsystems, the local ODC directory, a database with a task queue, task executors — are deployed using Docker containers. This architecture allows the processing of requests and heavy computation of large amounts of data to be separated providing fault tolerance and scalability of the considered computing node.
The current implementation of the proposed conceptual scheme is a prototype with a cartographic web interface that implements data search in the STAC catalog of the Landsat-8 satellite (https://landsat-stac.s3.amazonaws.com), as well as algorithms for calculating various spectral indices. satellite channels.
Further development of the computing node is supposed to be carried out in three directions: • increasing the functionality of the web interface by adding new visualization and data analysis tools; • development of a data access subsystem through the ability to connect arbitrary STAC catalogs, as well as integration with the satellite data access service of SGM RAS; • expansion of the list of algorithms provided by the data processing subsystem, as well as the implementation of the possibility of adding custom algorithms.
In addition, the approaches and modular architecture the computing node is based will make it possible to transform it into an independent cloud platform for solving a wide range of tasks for earth sciences. And the ease of use and flexibility of tuning for specific tasks will attract a wide range of scientists and researchers to its use.
The study is supported by the Government contract No. 0140-2019-0005 “Development of an information environment for integrating data from natural science museums and services for their processing for Earth sciences”.