Analysis of diverse data in geosciences requires access to various processing tools, including specialized software packages, proprietary algorithms, GIS systems, web services, etc. Such tools require from the user a certain level of skills to work with them, form a software environment, the availability of the necessary computing power, and sometimes significant time and financial costs. With the development of information technology, more and more software products, including professional software packages for data processing, are provided to users in the format of various cloud services and platforms. In such systems, computation takes place on the side of the service provider and is accessed through a web browser. The emergence of such open-access services and platforms makes it possible to organize a single workspace for a researcher with the ability to analyze his own data using various processing methods, including tools traditionally used in earth sciences. The report is devoted to the development of an approach for the integration of various tools for processing heterogeneous data with open access within a single multi-platform ecosystem. The software system developed based on the proposed approach is demonstrated. The report describes software modules that implement the functions of access to processing and analysis tools, as well as service modules for system administration, component monitoring and event logging. Services and processing platforms integrated into the ecosystem are considered, as well as scenarios for solving some geological problems.
Since 2017, at the Vernadsky State Geological Museum RAS, an information and analytical
environment is being developed to support scientific research in geology (http://geologyscience.
ru) [
Processing and analyzing data in geology requires the application of a large number of diferent algorithms, processing procedures and corresponding software solutions. With the development of information technology, approaches to the organization of such processing have changed. Solving the problem of analyzing large amounts of data or tasks requiring the use of resource-intensive methods required the acceleration of software computations. To solve such problems, both parallel computing using supercomputers and distributed computing using a large number of computing devices combined into a single computing system, such as GRID systems, were used. However, access to such systems is limited.
The advent of cloud services has allowed users to gain easy access to the computing resources they are interested in. The use of external services instead of custom applications allows data processing on the equipment that is most suitable for the corresponding tasks. Thus, data processing is more eficient, and the user gets the opportunity to process data with the most current versions of algorithms, using a web interface for this, without the need to install, configure and maintain software for processing on his personal computer. Due to the continuous growth in the amount of geological data and tools for their analysis, it becomes necessary to organize a single workspace for a geologist-researcher with the ability to use open access tools for analyzing geological information available in the world.
The purpose of the computing block of the information and analytical environment is to
provide researchers with access to geographically distributed services for processing and analyzing
geological information (https://service.geologyscience.ru) [
Basic functional requirements: ∙ ability to use external open information processing services in unlimited quantities; ∙ working with data provided by users and open information systems; ∙ cataloging external processing services.
To organize a single mechanism for interacting with web services, the OGC Web Processing
Service (WPS, http://www.opengeospatial.org/standards/wps) interface is used, developed by
the international organization for standardization of the Open Geospatial Consortium. This
interface is implemented on the basis of the HTTP protocol (HTTPS), is widely used in various
scientific data processing systems [
Basic requirements for this type of service: ∙ Permanent IP address (or domain name) and a port with external access via the Internet. ∙ Client program interface (API) and libraries implementing it in the Java language, or via the HTTP/HTTPS protocol (REST or SOAP); ∙ Ability to start the processing procedure with the specified parameters; ∙ Ability to obtain the result of processing in text or binary formats, including in the form of URL links; ∙ Ability to work with user data in one of the following ways: — reading remotely placed data by URL address; — temporary loading data to the computing node; — transferring data in binary format as a processing parameter.
Open source software package GeoServer (http://geoserver.org) was selected to create and host your own WPS processes. Thus, for each external service, a separate WPS process is created with the corresponding startup parameters. Using WPS as an access interface to remote services allows you to execute several computational processes sequentially. Thus, outcomes of one or several processes can be used as input parameters for another process, thereby providing the possibility to combine the results of processing diferent types of data.
Using a common access protocol to interact with cloud services that provide users with interactive access through a web interface is dificult due to the heterogeneity of the access protocols used in each platform. However, some common properties of platforms, such as the presence of centralized data storages and single authentication systems, make it possible to organize interaction by transferring data for analysis to the data storages corresponding to the platforms on behalf of a specific user. The implementation of such a mechanism has become possible when using web-application technology, which allows you to request permission from the user to access certain capabilities of the user account of various cloud service providers. This technology is supported by Microsoft, Google, Yandex, ESRI, etc. Before using the corresponding service, the user needs to upload data for analysis to his personal storage. The user is authorized on the website of the service provider, after which the application asks the user for permission to download and publish data to its storage. The user has the ability to move data for processing from one storage to another when choosing a cloud service from diferent providers.
Thus, we have proposed an approach and a technological solution for organizing a single data space for various cloud service providers.
A monitoring module has been developed to track the state of geographically distributed
components of the ecosystem [
The following general types of tests are presented.
B. Checking service performance at a remote site using the required communication protocol. C. Checking for changes in the operation of the service based on test requests.
The module of the catalog of external services for processing and analyzing geological information is used as a data source for monitoring.
The developed software ecosystem provides access to the following services and processing platforms.
— Computing node “Multidimensional methods of data analysis”, developed at the State
Geological Museum of the Russian Academy of Sciences, which allows you to process
tabular data by various methods of data analysis with setting their parameters and
visualizing the results. The computational node includes such groups of methods as
statistical analysis, regression analysis, factor analysis, clustering, machine learning,
visualization methods, and others. Calculations are performed in the Python environment
using well-known data processing packages: Scikit-learn, Pandas, Matlplotlib and others.
The processing of incoming requests for processing is carried out by the Flask framework
through the REST API using the task queue, implemented on the basis of the NoSQL Redis
database. This architecture allows the processing of requests and heavy computations of
large amounts of data to be separated, which provides fault tolerance and scalability of
the node. In the future, it is planned to expand the number of data processing methods
with the involvement of specialized processing packages for solving geological tasks.
— Petrological and geochemical data processing. An interactive database of methods for
processing petrological and geochemical data has been developed at the Schmidt Institute
of Physics of the Earth of RAS [
The ArcGIS platform allows users to work with 2D and 3D data to explore and visualize it. One of the key features is the ability for multiple users to collaborate on the same data. The platform provides tools for creating web maps, 3D scenes and notebooks. Using ArcGIS Notebook allows you to access Python resources to perform analysis, automate workflows, and visualize data. — Google Earth Engine is a satellite data analysis platform (https://earthengine.google.com).
This platform allows the user to upload their own data or use data from the Earth Engine catalog for further processing in an interactive mode. The catalog contains data processing products for the Modis radiometer (Aqua, Terra satellites), Sentinel-1A, Sentinel-1B, Sentinel-2A, Sentinel-2B, Landsat 8, etc. creation, editing and launching using Javascript and Python programming languages. To work, the user needs a Google account. For analysis and processing, you can use data from the Google cloud storage.
The general functional diagram of the multi-platform ecosystem is shown in Figure 1.
Testing of the system functions of the processing platform took place with the automatic testing tools provided by the framework used in the development.
∙ interaction with the platform data store; ∙ interaction with the service catalog; ∙ interaction with WPS services; ∙ interaction with third-party data stores.
The testing of the services presented on the platform was carried out in manual mode using the initial data, input parameters and expected results based on data taken from the sources corresponding to the service topic. So, for example, to test the operation of the computing node services using multidimensional methods of data analysis, the materials of the book by J. Davis “Statistics and analysis of geological data” were used.
This ecosystem is being developed to provide geological research with modern methods and tools for analyzing geological information, which implies a further increase in the number of processing tools and their varieties provided by the platform.
The principles of the ecosystem being developed can be used in the future to create various digital computing platforms to support and accompany scientific research.
The study is supported by the Government contract No. 0140-2019-0005 with SGM RAS “Development of an information environment for integrating data from natural science museums and their processing services for Earth sciences”.