For the first time in worldwide for hydrometeorology, data processing pipeline is proposed. Approaches to its implementation are defined. The stages and software for such processing are highlighted. The main method of the universal data replenishment mechanism with the results of the pipeline is an integrated database with a wide set of metadata. The creation of information products at various stages of the pipeline is considered. It is proposed to create new services for the pipeline. The main control mechanism for pipeline data processing is considered to be tools of monitoring the state of hardware, software and information resources. This requires a transition from monitoring individual stages to automatic comprehensive monitoring of the data processing pipeline state. Tasks of the administrator of the data processing pipeline are defined. In fact, when using pipeline processing, a transition must be made to fully automatic data processing without human intervention.
Two streams of data are in the field of hydrometeorology. First - the data is received in real time via the Global Telecommunication System of the World Meteorological Organization. Streaming data is continuously generated by thousands of data sources, which send telegrams with weather messages simultaneously and in small amounts (several kilobytes). Telegrams are processed and on their basis weather forecasts are issued. Real time data is three-hour cycles, but the load is distributed over the main time of observation and it turns out to be uniform and predictable. The second stream of data is beginning 30 days after the end of the previous month. For some hydrometeorological stations, data are received with a delay from a month to a year, depending on stations hard inaccessibility. In this stream any part of the data can process and recalculating as needed. To obtain annual information on climate change, it is necessary to process data and analyzes changes for the previous year within three months after the end of the year. Most of day-to-day operations associated with different software maintenance remain the prerogative of personnel. These are associated with significant material costs. The low level of automation of day-to-day operations is becoming a serious obstacle to improving the efficiency of existing systems.
In recent years, digitalization and digital transformation have been gaining
momentum [
A pipeline is a method of organizing calculations used for processing of stream data in order to fully automate this processing and perform several operations simultaneously at different stages of the pipeline. For this it is necessary to move from a loosely coupled set of applications to the creation of an integrated control system for all stages of data processing. A pipeline requires optimization of existing software, reducing its complexity and increasing manageability.
Ideas of self-governing systems in information technologies are expressing for a
long time, and movement in this direction is observing on. A pipeline data processing
is using for next directions [
Currently, term the pipeline has become often used in data processing, for
example, to organize a chain of processes for converting source data processing
scenarios into 2D and 3D representations [
Roshydromet has examples of "end-to-end" digital technologies. For example, in
the Hydrometeocentre of Russia [
The main components of the data processing pipeline are [
To improve the efficiency of data management it needs to integrate their. External
data requires increased information about their origin, especially about processes by
which they were created. Integrated data becomes easily discoverable and available in
online [
A pipeline processing robotization consists in automating such massive processing stages as data replenishment and subsequent ordering with data control according to specified rules. Rules of processing create and storage in the form of a knowledge base that allows moving to a new stage of data processing or return to the previous stage if there is a failure of software.
Now cloud technologies are actively developing, which provide not only data
storage services, but also dedicated software and hardware resources for their
processing. Micro services architecture [
The organization of continuous data processing and hydrometeorological support (GMS) of consumers observed, aggregated, forecasted and climatic information is in fact the implementation of a new paradigm of digital transformation. To organize such a GMS, it is necessary to create a computational pipeline for continuous processing of hydrometeorological data. The purpose of developing a pipeline for processing of hydrometeorological data is to increase the efficiency of processing incoming heterogeneous and distributed data. The objective of the research are determination of approaches for the implementation of pipeline processing of data (definition of stages and software for such processing, organizing data storage and processing, obtaining of information products, delivery of data and information to consumers, monitoring of the pipeline).
2.1
a
Highlighted stages of data processing on the example in oceanography are showing in Fig. 1 and include: Collecting data from observation platforms. Inverting of observed data - presentation in a different order in relation to observed data in the form, for example, time series. Quality control of received data. Interpolation of observed data in time (time series) and or space (points of a regular grid), which are widely used for aggregation and obtaining various statistical characteristics. Buoys
Ships of voluntary
observatins
Select Dates of reaching thresholds
Research vessels Select
Join Interpolation for time
Spatial ordering
Inverted data sets Filtering and anti
aliasing Calculation of the annual variation in parameter
values
Forecast at a point Calculation of probability, repeatability Calculation of correlation and spectral functions Approximation of correlation and spectral functions
Interpolation for vertical on
standard horizons Calculating of hydrophysical
parameters Calculation of basic statistical
moments with check for stationarity and normality of
distribution Ordering by space and time Calculation of direction and velocity of current by the dynamic method in grid points
Classification of fields of synoptic type Spatial forecast
Decision support Extreme values and dates of manifestation
Spatial interpolation to grid points
Coastal stations
Select Data sets for
disciplin Sound channel characteristics
Classification of vertical distribution of
parameters Calculation of the parameters of the
upper quasihomogeneous layer
Statistical characteristics Classification of water masses Characteristics of macro regions Typical fields
Grid data
Metadata
• Identifying anomalies or assessment exceeding of the threshold values of
observed parameters and assigning of dangerous levels.
• Information delivery to consumers using the Short Message Service (SMS).
• Forecast of impacts and issue of recommendations in case of parameters
dangerous levels.
• Monitoring of all stages of data processing [
Stages of hydrometeorological data processing, regulations for their work, metrics values and tools of implementation are presented in Table 1.
iIdmemnteifdiicaatteiloynafter 3 min after detection SMS
Forecast of impacts, give out of Immediately upon r3emceiinvui ntegsianfftoerrmation Support Decision 1.6 recommendations receipt
2.2 tJiomine,, ivnevretirctianl,gs,pinatceerpolation in
accessible
Obtaining climatic 2.3 generalization
Pipeline of data processing is based on the elimination of manual operations for
searching, preparing for processing, and delivery data from one stage to another,
automatic preparation of metadata and loading of created, or replenished, or updated
inverted, calculated, generalized and of climatic data into integrated data base (IDB).
For the pipeline processing data following tools are required [
Automatic data processing is as follows. After the new piece of data arrives, a
trigger starts the data decoding procedure. Further, after decoding is completed, the
second trigger starts the next stage of processing – data loading into IDB and etc.
calculation, interpolation, and detection of anomalies. To obtain aggregated
information, the data undergoes multi-stage processing - merging separate pieces of
data from various disciplinary data sets, ordering and interpolating in time or space,
obtaining calculated new parameters, statistics, etc. An important point of such a
pipeline is that results of almost all processing operations are saved for use in various
applied tasks, for example, to calculate various properties of the atmosphere and
hydrosphere (characteristics of the sound channel, statistical characteristics, annual
variations, etc.). Saving inverted and calculated data sets is technologically expedient
and economically beneficial, since the cost of creating them each time a query is
executed is much more expensive than them creating, saving and reusing [
The pipeline of data processing is based on their integration. Data integration can produce in two ways. The first - is data description using metadata, regular data delivery, and transformation of structures, casting from local names and codes to system-wide, data loading into IDB. The second method of data integration allows, based on previously created application programming interfaces (API), web services or Representational State Transfer (REST) services, to form data tables in real time to include in the existing processing scheme. This ensures that all required data sources are connected in a timely manner. This approach hides the location and format of the data from consumers. At the same time, the data is not moved to a centralized storage, but remains in the same place where it was created or stored. Each data provider is responsible for its accessible, security, completeness, quality and relevance of data. Integrated data must be structured, well described, understandable, easy to use. They are obtained through transparent, known transformations, so that can trace all the way from data source to consumers. An example of such data integration has in Unified system of information for World Ocean (ESIMO)1. 2.3
The scheme for obtaining and using of information products based on pipeline data processing, shown in Fig. 2. Consumers browse products catalog, available for distribution by tools of access to distributed data.
, tssenym iton Application funcatbioounts:dvisiasssuutaeblrisszc,ardtiipaottniao,dnde,eldtievecectriisyoi,onfnoorsfeudcpiapssaotssrt,tearns,aglyetnicesr,ation of warnings ittirso iitcan Data catalog - metadata: data retrieval (point, profile, mesh, object files) a f reg ,ied iifend itizaon Universal access to data: applicamtioanpss,eSrQviLce,sAPI, web services, REST services, :U ro itry tauh Integrated data from external sources and processing results: observations, ceu forecasts, climate (relational DBMSs,tdaatitoanfsr)om cloud storage, automatic weather S Fig. 2. Scheme of information products obtaining and using. 2.4
To organize the operation of the pipeline, it is necessary to have a tool for a simple quick setting of requirements for the interfaces of services, applications with information products. Interfaces must allow customizing data requirements (sources, regions, parameters, presentation forms - data tables, maps of observed and or predicted data, climate information, time series graphs). This will leads to better scalability of applications, more efficient use of computing resources. Need to manage product creation workflows, his distribution and respond more quickly to detected disasters. The customization interface includes features to help consumers manage their data requirements. A pipeline must include information product requirements editor and tool of workflow management of product creation.
2.5
To organize monitoring of the data processing pipeline, in addition to monitoring individual stages of processing, testing and restoration of individual complexes, it is planned to create an automatic monitoring of the state and operability of the entire pipeline. It is important to include the ability to automatically run tests, data quality check and metadata descriptions. It is necessary to receive proactive information about incidents in the execution of data processing stages so that the system can automatically take corrective actions. All incidents should be recorded. The protocol must contain information about at what stage of processing the incident occurred, for what reason and what was done by the system in this situation. With the help of monitoring, can accumulate statistics on stages of data processing and find bottlenecks for their subsequent optimization. In this case, tasks of the pipeline administrator are following: Analysis of data processing - reasons for the slowdown. Violation of the service level agreement (SLA). Investigation of SLA violations. Quick setup of monitoring operations for new objects. Calculation of the number of consumers and indicators of GMS. Assessment of indicators of state of data processing stages in time. Coordination of management of events occurring on the data processing pipeline. 3
If in the old paradigm of data processing the sources of information were separate data sets and databases, then in the new paradigm with the help of metadata a whole data domain is opened, representing results of observations and their processing, including forecasts, and climatic data. When using a new paradigm that uses pipelining data processing, it is possible not only to obtain information products from any stage of processing, but also to connect them to existing models of calculations, analyzes and forecasts. Thus, it becomes possible to set tasks for the system such as exporting data, preparing them for use in models, describing expected results of data processing. At the same time, the consumer does not need to worry about how to export data for his task. Everything that is the result of pipelining data processing is loaded into the IDB. In the ideal scenario of using data, any consumer, without the intervention of specialists, can find the necessary data, get it up-to-date and test the hypothesis of its use. In the future, the selected technology for obtaining and using data is included in the automatic service of this consumer.
In this case, there is a transition to autonomous data processing. APIs, web
services, REST services are the foundation of this automation. With the help of them
can automatically receive data from integrated, distributed sources, which present
responsible for authors from many organizations. When integrating data into ESIMO
[
Modern hydrometeorological support should be personalized, easily customizable for new types of products, new rules for identifying disasters. It is necessary to implement methods of fast deliver of data, which necessary for heads of enterprises. The personalization is configured starting from the preparation of information products and ending with its use in decision-making. To do this, it is necessary to identify individual characteristics of each consumer, fixing his "digital footprints" completed requests on data, used business processes that depend on the hydrometeorological situation, existing experience in using data, location, and current situation at the enterprise. 4
For the first time in worldwide for hydrometeorology, it is proposed to organize the end to end pipeline of data processing from observation to decision making. Such data processing is a natural stage in the development of applied tasks. As a result of the implementation of pipeline data processing, it will become possible: • Expanding the composition of integrated data from various sources. • Realization of preliminary data processing and creation of inverted and calculated data sets automatically. • Automatically preparing regular information products and analyze data. • Using of various methods of data processing (interpolation, aggregation, statistical processing, correlation analysis, etc.). • Automatic transmission of detected anomalies or exceeding of threshold values to consumers. • To issue forecasts of impacts and recommendations for decision-making.
A data processing pipelining is especially important for disasters risk reduction, climate change mitigation, and disaster adaptation. In this case, a universal mechanism for the preparation of indicators for various disasters are created on the basis of climatic, forecast and observed data, which should also be delivered automatically to enterprises heads.