<!DOCTYPE article PUBLIC "-//NLM//DTD JATS (Z39.96) Journal Archiving and Interchange DTD v1.0 20120330//EN" "JATS-archivearticle1.dtd">
<article xmlns:xlink="http://www.w3.org/1999/xlink">
  <front>
    <journal-meta />
    <article-meta>
      <title-group>
        <article-title>Construction of a Generalized Computational Experiment and Visual Analysis of Multidimensional Data</article-title>
      </title-group>
      <contrib-group>
        <contrib contrib-type="author">
          <string-name>A.E. Bondarev</string-name>
          <email>bond@keldysh.ru</email>
          <xref ref-type="aff" rid="aff0">0</xref>
        </contrib>
        <contrib contrib-type="author">
          <string-name>V.A. Galaktionov</string-name>
          <xref ref-type="aff" rid="aff0">0</xref>
        </contrib>
        <aff id="aff0">
          <label>0</label>
          <institution>Keldysh Institute of Applied Mathematics Russian Academy of Sciences</institution>
          ,
          <addr-line>Miusskaya sq. 4, 125047 Moscow</addr-line>
          ,
          <country country="RU">Russia</country>
        </aff>
      </contrib-group>
      <abstract>
        <p>The work is devoted to the problems of constructing a generalized computational experiment in the problems of computational aerodynamics. The construction of a generalized computational experiment is based on the possibility of carrying out parallel calculations of the same problem with different input data in multitasking mode. This allows carrying out parametric studies and solving problems of optimization analysis. The results of such an experiment are multidimensional arrays, for the study of which visual analytics methods should be used. The construction of a generalized experiment allows one to obtain dependences for valuable functionals on the determining parameters of the problem under consideration. The implementation of a generalized experiment allows one to obtain a solution for a class of problems in the ranges under consideration, and not just for one problem. Examples of constructing a generalized computational experiment for various classes of problems of computational aerodynamics are presented. The article also provides an example of constructing such an experiment for a comparative assessment of the accuracy of numerical methods.</p>
      </abstract>
      <kwd-group>
        <kwd>generalized computational experiment</kwd>
        <kwd>visual analysis</kwd>
        <kwd>multidimensional data</kwd>
        <kwd>parallel computations</kwd>
      </kwd-group>
    </article-meta>
  </front>
  <body>
    <sec id="sec-1">
      <title>1. Introduction</title>
      <p>The physical experiment was the main and often the only
source of information on the problems of gas dynamics long
before the advent of the computer age. In practice, the main goal
of a physical experiment was almost always not to model the
physical phenomenon itself, but to clarify the circumstances
under which it occurs, i.e. obtaining the dependence of the
appearance of the phenomenon on the determining parameters of
the problem, such as Mach numbers, Reynolds numbers, Prandtl
numbers, and the geometric parameters of the problem. Such
large-scale experimental work made it possible to obtain key
relationships for the dependence of the gasdynamic functions of
interest or the conditions for the appearance of a physical effect
on the key determining parameters. In fact, the establishment of
such
physical laws
for
shock
waves, separated
flows,
characteristic configurations of streamlined bodies was the main
task of fluid and gas mechanics.</p>
      <p>
        As an example of such a dependence, one can cite the famous
formula of G.I. Petrov, representing the fundamental law on the
ultimate pressure drop in the shock, which the turbulent
boundary layer is able to withstand without detachment from the
wall [
        <xref ref-type="bibr" rid="ref18">18</xref>
        ]:
      </p>
      <p>P2 / P1 = 0.713Me + 0.213.</p>
      <p>Here P2 / P1 is the pressure drop, Me is the Mach number
before the separation point, varying from 1.5 to 4.</p>
      <p>
        Another example is the famous Kozlov formula [
        <xref ref-type="bibr" rid="ref25">25</xref>
        ], which
represents the dependence of surface friction on Mach numbers,
Reynolds numbers and the temperature factor:

= 0,085 
−0,29+0,01lg   ̅ 0,39 ̅ 0,2.
      </p>
      <p />
      <p>Here   , Rew is the coefficient of surface friction and the
Reynolds
number calculated
with
reference to the
wall
temperature, Te is the temperature at the outer boundary of the
boundary layer, and Twe is the temperature factor.</p>
      <p>
        The advent of computer technology allowed solving the
problems of mathematical modeling of currents, which sharply
reduced the need for large-scale physical experiments. However,
in the problems of mathematical modeling, the main tendency of
carrying out series of calculations with the variation of the
defining parameters of the problem also remained. The main goal
was the same – to determine the conditions for the appearance of
a physical phenomenon when the external conditions of the
problem are varied. An example of such approach is described in
article [
        <xref ref-type="bibr" rid="ref8">8</xref>
        ], which presents a series of numerical experiments on
the flow of a backward ledge by a viscous gas flow. As a result
of the experiments, a generalized formula is obtained that
represents the characteristic time of the establishment of the flow
as a function of the Mach and Reynolds numbers of the external
flow.
      </p>
      <p>Before the advent of parallel computing technologies, such
calculations were difficult to implement and they were quite rare.</p>
      <sec id="sec-1-1">
        <title>However, they were the prototype</title>
        <p>of the generalized
computational experiment.</p>
        <p>The generalized computational experiment is based on
solving direct and inverse problems of mathematical modeling.
These tasks can be considered in a parametric and optimization
setting. Problem solving is carried out using parallel technologies
multitasking</p>
        <p>mode. Numerical solutions are volumes of
multidimensional data. To process and analyze this data, it is
necessary to apply the methods of Data Analysis and Visual
Analytics. The construction of a generalized computational
experiment makes it possible to obtain a solution not for one,
separately taken problem, but for a whole class of problems. The
class of problems is determined in the ranges of variation of the
defining parameters of the problem, such as characteristic
numbers (Mach,</p>
      </sec>
      <sec id="sec-1-2">
        <title>Reynolds,</title>
        <p>
          Strouhal
numbers,
etc.)
and
geometric characteristics. In a practical sense, this makes it
possible to reveal hidden dependences of valuable functionals on
the determining parameters of the problem, similar to the above
formulas. This work continues a series of works devoted to the
development and implementation of a generalized computational
experiment for various classes of computational aerodynamics
problems [
          <xref ref-type="bibr" rid="ref10 ref11 ref12 ref13 ref14 ref15 ref16 ref17 ref2 ref3 ref9">2, 3, 9-17</xref>
          ].
        </p>
        <p>Despite the fact that there are very few works devoted to the
development of a generalized computational experiment, the
development of such experiments is gradually taking place in
many areas. First of all, tools are being developed to implement
such experiments in</p>
        <p>
          many software packages for solving
optimization problems. Here we can cite as an example the work
[
          <xref ref-type="bibr" rid="ref22">22</xref>
          ], where algorithms are implemented that allow a generalized
computational experiment in such fields as seismic exploration,
plasma physics and turbid media optics, solving fundamental and
applied problems of studying magnetic materials and creating
spintronics devices, simulation of field development for the oil
reservoir that contains kerogen with in-situ combustion taken
into account, simulation of poroelastic medium problems and
hydraulic fracture problems.
        </p>
      </sec>
    </sec>
    <sec id="sec-2">
      <title>2. Prerequisites for the creation of a generalized computational experiment</title>
      <p>The development of technologies and software tools for
constructing a generalized computational experiment occurs as
the modern development of mathematical methods and
highperformance computing tools. Two main reasons should be
pointed out as the main factors determining the possibility of
efficiently constructing a generalized computational experiment.</p>
      <p>The first of these is the emergence of high-performance
computing clusters and parallel technologies. It is generally
accepted that parallel technologies provide a) the ability of fast
computing and b) the ability to use detailed computational grids.
However, parallel technologies also provide researchers with
another crucial opportunity. This is an opportunity of parallel
calculation of the same problem
with different input data in
multitask mode. From the point of view of the author, this
possibility is not yet fully appreciated. This possibility allows
one to effectively solve parametric and optimization problems
and construct a generalized computational experiment.</p>
      <p>The second reason is the intensive development of methods
and</p>
      <p>approaches
multidimensional
computational</p>
    </sec>
    <sec id="sec-3">
      <title>3. Generalized numerical experiment</title>
      <p>
        A generalized numerical experiment involves splitting each
of the defining parameters of a problem within a certain range. A
grid
decomposition is formed for some
multidimensional
parallelepiped composed of the defining parameters of the
considered problem of gas dynamics. For each point of this grid,
the problem is calculated in the space of the determining
parameters. According to [
        <xref ref-type="bibr" rid="ref13 ref9">9, 13</xref>
        ], this can be written as follows.
      </p>
      <p>Suppose that there is a reliable numerical method for solving
two-dimensional and three-dimensional nonstationary problems
of computational gas dynamics. Then we can obtain a numerical
solution  ( ,  ,  ,  ,  1, … ,   ) for any point in the space of a
computational domain, where x, y, z are the spatial coordinates,
t is the time,  1, … ,</p>
      <p>are the defining parameters of the
problem. As defining parameters of the problem, we will keep in
mind the characteristic numbers describing the properties of the
flow under consideration, such as the Mach numbers, Reynolds,</p>
      <sec id="sec-3-1">
        <title>Prandtl,</title>
        <p>Strouhal,
etc.,
and
the
characteristic
geometric
parameters. Each of the characteristic parameters is limited in a
certain range
 
≤   ≤  

,  = 1, … ,  .</p>
        <p>We divide each of the parameters   into k-1 parts, so we
obtain for each parameter a partition consisting of k points. The
volume of an N-dimensional space formed by a set of defining
parameters   is filled with a set of   points.</p>
        <p>Denoting the point from the given set, as ( 1∗, … ,  ∗ ), we
arrive at the fact that for each point of the collection it is
necessary to obtain a numerical solution of the gas-dynamic
problem  (,  ,  ,  ,  ,  1∗, … ,  ∗ ).</p>
        <p>It is easy to see that this will require solving   gasdynamic
problems, which is impossible
without the use of parallel
calculations in a multitask mode. In practice, the number N
usually does not exceed 5, which corresponds to the computing
capabilities at the current time.</p>
        <p>
          It should also be noted that we formulated the classical
problem of parametric study. Parametric numerical studies allow
one to obtain a solution not for one particular mathematical
modeling problem, but for a class of problems defined in a
multidimensional space of defining parameters. Also, such a
formal formulation allows numerical study of optimization
analysis problems, when the inverse problem is solved at each
point of the grid partition of the multidimensional space of the
determining parameters. Both types of similar problems are
considered in a series of papers [
          <xref ref-type="bibr" rid="ref10 ref11 ref12 ref13 ref14 ref9">9-14</xref>
          ].
        </p>
        <p>
          The only way to effectively carry out a generalized numerical
experiment is applying of parallel computations. The problem of
the optimal and effective way of parallelization was thoroughly
discussed in the papers [
          <xref ref-type="bibr" rid="ref11 ref12">11, 12</xref>
          ]. There were considered parts of
the whole algorithm for parameter optimization and analysis. For
these parts the main criterion of applicability for parallelizing is
independence of specific numerical method. From this point of
view the most perspective way for parallelizing is applying the
approach of multitask parallelism using the principle “one task –
one process”. Due to minimal quantity of internal exchanges
between the processes we are able to create an effective practical
tool for generalized numerical experiment. We assume that k
processes are provided for parallel computation. The control
process P0 creates the grid in the multidimensional space of
determining parameters, then P0 forms tasks and sends the tasks
to others processes and to itself also. After task completion P0
collects the results and implements all procedures defined by
user, such as data processing and transformation. Due to the
absence of internal exchanges between the processes the
procedure of parallelizing amounts to creation
of control
interface for tasks distribution and data collecting in one
multidimensional array.
        </p>
        <p>
          There are some effective and easy ways to create such
interface for parallel computations. These
ways use such
computational technologies as MPI (Message Passing Interface)
[
          <xref ref-type="bibr" rid="ref26">26</xref>
          ] and DVM technology [
          <xref ref-type="bibr" rid="ref1 ref4 ref5 ref6 ref7">1, 4-7</xref>
          ]. DVM technology [
          <xref ref-type="bibr" rid="ref1 ref4 ref5 ref6 ref7">1, 4-7</xref>
          ]
was elaborated in Keldysh Institute of Applied Mathematics
RAS. DVM-system provides unified toolkit to develop parallel
programs of scientific-technical calculations in C and Fortran.
Unified parallel model is built in C and Fortran languages on the
base of the constructions, that are "transparent" for standard
compilers, that allows to have single version of the program for
sequential and parallel execution. This way of code parallelizing
allows one to save a lot of human resources for coding and
debugging. For both types of parallel technologies special control
interfaces for parameter optimization and analysis were designed
form is rather difficult to use, although its availability for further
purposes is necessary. In order to get useful information from a
calculated multidimensional data array, first of all we need to
reduce its dimension. By lowering its dimension, we are able to
apply the
methods
of visualization
and
visual analytics
[
          <xref ref-type="bibr" rid="ref23 ref24 ref27 ref28">23,24,27,28</xref>
          ] in order to understand the internal structure of the
array and to reveal hidden interdependencies between its
defining factors. The revealed dependencies can be further
approximated by geometric primitives in order to obtain a
generalizing dependence, which will represent the solution of
interest for a class of problems. Examples of the implementation
of this approach are presented in [
          <xref ref-type="bibr" rid="ref10 ref11 ref12 ref13 ref14 ref9">9-14</xref>
          ].
        </p>
        <p>
          Also, to reduce the dimension of a multidimensional array,
methods of mapping into embedded
manifolds of smaller
dimension are very effective [
          <xref ref-type="bibr" rid="ref20 ref21 ref29">20, 21, 29</xref>
          ]. Among them, the most
common method is the principal component method (PCA). The
essence of the method consists in the transition from the initial
coordinate
system
to
the
new
orthogonal basis in the
multidimensional space under consideration, whose axes are
oriented along the directions of maximum dispersion. The
possible scheme of working with an array in this case is the
approximation by primitives of the data array in the space of the
first three main components and the subsequent transition to the
initial space of the determining parameters.
of
numerical
        </p>
        <p>This section contains the examples of the generalized
numerical experiment application to some practical problems. It
is applied in some variations due to different aims for each class
of problems.</p>
        <p>
          The first example of generalized numerical experiment is
devoted to the problem of tuning the properties of hybrid
finitedifference schemes [
          <xref ref-type="bibr" rid="ref16">16</xref>
          ]. The paper [
          <xref ref-type="bibr" rid="ref16">16</xref>
          ] contains the description
of developed program tool Burgers2. This program tool is
intended for tuning and optimization of computational properties
for hybrid finite-difference schemes applied to Burgers equation.
One-dimensional model Burgers equation describes propagation
of disturbances for dissipative medium. The equation has exact
solution, so it is widely used for tuning-up of computational
tools. Described program tool is based on combining of
optimization problem solution and visual data presentation.
Visual presentations of maximal error surface and error function
are implemented as program tool features. Users have possibility
of creating hybrid finite-difference schemes and analyzing
computational properties for chosen grid template provided by
program tool. Visual presentation of optimization problem
solution allows finding of suitable weight coefficients for hybrid
finite-difference scheme under consideration. The user is able to
make simultaneous calculations varying weight coefficients in
the scheme and viscosity coefficient in Burgers equation. The
user can make the calculations simultaneously different sets of
weight coefficients in accordance with the concept of generalized
numerical experiment. Figure 1 presents the surface of absolute
error for one of the hybrid scheme variants. The negative data
area indicates where the oscillations occur.
        </p>
        <p>The following example is also devoted to improving the
computational properties of finite-difference schemes. The
problem of mathematical modelling of the flow in the far wake
behind the body is solved. In the general case, in a rectangular
computational domain, a viscous compressible heat-conducting
gas flow is considered, described by a complete system of
timedependent Navier-Stokes equations. At the input boundary, the
distributions of gas-dynamic parameters are given, obtained from
calculations of the flow around an axisymmetric body and a
portion of the track behind it. The main goal of the generalized
computational method was to thoroughly study the properties of
artificial viscosity incorporated in the hybrid difference scheme.
For this purpose, we studied the properties of the weight
coefficients of the hybrid scheme on the example of the problem
of flow in the far wake and determined the limitations for the
weight coefficients. In this task, the following defining
parameters were varied, such as the steps of the grid
decomposition in the x and y directions, the weighting
coefficients of the difference scheme, the Reynolds number of
the problem. As a result of the generalized computational
experiment, a limit surface was constructed for the dependence
of the weight coefficient on the other determining parameters of
the problem. An example of the limiting surface is presented in
figure 2. When choosing the value of the weighting factor below
the surface, in the numerical solution, non-physical oscillations
arise, which can lead to the collapse of the solution. Such
surfaces are constructed for non-viscous and viscous flow. In the
case of viscous flow, laminar and turbulent regimes are
considered.</p>
        <p>
          The next example considers the problem of the evaluation of
the accuracy for different numerical methods. The problem of
inviscid compressible flow around a cone at zero angle of attack
is used as a base one. The results obtained with the help of
various OpenFOAM solvers are compared with the known
numerical solution of the problem with the variation of cone
angle and flow velocity [
          <xref ref-type="bibr" rid="ref17">17</xref>
          ]. Cone angle β changes from 10° to
35° in steps of 5°. Mach number varies from 2 to 7. For
comparison, four solvers were selected from the OpenFOAM
software package: RhoCentralFoam, SonicFoam,
RhoPimpleFoam, RhoPimpleFoam. The results of such kind of
numericsl experiment were presented as errors in the form of an
analog of the L2 norm for all solvers. Figure 3 illustrates the
results in a form of a change in deviation from the exact solution
for pressure depending on the cone angle and the velocity for the
solver rhoCentralFoam. Such changes were obtained for all
solvers.
        </p>
        <p>Figure 3 shows a multidimensional dataset for pressure
obtained as a result of parametric calculations in the space of the
first three principal components. Yellow shows the results for
rhoCentralFoam solver, red for pisoCentralFoam, green for
sonicFoam and blue for rhoPimpleFoam.</p>
        <p>Figure 3 shows that the errors for rhoCentralFoam and for
pisoCentralFoam can be roughly approximated by a plane
reflecting the dependence of the error on the Mach number and
cone angle. The results for sonicFoam and especially for
rhoPimpleFoam are significantly separated from the results for
the first two solvers due to their particular numerical
characteristics. This methodical research can serve as a basis for
selecting the OpenFoam solver for calculating the inviscid
supersonic flow around the elongated bodies of rotation. The
results of solvers comparison can also be useful for developers
of OpenFoam software content. The results obtained made it
possible to get a general idea of the calculation errors for all
solvers.</p>
        <p>Fig. 3. Errors for different OpenFOAM solvers in the space of
principal components.</p>
        <p>
          The next example of application of general numerical
experiment considers optimization problem. The example
presents a search for optimal shape of three-dimensional blade
assembly intended for power plant [
          <xref ref-type="bibr" rid="ref3">3</xref>
          ]. This experiment is based
on developed computational technology for the computation of
power loads on the 3D blade assembly of a power plant in a wind
flow. The calculation for various combinations of the key
geometric parameters of the assembly using parallel
computations makes it possible to find the optimal shape of the
assembly with respect to its power characteristics. A virtual
experimental facility for simulating the flow around the power
plant based on the solution of the Navier–Stokes equations was
created. Computations aimed at determining the optimal shape of
the blade assembly taking into account constraints on its design
were carried out, and the results were thoroughly analyzed using
the proposed optimization procedure. The solution of the
optimization problem is based on the parameterization of the
design using three key parameters. On the discrete set of values
of these parameters, the maximums of two objective functions—
the magnitude of the total aerodynamic force and the magnitude
of the rotation torque—determining the lift-to-drag ratio of the
power plant are found. Figure 4 presents the shape of 3D blade
assembly and pressure distribution on its surface.
        </p>
        <p>Fig. 4. Pressure distribution on the surface of 3D blade
assembly.</p>
        <p>The given examples show the applicability of the presented
approach for a wide range of practical applications; therefore, the
construction of a generalized computational experiment can be
considered as a rather universal and useful approach.</p>
      </sec>
    </sec>
    <sec id="sec-4">
      <title>5. The perspectives computational experiment of generalized</title>
      <p>The above examples show the possibility of constructing a
generalized computational experiment for various problems in
the field of computational gas dynamics. Parametric studies can
serve as such an experiment, where the basis is the ability to solve
the direct problem of mathematical modeling. An example of an
optimization analysis problem is given, where the generalized
computational experiment is based on solving the inverse
problem in an optimization statement. Thus, to create such an
experiment is quite realistic for almost any mathematical
modeling problem.</p>
      <p>
        A separate area of application of a generalized computational
experiment can be a comparative assessment of the accuracy of
numerical methods. Similar attempts are presented in [
        <xref ref-type="bibr" rid="ref15 ref2">2, 15</xref>
        ]
devoted to the problem of accuracy estimation with the help of
the ensemble of solutions. According to [
        <xref ref-type="bibr" rid="ref15 ref2">2, 15</xref>
        ], if a researcher is
able to calculate the same problem using several numerical
methods with different computational properties, in particular,
different approximation orders, then in some cases one can
estimate the neighborhood of the approximate solution
containing the exact solution (exact solution enclosure). If an
ensemble of numerical solutions can be divided into clusters of
“accurate” and “inaccurate” solutions, then the error ranking of
values can be performed using an a posteriori analysis of the
distances between the numerical solutions. This can serve as a
computational proof of the existence of an exact solution in the
case of nonlinear problems. This approach can be considered as
perspective. Nevertheless, it has evident draw-back. For using of
this approach one should have a set of solvers with different
accu-racy order.
      </p>
    </sec>
    <sec id="sec-5">
      <title>6. Conclusions</title>
      <p>The concept of generalized numerical experiment presented
in the article has a wide range of possible applications. For the
problems of computational fluid dynamics such an approach
makes it possible to obtain a solution not only for one, separately
taken, problem, but for a whole class of problems defined in a
certain range of the complex of determining parameters. Practical
implementation of the approach becomes possible with the use
of parallel calculations in multitask mode. The results of
calculations are multidimensional volumes of data that can be
processed using data analysis tools and visual analytics. The
application of these methods reveals hidden interdependencies
between the determining parameters of the class of problems.
Also, these methods allow in many cases to build the dependence
of the valuable functional on the determining parameters, which
makes it possible to further approximate it with geometric
primitives and present it in an analytical form. The examples
presented in the article illustrate generalized computational
experiments for various types of tasks, such as improving the
computational properties of difference schemes, finding the
optimal body shape in a stream, a comparative assessment of the
accuracy of algorithms.</p>
    </sec>
    <sec id="sec-6">
      <title>7. Acknowledgments</title>
    </sec>
    <sec id="sec-7">
      <title>8. References</title>
      <p>This work was supported by RFBR grant 19-01-00402.</p>
    </sec>
  </body>
  <back>
    <ref-list>
      <ref id="ref1">
        <mixed-citation>
          [1]
          <string-name>
            <surname>Aleksahin</surname>
            <given-names>V. F.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Bakhtin</surname>
            <given-names>V. A.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Zhukova</surname>
            <given-names>O. F.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Kolganov</surname>
            <given-names>A. S.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Krukov</surname>
            <given-names>V. A.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Ostrovskaya</surname>
            <given-names>I. P.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Podderugina</surname>
            <given-names>N. V.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Pritula</surname>
            <given-names>M. N.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Savitskaya</surname>
            <given-names>O. A.</given-names>
          </string-name>
          “
          <article-title>Parallelization of NAS parallel benchmarks for Intel Xeon Phi coprocessor in Fortran-DVMH language”</article-title>
          , Vestn. YuUrGU. Ser. Vych. Matem. Inform.,
          <volume>4</volume>
          :
          <issue>4</issue>
          (
          <issue>2015</issue>
          ),
          <fpage>48</fpage>
          -
          <lpage>63</lpage>
        </mixed-citation>
      </ref>
      <ref id="ref2">
        <mixed-citation>
          [2]
          <string-name>
            <surname>Alekseev</surname>
            <given-names>A.K.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Bondarev</surname>
            <given-names>A.E.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Kuvshinnikov</surname>
            <given-names>A.E.</given-names>
          </string-name>
          <article-title>Verification on the</article-title>
          <source>Ensemble of Independent Numerical Solutions // Lecture Notes in Computer Science</source>
          , Vol.
          <volume>11540</volume>
          , pp.
          <fpage>315</fpage>
          -
          <lpage>324</lpage>
          ,
          <year>2019</year>
          . DOI:
          <volume>10</volume>
          .1007/978-3-
          <fpage>030</fpage>
          - 22750-0_
          <fpage>25</fpage>
        </mixed-citation>
      </ref>
      <ref id="ref3">
        <mixed-citation>
          [3]
          <string-name>
            <surname>Andreev</surname>
            <given-names>S.V.</given-names>
          </string-name>
          et al.:
          <article-title>A Computational Technology for Constructing the Optimal Shape of a Power Plant Blade Assembly Taking into Account Structural Constraints</article-title>
          .
          <source>Programming and Computer Software</source>
          ,
          <volume>43</volume>
          (
          <issue>6</issue>
          ),
          <fpage>345</fpage>
          -
          <lpage>352</lpage>
          (
          <year>2017</year>
          ). DOI:
          <volume>10</volume>
          .1134/S0361768817060020.
        </mixed-citation>
      </ref>
      <ref id="ref4">
        <mixed-citation>
          [4]
          <string-name>
            <surname>Bakhtin</surname>
            <given-names>V.A.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Chetverushkin</surname>
            <given-names>B.N.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Kryukov</surname>
            <given-names>V.A.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Shil'</surname>
            nikov
            <given-names>E.V.</given-names>
          </string-name>
          , “
          <article-title>Extension of the DVM parallel programming model for clusters with heterogeneous nodes”</article-title>
          ,
          <source>Doklady Mathematics</source>
          ,
          <volume>84</volume>
          (
          <issue>3</issue>
          ),
          <fpage>879</fpage>
          -
          <lpage>881</lpage>
          (
          <year>2013</year>
          ).
        </mixed-citation>
      </ref>
      <ref id="ref5">
        <mixed-citation>
          [5]
          <string-name>
            <surname>Bakhtin</surname>
            <given-names>V. A.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Kolganov</surname>
            <given-names>A. S.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Krukov</surname>
            <given-names>V. A.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Podderyugina</surname>
            <given-names>N. V.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Pritula</surname>
            <given-names>M. N.</given-names>
          </string-name>
          “
          <article-title>Mapping DVMHprograms with regular dependencies onto clusters with GPU”, Vestn</article-title>
          . YuUrGU. Ser. Vych. Matem. Inform.,
          <volume>2</volume>
          :
          <issue>4</issue>
          (
          <issue>2013</issue>
          ),
          <fpage>44</fpage>
          -
          <lpage>56</lpage>
        </mixed-citation>
      </ref>
      <ref id="ref6">
        <mixed-citation>
          [6]
          <string-name>
            <surname>Bakhtin</surname>
            <given-names>V. A.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Klinov</surname>
            <given-names>M. S.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Kolganov</surname>
            <given-names>A. S.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Krukov</surname>
            <given-names>V. A.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Podderyugina</surname>
            <given-names>N. V.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Pritula</surname>
            <given-names>M. N.</given-names>
          </string-name>
          “
          <article-title>Automatic mapping of Fortran programs onto clusters with graphics processing units”, Vestn</article-title>
          . YuUrGU. Ser. Vych. Matem. Inform.,
          <volume>3</volume>
          :
          <issue>3</issue>
          (
          <issue>2014</issue>
          ),
          <fpage>86</fpage>
          -
          <lpage>96</lpage>
        </mixed-citation>
      </ref>
      <ref id="ref7">
        <mixed-citation>
          [7]
          <string-name>
            <surname>Bakhtin</surname>
            <given-names>V. A.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Zaharov</surname>
            <given-names>D. A.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Kolganov</surname>
            <given-names>A. S.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Krukov</surname>
            <given-names>V. A.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Podderyugina</surname>
            <given-names>N. V.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Pritula</surname>
            <given-names>M. N.</given-names>
          </string-name>
          “
          <article-title>Development of parallel applications using DVM-system”, Vestn</article-title>
          . YuUrGU. Ser. Vych. Matem. Inform.,
          <volume>8</volume>
          :
          <issue>1</issue>
          (
          <issue>2019</issue>
          ),
          <fpage>89</fpage>
          -
          <lpage>106</lpage>
        </mixed-citation>
      </ref>
      <ref id="ref8">
        <mixed-citation>
          [8]
          <string-name>
            <surname>Bondarev</surname>
            <given-names>A.E.</given-names>
          </string-name>
          “
          <article-title>Effect of supersonic stream parameters on the characteristic time of transient step flow”</article-title>
          .
          <source>1989 Izvestiya AN SSSR, Mekhanika zhidkosti I gaza 4</source>
          p
          <fpage>137</fpage>
          -
          <lpage>40</lpage>
          [In Russian]
        </mixed-citation>
      </ref>
      <ref id="ref9">
        <mixed-citation>
          [9]
          <string-name>
            <surname>Bondarev</surname>
            <given-names>A.E.</given-names>
          </string-name>
          “
          <article-title>Analysis of Space-Time Flow Structures by Optimization</article-title>
          and Visualization Methods” // Transactions on Computational Science XIX, LNCS
          <volume>7870</volume>
          ,
          <fpage>158</fpage>
          -
          <lpage>168</lpage>
          (
          <year>2013</year>
          ).
        </mixed-citation>
      </ref>
      <ref id="ref10">
        <mixed-citation>
          [10]
          <string-name>
            <surname>Bondarev</surname>
            <given-names>A.E.</given-names>
          </string-name>
          and
          <string-name>
            <surname>Galaktionov</surname>
            <given-names>V.A.</given-names>
          </string-name>
          “
          <article-title>Parametric Optimizing Analysis of Unsteady Structures and</article-title>
          Visualization of Multidimensional Data” // International Journal of Modeling, Simulation and
          <string-name>
            <given-names>Scientific</given-names>
            <surname>Computing</surname>
          </string-name>
          , V.
          <volume>04</volume>
          ,
          <issue>supp01</issue>
          , (
          <year>2013</year>
          ) DOI 10.1142/S1793962313410043.
        </mixed-citation>
      </ref>
      <ref id="ref11">
        <mixed-citation>
          [11]
          <string-name>
            <surname>Bondarev</surname>
            <given-names>A.E.</given-names>
          </string-name>
          and
          <string-name>
            <surname>Galaktionov</surname>
            <given-names>V.A.</given-names>
          </string-name>
          “
          <article-title>Analysis of SpaceTime Structures Appearance for Non-</article-title>
          Stationary CFD Problems” // Procedia Computer Science,
          <volume>51</volume>
          ,
          <fpage>1801</fpage>
          -
          <lpage>1810</lpage>
          (
          <year>2015</year>
          ).
        </mixed-citation>
      </ref>
      <ref id="ref12">
        <mixed-citation>
          [12]
          <string-name>
            <surname>Bondarev</surname>
            <given-names>A.E.</given-names>
          </string-name>
          “
          <article-title>Design and analysis of multidimensional parametrical solutions for time-dependent CFD problems</article-title>
          ” // Mathematica Montisnigri, XXXIII,
          <fpage>58</fpage>
          -
          <lpage>68</lpage>
          (
          <year>2015</year>
          ).
        </mixed-citation>
      </ref>
      <ref id="ref13">
        <mixed-citation>
          [13]
          <string-name>
            <surname>Bondarev</surname>
            <given-names>A.E.</given-names>
          </string-name>
          <article-title>On the Construction of the Generalized Numerical Experiment</article-title>
          in Fluid Dynamics // Mathematica Montisnigri, Vol. XLII,
          <year>2018</year>
          , p.
          <fpage>52</fpage>
          -
          <lpage>64</lpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref14">
        <mixed-citation>
          [14]
          <string-name>
            <surname>Bondarev</surname>
            <given-names>A.E.</given-names>
          </string-name>
          <article-title>On visualization problems in a generalized computational experiment (</article-title>
          <year>2019</year>
          ).
          <source>Scientific Visualization</source>
          <volume>11</volume>
          .2:
          <fpage>156</fpage>
          -
          <lpage>162</lpage>
          , DOI: 10.26583/sv.11.2.
          <fpage>12</fpage>
        </mixed-citation>
      </ref>
      <ref id="ref15">
        <mixed-citation>
          [15]
          <string-name>
            <surname>Bondarev</surname>
            <given-names>A.E. .</given-names>
          </string-name>
          <article-title>On the Estimation of the Accuracy of Numerical Solutions in</article-title>
          CFD Problems // Lecture Notes in Computer Science, Vol.
          <volume>11540</volume>
          , pp.
          <fpage>325</fpage>
          -
          <lpage>333</lpage>
          ,
          <year>2019</year>
          . DOI:
          <volume>10</volume>
          .1007/978-3-
          <fpage>030</fpage>
          -22750-0_
          <fpage>26</fpage>
        </mixed-citation>
      </ref>
      <ref id="ref16">
        <mixed-citation>
          [16]
          <string-name>
            <surname>Bondarev</surname>
            ,
            <given-names>A.E.</given-names>
          </string-name>
          et al.:
          <article-title>Design of program tool BURGERS2 for hybrid finite-difference schemes optimization and visualization</article-title>
          .
          <source>Scientific Visualization</source>
          <volume>5</volume>
          (
          <issue>1</issue>
          ),
          <fpage>26</fpage>
          -
          <lpage>37</lpage>
          (
          <year>2013</year>
          ).
        </mixed-citation>
      </ref>
      <ref id="ref17">
        <mixed-citation>
          [17]
          <string-name>
            <surname>Bondarev</surname>
            <given-names>A.E.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Kuvshinnikov</surname>
            <given-names>A.E.</given-names>
          </string-name>
          <article-title>Analysis of the Accuracy of OpenFOAM Solvers for the Problem of Supersonic Flow Around</article-title>
          a Cone // Lecture Notes in Computer Science 10862, pp.
          <fpage>221</fpage>
          -
          <lpage>230</lpage>
          ,
          <year>2018</year>
          . https://doi.org/10.1007/978-3-
          <fpage>319</fpage>
          -93713-7_
          <fpage>18</fpage>
        </mixed-citation>
      </ref>
      <ref id="ref18">
        <mixed-citation>
          [18]
          <string-name>
            <surname>Bondarev</surname>
            <given-names>E.N.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Dubasov</surname>
            <given-names>V.T.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Ryzhov</surname>
            <given-names>Y.A.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Svirschevsky</surname>
            <given-names>S.B.</given-names>
          </string-name>
          and
          <string-name>
            <surname>Semenchikov</surname>
            <given-names>N.V.</given-names>
          </string-name>
          <year>1993</year>
          Aerigidromeckanika. (Moscow: Mashinostroenie) p
          <fpage>608</fpage>
          [In Russian]
        </mixed-citation>
      </ref>
      <ref id="ref19">
        <mixed-citation>
          [19]
          <string-name>
            <surname>DVM-system Available</surname>
          </string-name>
          at: http://dvm-system.
          <source>org/ (accessed 25 July</source>
          <year>2019</year>
          )
        </mixed-citation>
      </ref>
      <ref id="ref20">
        <mixed-citation>
          [20]
          <string-name>
            <surname>Gorban</surname>
            <given-names>A</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Kegl</surname>
            <given-names>B</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Wunsch</surname>
            <given-names>D</given-names>
          </string-name>
          and
          <article-title>Zinovyev A 2007 Principal Manifolds for Data Visualisation</article-title>
          and Dimension
          <string-name>
            <surname>Reduction</surname>
          </string-name>
          (Berlin - Heidelberg - New York: Springer)
        </mixed-citation>
      </ref>
      <ref id="ref21">
        <mixed-citation>
          [21]
          <string-name>
            <surname>Gorban</surname>
            <given-names>A.N.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Zinovyev</surname>
            <given-names>A.</given-names>
          </string-name>
          “
          <article-title>Principal manifolds and graphs in practice: from molecular biology to dynamical systems”</article-title>
          ,
          <source>International Journal of Neural Systems</source>
          , Vol.
          <volume>20</volume>
          , No.
          <volume>3</volume>
          (
          <year>2010</year>
          ). Pp.
          <volume>219</volume>
          -
          <fpage>232</fpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref22">
        <mixed-citation>
          [22]
          <string-name>
            <surname>Ivanov</surname>
            <given-names>A. V.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Khilkov</surname>
            <given-names>S. A.</given-names>
          </string-name>
          “
          <article-title>Aiwlib library as the instrument for creating numerical modeling applications”</article-title>
          ,
          <source>Scientific Visualization</source>
          ,
          <volume>10</volume>
          (
          <issue>1</issue>
          ),
          <fpage>110</fpage>
          -
          <lpage>127</lpage>
          (
          <year>2018</year>
          ).
          <source>DOI: 10.26583/sv.10.1.09</source>
        </mixed-citation>
      </ref>
      <ref id="ref23">
        <mixed-citation>
          [23]
          <string-name>
            <surname>Keim</surname>
            <given-names>D.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Kohlhammer</surname>
            <given-names>J.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Ellis</surname>
            <given-names>G.</given-names>
          </string-name>
          and Mansmann F. (Eds.)
          <article-title>Mastering the Information Age - Solving Problems with Visual Analytics</article-title>
          , Eurographics Association,
          <year>2010</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref24">
        <mixed-citation>
          [24]
          <string-name>
            <surname>Kielman</surname>
            ,
            <given-names>J.</given-names>
          </string-name>
          and Thomas,
          <string-name>
            <surname>J.</surname>
          </string-name>
          (Guest Eds.) (
          <year>2009</year>
          ).
          <article-title>Special Issue: Foundations and Frontiers of Visual Analytics / Information Visualization</article-title>
          , Volume
          <volume>8</volume>
          ,
          <string-name>
            <surname>Number</surname>
            <given-names>4</given-names>
          </string-name>
          , p.
          <fpage>239</fpage>
          -
          <lpage>314</lpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref25">
        <mixed-citation>
          [25]
          <string-name>
            <surname>Kozlov</surname>
            <given-names>L.V.</given-names>
          </string-name>
          “
          <article-title>Experimental investigation of surface friction on a flat plate in a supersonic flow in the presence of heat transfer”</article-title>
          .
          <source>1963 Izvestiya AN SSSR, Mekhanika i mashinostroenie</source>
          <volume>2</volume>
          <fpage>p11</fpage>
          -
          <lpage>20</lpage>
          [In Russian]
        </mixed-citation>
      </ref>
      <ref id="ref26">
        <mixed-citation>
          [26]
          <string-name>
            <surname>Pacheco</surname>
            <given-names>P.</given-names>
          </string-name>
          ,
          <article-title>Programming Parallel with MPI</article-title>
          . San Francisco, CA: Morgan Kaufmann,
          <year>1997</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref27">
        <mixed-citation>
          [27]
          <string-name>
            <surname>Thomas</surname>
            <given-names>J.</given-names>
          </string-name>
          and
          <string-name>
            <surname>Cook</surname>
            <given-names>K.</given-names>
          </string-name>
          <year>2005</year>
          <article-title>Illuminating the Path: Research and Development Agenda for Visual Analytics</article-title>
          (IEEE-Press)
        </mixed-citation>
      </ref>
      <ref id="ref28">
        <mixed-citation>
          [28]
          <string-name>
            <surname>Wong</surname>
            <given-names>P. C.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Thomas</surname>
            <given-names>J</given-names>
          </string-name>
          .
          <source>Visual Analytics // IEEE Computer Graphics and Applications</source>
          .
          <year>2004</year>
          . V. 24,
          <string-name>
            <surname>N.</surname>
          </string-name>
          <year>5</year>
          . - P.
          <fpage>20</fpage>
          -
          <lpage>21</lpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref29">
        <mixed-citation>
          [29]
          <string-name>
            <surname>Zinovyev</surname>
            <given-names>A 2000</given-names>
          </string-name>
          <article-title>Vizualization of multidimensional data (Krasnoyarsk: Krasnoyarsk Univ</article-title>
          .)
        </mixed-citation>
      </ref>
    </ref-list>
  </back>
</article>