Generalized computational experiment in the problems of numerical methods verification A.K. Alekseev1, A.E. Bondarev1, A.E. Kuvshinnikov1 aleksey.k.alekseev@gmail.com | bond@keldysh.ru | kuvsh90@yandex.com 1 Keldysh Institute of Applied Mathematics RAS, Moscow, Russia; This work is devoted to the application of a generalized computational experiment for a comparative assessment of numerical methods accuracy. The construction of a generalized computational experiment is based on the simultaneous solution using parallel computations in a multitasking mode of a basic problem with different input parameters, obtaining results in the form of multidimensional data volumes and their visual analysis. This approach can be effective in problems of verification of numerical methods. A comparative assessment of the accuracy for solvers of the open software package OpenFOAM is carried out. The classic inviscid problem of oblique shock wave is used as a basic task. Variations of the key parameters of the problem — the Mach number and angle of attack — are considered. An example of constructing error surfaces is given when the solvers of the OpenFOAM software package are compared. The concept of an error index is introduced as an integral characteristic of deviations from the exact solution for each solver in the class of problems under consideration. The surfaces of deviations from the exact solution in the L2 norm, constructed for each solver, together with the calculated error indices, make it possible to obtain a complete picture of the accuracy of the solvers under consideration for the class of problems defined by the ranges of variation of the Mach number and angle of attack. Keywords: generalized computational experiment, numerical methods, verification problems, error index The concept, basic methods and approaches of a 1. Introduction generalized computational experiment, as well as a The tasks of numerical methods verification have number of software tools for its implementation were always been of paramount importance throughout the developed in Keldysh Institute of Applied Mathematics history of the development of computational mathematics. RAS. The main aspects of constructing a generalized Today, verification problems are of particular importance computational experiment and examples of its in the problems of computational gas dynamics. In implementation are described in detail in [4–11]. mathematical modeling of practical problems in This work is devoted to the application of a generalized aerodynamics, the researcher must be sure of the accuracy computational experiment for a comparative assessment of of the numerical method used. numerical methods accuracy. A comparative assessment of numerical methods 2. Generalized computational experiment accuracy is of particular importance at present. This is due to the wide distribution of software packages, both open The emergence of the concept of a generalized and commercial, allowing to solve a wide range of computing experiment is associated with the development problems. As a rule, a large number of numerical methods of high-performance computing clusters and parallel implemented in various solvers are integrated into such technologies. In problems of computational aerodynamics, packages. When solving practical problems, it is not easy parallel technologies usually provide the ability to quickly for the user to choose the most suitable solver for the calculate on detailed grids. However, parallel technologies studied class of problems. provide us with another important opportunity. This is the The relevance of verification tasks is confirmed by the ability to simultaneously calculate on different nodes the introduction in 2018 of the Federal Standard for the same task with different input data. As a rule, such a numerical simulation of supersonic inviscid gas flows and calculation is performed in multitasking mode. software verification [1]. Similar foreign standards have This opens up the possibility of implementing a been around for quite some time [2,3]. Such standards will generalized computational experiment. The key determine the direction of research in this area over the parameters of the problem under consideration are divided next decade. However, all these methodological in certain ranges with a certain step, forming a grid documents are focused on verification in relation to a partition of a multidimensional box in a multidimensional specific task with fixed values of key parameters. space of key parameters. The basic problem is solved At the present stage, researchers need more using parallel technologies at each point of the grid comprehensive estimates of the accuracy of numerical partition. The obtained results represent multidimensional methods. For example, in assessing accuracy, not for a data volumes. Processing, analysis and visual presentation single task, but for a class of tasks. By a class of tasks is of this data is carried out using methods of visual analytics meant a basic task considered in the ranges of change in and scientific visualization. This computing technology is the set of key parameters. Such parameters in the most general description of a generalized computing computational aerodynamics can serve characteristic experiment. numbers that determine flow velocity, viscosity, Obviously, such a concept can be applied to a wide thermophysical properties of the medium, geometric range of tasks. This range includes parametric studies, parameters, etc. An opportunity of getting solution for a optimization problems. A generalized computational class of problems is provided by the construction of a experiment is an effective tool for solving inverse generalized computational experiment. problems. Copyright © 2020 for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0) A large number of different applications of a cases, approaches to lowering the dimension of the generalized computational experiment are described in multidimensional space of key parameters under detail in [4-14]. The concept of a generalized consideration are useful. computational experiment was applied to a wide range of A separate problem is the estimation of the accuracy of both model and practical problems. numerical methods in the absence of a reference solution. These tasks include the analysis of the interaction of a Here, to assess the accuracy, foreign standards recommend viscous supersonic flow with a jet barrier, the flows in the to apply Richardson method [2,3]. However, for practical wake of the body, the problems of the interaction of jets, problems of computational aerodynamics this is very the problem of flowing around a cone at an angle of attack, difficult due to the enormous computational costs. The the problem of oblique shock waves, and many others. The computational costs are due to the fact that the approach to constructing a generalized computational implementation of Richardson method requires multiple experiment was applied to the problem of finding the calculations with a decrease in the step of the spatial grid optimal three-dimensional shape of the blades assembly decomposition. One of the alternatives in this case is the for a power plant in terms of power loads. estimation of accuracy on the ensemble of solutions. The Also, this approach was applied to the problems of ensemble of solutions obtained by various numerical verification of numerical methods. A comprehensive methods on the same grid allows us to estimate the comparative analysis of a number of solvers of the location of the exact solution and to divide the obtained OpenFOAM open software package [15] was carried out numerical solutions into clusters of different levels of in [12–14, 20, 21]. As basic tasks, we used problems that accuracy. This direction is being actively developed at have a reference solution (exact solution or experimental present and is presented in [16-19]. A natural drawback of data). These tasks include the problem of a supersonic this approach is the need for researcher to have at his inviscid flow around a cone at an angle of attack and the disposal a certain number of solvers that implement problem of an oblique shock wave formation. In both numerical methods with different computational cases, a class of problems was considered, formed by key properties. parameters variations of the problem in question. 4. The example of verification problem 3. The problems of numerical methods This section provides an example of constructing a verification generalized computational experiment for a comparative As already mentioned above, verification problems assessment of numerical methods accuracy. As an have been an important section throughout the history of example, we use the numerical results described in detail the development of computational mathematics and in the authors' works [20,21]. In these papers, a class of mathematical modeling. As a rule, a comparison of the computational gas dynamics problems is considered that numerical results was carried out with some reference describe the incidence of an inviscid supersonic gas flow solution, in the role of which the exact solution was used onto a flat plate at an angle of attack. if available or the available experimental data. With such an incidence, an oblique shock wave is If there is a reference solution, the accuracy of the formed. The Mach number and angle of attack are used as numerical method can be estimated for the solution key parameters. These values vary in certain ranges. This element or for the entire calculation domain. For example, problem has an exact solution. With the exact solution, a for problems containing discontinuities (shock waves), comparison is made at each point of the calculation previously, the width of the “smearing” of the solution at domain, and for each combination of key parameters, an discontinuity was traditionally considered as a error is evaluated in the norm of L1 and L2. The results characteristic of the numerical method accuracy. In this obtained make it possible to construct an error surface as case, a comparison with the reference solution over the an error function of two key parameters of the problem. entire flow field was also applied. For an objective Carrying out similar calculations for several numerical assessment of numerical method accuracy, it seems methods implemented in the solvers of the open software appropriate and reliable to apply both approaches. In the package OpenFOAM, makes it possible to build several presence of a reference solution, the construction of a such surfaces on one drawing. This opens up the generalized computational experiment allows us to possibility of a deep and clear comparative analysis of the compare not only for one problem with fixed key accuracy of the studied numerical methods. The parameters, but also for problems in the entire field of construction of such a generalized computational variation of key parameters. experiment involves the creation of computational If the class of problems is determined by two key technology from solving a direct problem up to visual parameters, then for each numerical method involved in analysis of the results. One of the most expressive and the comparison, the dependence of the error on these visual forms of visualization is the construction of stereo parameters is constructed in the form of an error surface. animations of the results of numerical studies. A similar In the case of three key parameters, scientific visualization construction of stereo images for this task was carried out methods are used to analyze a three-dimensional figure and described in [22]. representing the dependence of the error on key Fig. 1 presents the results of constructing error surfaces parameters. In the case when the number of key for four OpenFOAM solvers with variations in the Mach parameters is more than three, then methods of visual number from 2 to 4 and variations in the angle of attack analytics should be used to analyze the results. In some from 6 to 20 degrees [21]. It should be noted that error surfaces for the class of problems for the comparative that the best accuracy in the class of problems is provided analysis of the accuracy of numerical methods were by the rCF and pCF solvers, for which the error surfaces constructed in [21] for the first time. Four error surfaces almost coincide. for OpenFOAM solvers are presented - rhoCentralFoam Thus, the construction of a generalized computational (rCF), pisoCentralFoam (pCF), sonicFoam (sF) and experiment allows us to conduct a full-fledged QGDFoam (QGDF). comparative accuracy assessment for four solvers of the These surfaces allow a thorough visual comparison of OpenFOAM software package in the class of problems. deviations from the exact solution in the class of problems The class of tasks in this particular case is determined by under consideration. It can be seen that all 4 surfaces the basic task (oblique shock wave) and the ranges of behave in a very similar way. The deviation from the exact variation of the key parameters of the problem - the Mach solution increases with the growth of key parameters - the number and angle of attack. angle of attack and the Mach number. Fig. 1 also shows Fig. 1. Image of the surface deviation from the exact solution for 4 OpenFOAM solvers with variation of the Mach number and angle of attack for the oblique shock wave [21] The image of error surfaces presented in Figure 1 gives the accuracy of numerical methods is considered. An a fairly clear idea of the comparative accuracy of example of constructing a generalized computational OpenFOAM solvers in the class of problems. However, experiment for a class of problems is described. The class for a more complete assessment, an integral characteristic of tasks is formed on the basis of the basic problem (the for each surface can be introduced. oblique shock wave) and variations of the determining We call this characteristic the Error Index (EI). parameters of the problem - the Mach number and angle The error index is defined as follows. Let i = 1, M and of attack. An example of constructing error surfaces is j = 1, N be the grid partitions of key parameters, and Aij - given. The concept of a numerical method error index for the deviation from the exact solution at each point of the a class of problems is introduced. grid partition. Then the error index is defined as: The construction of a generalized computational 𝐸𝐸𝐸𝐸 = � 𝐴𝐴𝑖𝑖𝑖𝑖 �(𝑀𝑀 ∗ 𝑁𝑁). experiment can serve as an effective tool for verification problems. 𝑖𝑖𝑖𝑖 Then the error index values for the surfaces shown in Acknowledgments Fig. 1 can be calculated and written in the form of table 1. This work was supported by RFBR grants 19-01- Table 1. Error Index Values for 4 OpenFOAM Solvers 00402 and 20-01-00358. Solver rCF pCF QGDFOAM sF Error Index 0.037734 0.038751 0.0453406 0.058216 References [1] Federal standard Р 57700.12–2018. Numerical Table 1 shows that the values of the error index EI simulation of supersonic flows for an inviscid gas. completely correspond to the relative positions of the Software verification / National standard of the surfaces in Figure 1. Therefore, the calculated error index Russian Federation for numerical modeling of can serve as a characteristic of the accuracy of numerical physical processes. 2018, 20 p. methods in the selected class of problems. [2] Guide for the Verification and Validation of Computational Fluid Dynamics Simulations, 5. Conclusion American Institute of Aeronautics and Astronautics, The application of a generalized computational AIAA-G-077-1998, Reston, VA, 1998. experiment to the problems of comparative estimation of [3] Standard for Verification and Validation in "Mathematica Montisnigri", Vol. XXXVIII, 2017, p. Computational Fluid Dynamics and Heat Transfer, 63-77. ASME V&V 20-2009, 2009. [17] A.K. Alekseev, A.E. Bondarev, A.E. Kuvshinnikov. [4] Bondarev A.E. Analysis of Space-Time Flow Verification on the Ensemble of Independent Structures by Optimization and Visualization Numerical Solutions // Lecture Notes in Computer Methods // Transactions on Computational Science Science, Vol. 11540, pp. 315–324, 2019. DOI: XIX, LNCS 7870, pp. 158-168. Springer, Heidelberg 10.1007/978-3-030-22750-0_25 (2013) [18] Alekseev A.K., Bondarev A.E. Estimation of the [5] Bondarev A.E, Galaktionov V.A. Parametric Distance between True and Numerical Solutions // Optimizing Analysis of Unsteady Structures and Computational mathematics and mathematical Visualization of Multidimensional Data // physics, 2019, V. 59, № 6, с. 857-863 DOI: International Journal of Modeling, Simulation and 10.1134/S0965542519060034 Scientific Computing, 2013, V.04, N supp01, 13 p., [19] A.K. Alekseev, A.E. Bondarev, A.E. Kuvshinnikov. DOI 10.1142/S1793962313410043 On uncertainty quantification via the ensemble of [6] Bondarev A.E. On the Construction of the independent numerical solutions // Journal of Generalized Numerical Experiment in Fluid Computational Science 42 (2020) 101114, DOI: Dynamics // Mathematica Montisnigri, Vol. XLII, 10.1016/j.jocs.2020.101114 2018, p. 52-64 [20] Alekseev A.K., Bondarev A.E., Kuvshinnikov A.E. [7] A.E. Bondarev. On visualization problems in a Comparative analysis of the accuracy of OpenFOAM generalized computational experiment (2019). solvers for the oblique shock wave problem // Scientific Visualization 11.2: 156 - 162, DOI: Matematica Montisnigri, 2019, vol. XLV, p. 95-105 10.26583/sv.11.2.12 DOI: 10.20948/mathmontis-2019-45-8 [8] Alexander E. Bondarev and Artem E. Kuvshinnikov. [21] Bondarev A., Kuvshinnikov A. Parametric Study of Analysis of the Accuracy of OpenFOAM Solvers for the Accuracy of OpenFOAM Solvers for the Oblique the Problem of Supersonic Flow Around a Cone // Shock Wave Problem // IEEE The Proceedings of the Lecture Notes in Computer Science 10862, pp. 221– 2019 Ivannikov ISPRAS Open Conference 2019. P. 230, 2018. https://doi.org/10.1007/978-3-319-93713- 108-112, DOI:10.1109/ISPRAS47671.2019.00023 7_18 [22] S.V. Andreev, A.E. Bondarev, N.A. Bondareva. [9] A.E. Bondarev. On the Estimation of the Accuracy of Stereo images of error surfaces in problems of Numerical Solutions in CFD Problems // Lecture numerical methods verification (2020). Scientific Notes in Computer Science, Vol. 11540, pp. 325–333, Visualization 12.2: 151 - 157, DOI: 2019. DOI: 10.1007/978-3-030-22750-0_26 10.26583/sv.12.2.12 [10] A.E. Bondarev, V.A. Galaktionov. Construction of a generalized computational experiment and visual About the authors analysis of multidimensional data // CEUR Workshop Alekseev Aleksey K., Doctor in physics and mathematics, Proceedings, V. 2485, 2019, p. 117-121. DOI: senior researcher, Keldysh Institute of Applied Mathematics 10.30987/graphicon-2019-2-117-121 RAS. E-mail: aleksey.k.alekseev@gmail.com. [11] A.E. Bondarev, V.A. Galaktionov. Generalized Bondarev Alexander E., PhD, senior researcher, Keldysh Computational Experiment and Visual Analysis of Institute of Applied Mathematics RAS. E-mail: Multidimensional Data (2019). Scientific bond@keldysh.ru. Visualization 11.4: 102 - 114, DOI: Kuvshinnikov Artem E., junior researcher, Keldysh Institute of Applied Mathematics RAS. E-mail: kuvsh90@yandex.ru 10.26583/sv.11.4.09 [12] Bondarev A.E., Kuvshinnikov A.E. Comparative study of the accuracy for OpenFOAM solvers // Proceedings of Ivannikov ISPRAS Open Conference (ISPRAS), 2017, IEEE, IEEE Xplore, p.132-136 DOI: 10.1109/ISPRAS.2017.00028 [13] Bondarev A.E., Kuvshinnikov A.E. Analysis of the accuracy of OpenFOAM solvers for the problem of supersonic flow around a cone // Lecture Notes in Computer Science, vol. 10862, pp. 221–230, 2018. https://doi.org/10.1007/978-3-319-93713-7_18. [14] Bondarev A., Kuvshinnikov A. Comparative Estimation of QGDFoam Solver Accuracy for Inviscid Flow Around a Cone // IEEE The Proceedings of the 2018 Ivannikov ISPRAS Open Conference (ISPRAS-2018). P. 82-87, https://doi.org/10.1109/ISPRAS.2018.00019 [15] OpenFOAM Foundation. [Online]. Available: http://www.openfoam.org. [16] Alekseev A.K., Bondarev A.E. On exact solution enclosure on ensemble of numerical simulations //