The paper considers the problem of evaluating the state of a generalized computational experiment in the context of a general problem of creating methods for adaptive planning and control of a generalized computational experiment in mathematical modeling of real physical processes. A generalized computational experiment implies multiple solution of the numerical simulation problem for various sets of values of defining model parameters. As a method for assessing a generalized computational experiment state, it is proposed to visualize a set of experimental data specifying this state then followed by analysis of the resulting visual image. An approach to visualization of a generalized computational experiment state is proposed based on the sequential applying of two methods: visualization of a series of dependencies of the output simulation parameters on the input ones for a given set of approximating functions and visualization of approximation accuracy parameters for different ranges of values of the input parameters. A description of each of these methods is given. Examples of their application are considered when assessing the accuracy of numerical models of the OpenFOAM software platform for a three-dimensional problem of inviscid flow around a cone.
Generalized computational experiment, generalized computational experiment state, multidimensional data, visualization, visual analytics, approximation, problem of flow around a cone, OpenFOAM.
parameters.
automation [
The study of mathematical models of physical processes, usually, involves a series of computational
experiments the purpose of which is to investigate the state of the model and its behavior in different
ranges of variation of simulation parameters. In this regard, the greatest interest often lies in
simultaneously investigating the influence of several parameters on relevant model characteristics,
including investigating their joint influence in various combinations of variation ranges. For this
purpose, construction of the so-called generalized computational experiment (GCE) is performed [
Currently, there are examples of successful construction and application of a GCE in solving
problems of computational fluid dynamics [
2021 Copyright for this paper by its authors.
From the point of view of representing the results, a generalized computational experiment is
characterized by a multidimensional array the elements of which specify the distribution of values of
simulation output parameters for a given set of input parameters. In this case, some generalized
indicators often act as output parameters, which are the results of processing primary experimental data,
contain information about general patterns and relationships inherent in the object of modeling, and are
used for interpretation, search for patterns, formation and testing of hypotheses. Examples of
generalized indicators are the principal components in dimension reduction problems [
It is obvious that both conducting a GCE and processing and interpreting its results are very resource-intensive tasks. Moreover, it is not possible to conduct an experiment with all allowable combinations of models and simulation parameters. Therefore, it is necessary to resort to GCE planning choosing a specific scenario for its implementation. At the same time, it is advisable to build a GCE on the basis of not a static, predetermined, but a dynamic, adaptively changing plan. The principle of constructing such a plan can be as follows: based on the results of a series of experiments for a certain set of values of input parameters and processing of its results in conjunction with the results of previous series of experiments, the current GCE state is recorded. This state must be evaluated and analyzed in order to determine or adjust the plan for the subsequent series of experiments. To do this, it is necessary to establish dependencies of the output simulation parameters on the input ones and on this basis to select the value ranges of the input parameters for which more detailed studies are required with new or refined sets of parameters. Such situations may arise, for example, when new patterns are discovered that require confirmation and refinement, or if the results of some already conducted experiments do not correspond to the expected patterns and thus require rechecking.
As a method for evaluating a GCE state, it is proposed to visualize the experimental data that define
it, followed by analysis of the resulting visual image. The approach based on visualization of
multidimensional data has proven itself well in the tasks of exploratory analysis and hypothesis
formation [
A number of papers are devoted to the study of visualization problems in a generalized
computational experiment, among which, for example, [
We believe that within the framework of a GCE on a set of models M = {m1, m2, …, mNm), where Nm is the number of models, Nk computational experiments were carried out within which the input simulation parameters P = {p1, p2, …, pNp} were varied, where Np is the number of input simulation parameters. As a result of the computational experiments, the output parameters S = {s1, s2, …, sNs} were determined, where Ns is the number of output parameters. Each computational experiment k was carried out for a fixed set of input simulation parameters Pk = {pk,1, pk,2, …, pk,Nk}. In this case, situations are possible when, for a fixed set of simulation parameters, computational experiments were not performed on all models. The results of the performed computational experiments form a set R = {rk,m,v}, where k is the number of the computational experiment (1 ≤ k ≤ Nk ), m is the number of the model (1 ≤ m ≤ Nm), v is the number of the output parameter (1 ≤ v ≤ Ns).
Let us visualize the state of the GCE using a series of two-dimensional approximating curves of the results of computational experiments. Considering the fact that there can be several input simulation parameters, let us apply the following algorithm (Fig. 1) to construct two-dimensional approximating curves for one model m. Np, P, Ns, S, R, m i := 1; v := 1
d := 1 Fi,d,v, Nfi,d,v
j := 1
Selection of the corresponding results of computational experiments from the set R and the values of the input parameter i
Formation of pairs of points (x; y) for approximation
v := v + 1 v > Ns? Yes i := i + 1; v := 1
i > Np? Yes
Yes No No d := d + 1 d > Nti?
Yes No j := j + 1 j > Nfi,d,v?
No Visual analytics of graphs of approximating curves
End 1. Fix sequentially each input parameter pi. Values of this parameter determine the values of x pairs (x; y) for which we will further carry out approximation. 2. Determine set T = {td} of all possible combinations of the remaining input parameters pj (p ≠ pi), where 1 ≤ d ≤ Nti, Nti is the number of possible combinations of the remaining input parameters for a fixed parameter pi. 3. For each such combination td, obtain Nsd dependences of the output parameters sv on the parameter pi (it is assumed that not all combinations of input values could be obtained for all output parameters). 4. Carry out construction of the functional dependence using approximating functions. To do this, first determine the number – Nfi,d,v and the form of possible approximating functions , , = { , , ,1, , , ,2, … , , , , , , } for each resulting parameter sv, a combination of the input parameters d and a fixed input parameter i. The values of the resulting parameters sv (the corresponding rk,m,v are selected from the set R) specify the values of y in pairs (x; y) for which we will carry out the approximation. 5. For each approximating function fi,d,v,j for the resulting parameter sv and each combination td of input parameters, a graph of functional dependence on the parameter pi construct and approximation accuracy is determined ei,d,v,j. 6. Visually compare of graph shapes is carried out for a fixed parameter pi for different approximating functions, and deviations and patterns are revealed. Among other things, shapes of the curves obtained for different resulting parameters are compared. 7. Choose the following fixed input parameter: i = i + 1 and go to step 2. 8. If all the input parameters are exhausted, then the algorithm is completed.
The presented algorithm is repeated for each model.
The method based on application of this algorithm makes it possible to visually identify deviations in the results of experiments and determine patterns; however, it does not reflect a GCE state as a whole, since it forms not a single visual image but a series of visual images. To form a single visual image, it is proposed to use the following method.
The obtained characteristics of approximation accuracy ei,t,v,j are summarized in tables: for each pair of a fixed input parameter i and an output parameter v, we obtain one table, the columns of which are approximating functions fj,v, and the rows are specific values of the remaining parameters. The cells of this table are the corresponding values of approximation accuracy – ei,t,v,j. For each row of these tables, let us define characterizing values by the following methods: 1. Minimum accuracy of approximation (one parameter). 2. Maximum accuracy of approximation (one parameter). 3. Average accuracy of approximation and root-mean-square deviation of different methods of approximation from the mean (two parameters).
Visualization of the obtained characteristics will be carried out using two-dimensional dot plots. On this graph, for each resulting parameter v, the corresponding points will be of the same color. For different resulting parameters v, the colors will be different. We will also visualize different input parameters i using different colors (different types of markers can be also used). The ordinal number of the table rows will be used as the value of the point along the abscissa axis in a two-dimensional visualization. The characterizing values will be used as values along the ordinate axis. When using the 3rd method for determining the characteristic values, the average accuracy will determine the value along the ordinate axis, and the root-mean-square deviation will determine the size of the point.
Thus, on one two-dimensional dot plot, we can reflect a GCE state for one specific model. Analyzing the relative position (and the size of the points for the third method of determining the characteristic values) of the points, it is possible to visually determine for which input parameters there are problems with the approximation accuracy, and as a consequence, errors in carrying out computational experiments are possible, and for which it is not yet possible to determine the best approximation methods and, consequently, determine the value ranges of the input parameters for which it makes sense to carry out additional computational experiments in intermediate values.
Let us consider application of the proposed visualization methods for a GCE carried out within
evaluating the accuracy of solvers of the OpenFOAM platform (in OpenFOAM terminology, solvers
are software modules in which various numerical models of mechanics of continua are
implemented [
The simulation input parameters (P) are Mach number (Ma), cone half-angle (Betta, in degrees), and angle of attack (Angle, in degrees). The output parameters (S) of computational experiments are: the results of calculating deviation norms L1 and L2 of the numerical calculation from the analytical solution.
As approximating functions, we will use the following: linear (y = ax + b); exponential (y = aebx); logarithmic (y = aln(x) + b; quadratic (y = ax2 + bx + c).
As the approximation accuracy, we will use the value of the approximation reliability R2 [
As a result of the approximation of the obtained data, 178 approximating functions were constructed for each solver, considering the fact that some types of approximations in specific cases could not be carried out. For example, for the quadratic approximation, in the case of only two points, the graph was expressed in a line and therefore was not taken into account. Also, for the logarithmic function, it was impossible to determine approximating functions for the cases when the value of the parameter x could be equal to 0. The visualization results of the obtained approximating functions are partially shown in Fig. 3-5. For convenience, one graph displays the curves for all output parameters (norms L1 and L2).
After constructing a series of diagrams and determining the approximation accuracy, their mean values and root-mean-square deviations were calculated. Bubble charts were used for visualization, a different bubble color means belonging to different data series (a combination of a fixed input parameter and an output parameter), and the size of a point (bubble) characterizes the root-mean-square deviation of the approximation accuracy by different types of curves. Fig. 6 shows thus obtained visual GCE state diagram for sonicFoam solver. This diagram summarizes the results of evaluating the accuracy of the given solver by representing them as a single visual image.
Carrying out a visual analysis of the constructed approximating functions, it can be noted that, in most cases, the curve shapes for norms L1 and L2 are similar for a fixed value of the input parameter and the corresponding combinations of the remaining input parameters. However, in some cases deviations are observed. In particular, for sonicFoam solver (Fig. 5) with a fixed input parameter – Mach number and cone half-angle – 10° and angle of attack – 5°, a significantly greater curvature of the approximating curve for the L1 norm is observed, as well as the osculation of curves for the L1 and L2 norms. Similar anomalies were also observed for the same solver and the L1 norm in two other cases: 1. Fixed input parameter – angle of attack, Mach number – 5, half-angle – 10°. 2. Fixed input parameter – half-angle, Mach number – 5, and angle of attack – 5°.
These anomalies suggest that at Mach number of 5, half-angle of 10°, and angle of attack of 5° for sonicFoam solver, an error could have been made in the computational experiment or in the calculation of the L1 norm. Further clarification of this fact by the authors of the computational experiment confirmed the presence of a technical error related to tabulating the results of the computational experiments for the given combination of input parameters – the corresponding value of the L2 norm fell into the table cell for the L1 norm by mistake.
Analyzing the diagram of the GCE state visualization for sonicFoam solver (Fig. 6), it can also be noticed that for 3 cases the applied approximation methods give rather different results in terms of flow (3 large bubbles) and all of them are for the L1 norm: 1. Fixed input parameter – Mach number, half-angle – 10°, angle of attack – 5°. 2. Fixed input parameter – half-angle, Mach number – 5, angle of attack – 5°. 3. Fixed input parameter – angle of attack, Mach number – 7, half-angle – 20°.
The first two cases correspond to an already discovered problem by visualizing the approximating curves. The third case is typical for other computational experiments and may indicate the need for an additional series of experiments for intermediate values of the corresponding parameters in order to clarify the nature of the dependence and, possibly, to correct the list of types of approximating curves for sonicFoam solver.
The paper considers the problem of evaluating the state of a generalized computational experiment and the method for its solution using visualization of a set of experimental data that determines the GCE state, followed by analysis of the resulting visual image. An approach is proposed to visualize the GCE state based on the sequential applying of two methods: visualization of a series of dependencies of the output simulation parameters on the input ones for a given set of approximating functions and visualization of approximation parameters for different value ranges of the input parameters.
Due to the use of the proposed visualization methods, it was been possible to identify experiments that have signs of errors as well as the value ranges of the input parameters for which it is advisable to conduct additional experiments for intermediate values. These circumstances make it possible to correct the further plan of conducting computational experiments.
In combination with other methods for verifying GCE data [
It should also be noted that application of the above visualization methods can only be possible with a sufficient number of already performed computational experiments within a GCE, since to construct the approximating curves, at least 2 points are required with a fixed input parameter.
The research has been supported by Russian Science Foundation (project No. 18-11-00215).