We introduce a generalized physics-informed machine learning workflow to accurately predict the behavior of a transient physical system with enhanced physics conformity. A physics-guided machine learning (PGML) model is developed to achieve this goal. Our model consists of two main parts for a given transient system: (1) a physics-based numerical model which solves the system using conventional numerical methods and returns the stiffness matrix and force vector at each time step; (2) a neural network (NN) based machine learning (ML) surrogate model which predicts the solution of the system using a custom physics-guided loss function constructed from system matrix and force vector. The proposed workflow results in a physics-aware Machine Learning (ML) model. Such a trained model can be used to avoid the prohibitively expensive step of running a transient system simulation at the desired resolutions in space and time. We demonstrate and test the model on single-degree-of-freedom (SDOF) and multiple-degree-of-freedom (MDOF) system's examples from structural dynamics. Our results show that the method predicts the simulation results accurately. The proposed workflow can be directly adapted to any other physics and numerical method as it is not tailored towards a specific physics or a numerical method.
The partial differential equations (PDE) which govern
transient physical systems are solved using numerical methods
such as Finite Element Method (FEM)
These two shortcomings have made researchers explore the
possibilities of integrating knowledge of physical laws into
ML models.
Related Work
constant parameters of the numerical
method numerical model matrices K and F
Input data Z (system + numerical method parameters for the timestep) Custom loss using
K, F and x Network Loss < Tolerance YES Trained Model variable parameters at each timestep neural network prediction x NO Trained Model
Transient iterations of the numerical method
Input to the
system Predict System Response x YES
Time < End time
End previous predictions NO supervised, semi-supervised and reinforcement learning for predicting flow-map of a dynamic system. Their model predicted the flow-map of a system iteratively using the present state. The error correcting terms and extra physics based constraints resulted in striking improvement during prediction of the Lorenz System.
Recently,
Our Contribution It is important to note that almost all of the previous work explicitly define the governing PDE(s) of the given physical system as the NN’s training loss function or, constitute a ML model architecture specific to the given physical system. A major drawback of such approach is that it is not generalizable. When a new transient system is provided, one needs to modify the ML model’s training loss function or its architecture prior to the start of model training. To overcome this problem, we introduce a generalized physics-informed machine learning workflow for transient problems. The proposed approach is highly generalizable, makes use of physics-guided loss function which results in physics-conforming and data-efficient ML model. The proposed method can be used with any numerical method which results in a matrix form of equation system as in 1 .
Kxh = F Where xh is the unknown of the problem and K, F are the matrices results from the numerical method used, discretization scheme used and boundary and initial conditions applied.
We analyze the predictive ability of our approach on timeseries data of a linear SDOF and MDOF systems from structural dynamics.
2
Consider a transient physical system characterized by a partial differential equation (PDE) defined on a domain given by:
L(x) = 0
where xd and g are the Dirichlet and Neumann boundary
conditions given by equations 3 and 4 respectively. The
solution to equation 2 can be computed using various methods
such as FEM and FDM. In this contribution, we restrict the
discussion to Galerkin-based FEM
kn;1 | k1;2 k2;2 . .
. kn;2
K{(xzh) . . . kn;n
xn } | x{hz }
Fn | {z }
F (5) where K(xh) is the non-linear stiffness matrix, xh is the discrete system response, and F is the Force vector. The elements of the stiffness matrix and force vector depends upon the time integration, numerical method and space discretization used.
Application of FDM or FVM also results in such system equations represented by matrices. The proposed method can also be applied with other numerical methods too. 2.1
We propose a generalized physics-informed ML workflow to devise physics conforming ML models. Our workflow consists of the following steps (see Figure 1): Input Z to the system are the physical system parameters and numerical method specific parameters of the problem. A subset of this which varies for each timestep are the input to the neural network. 1. A conventional physics-based FEM model applies some numerical method to solve the PDE and outputs; response of the system xh, stiffness matrix K(xh) and force vector F at discrete intervals of time. 2. The matrices from the physics-based FEM model are used in the custom loss function of the NN-based ML model. The ML model is a surrogate for the solution to the system described in equation 2. It takes last three timestep response as input and predicts the response at the present timestep. Last three timestep responses are taken as input since most of common time-integration schemes uses last two or three timesteps. The model is trained with the help of a physics-guided loss-function. 3. Upon completion of training, deploy the trained model to avoid the computationally expensive step of using conventional time-integration schemes and forward linear solvers to run simulations of transient physical systems. In the following, we tested the accuracy of the trained model on the untrained region instead of recursive prediction. 2.2
In conventional training setting, the actual response of the system (x) is compared with the one predicted by ML model. This comparison is done using an loss (error) function and the model tries to minimize the output of the loss function during training. Mean Squared Error (MSE) is one of the most common choice. As mentioned in Section 1, this approach has a major drawback – the resulting model is a ”blackbox”.
Our approach addresses the latter by making use of a custom physics-guided loss function in the training process instead of a conventional MSE loss function. This custom loss function operates on the stiffness K(xh) and force F matrices produced by the physics-based FEM model. It is given by:
T 0 n 0 n Loss = X X X FiA where T indicates the number of time steps used for training the model, n represents the number of unknowns which describe the discrete system response (xh). The loss term represents the residual R of the equation in numerical methods. The prediction error gets magnified by K and results in ”NaN” for some physical problems where K is high. This is avoided by scaling the loss term with Fnorm,the L2 norm of the force matrices used for the training. It also brings all the rows of the Pin=1 Pjn=1 Ki;j xj Fi in same scale and avoid the optimizer in concentrating on one row of the prediction array. Thus, the final loss function is given by:
T 0 n 0 n Loss = X X X Fnorm 1 A
Fi Fnorm (6) 12 A (7) 3
This section discusses the results of the proposed method in predicting the transient simulation results for SDOF and MDOF systems in structural dynamics. In both the examples, the model takes last three time-step values as input to predict the present time-step value. Currently the method is tested on the untrained data, not for a recursive prediction. Even though we focus on these two problems, the method can be directly used with any transient simulation solved using any numerical methods. 3.1
A simple vibration system can be represented by a single mass connected to a spring and a damper. Such systems are called SDOF system and governed by the second order differential equation of single variable given by:
d2x dx m dt2 + c dt + kx = f (t) (8) where m is the mass, c the damping constant, k the stiffness and f (t) the excitation force. The response of the system can be represented as [x; x_ ; x], where x is the displacement, x_ is the velocity and x is the acceleration of the system. m c k f (t)
10 kg 10 Ns/m 1580 N/m 1000sin(4 t)N
We use a physics-based solver to solve the equation and
compute response of the system for a given excitation force
f (t) with the help of
10000 litroeen 75500000 a c c ,A 2500 y iltcoe 0 V tn 2500 , e em 5000 c a l ispD 7500
We devise a physics-guided ML model (PGML) based on neural network and uses a custom physics-guided loss function during training. The model’s architecture is composed of a simple Long-short-term-memory (LSTM) network with 60 hidden units. We train this model on the system response of first 500 time-steps and the model is tested on the next 500 time-steps. The proposed neural network model takes system response from the previous three time-steps as input and predicts the response at the subsequent time-step. Adamax optimizer with a learning rate of 1e 4 is used for the training. A dropout value of 0:2 , 1 = 0:9 and 2 = 0:99 are used in the network.
Figure 3 shows the displacement of the system predicted using the trained network. It also shows the reference solution calculated using the physics-based FEM model with the help of Newmark time-integration scheme. A good accuracy is maintained between the predicted (PGML) and numerical 6 Newmark PGML Absolute Error 9
10 Newmark PGML Absolute Error (b) Zoomed-in displacement 648000000 PANGbeswMomLluaterkError 1)s 200 (tym 0 i c loeV 200 400 600 800 5.0 5.2 5.8
6.0 5.4 Time (s) 5.6 method computed (Newmark) solutions. The plot of absolute values of errors between Newmark and PGML are also shown in Figure 3. The prediction of velocity and acceleration also follows the same behavior as that of displacement. It is to be noted that the proposed method is able to predict all three variables accurately even when their magnitude were in different scales.
The error distribution in predicting displacement, velocity and acceleration are given in Figure 4. The relative error is plotted against the number of occurrence. It can be seen that error in all three variables are concentrated near zero. The relative error have a mean of 0:0401, 0:0299, 0:0033 for displacement, velocity and acceleration respectively. It is found that relative prediction error in acceleration have a high standard deviation of 0:1534. Whereas, standard deviation for relative error is 0:0463 and 0:0200 for displacement and velocity respectively. 3.2
The equation of motion of a multi degree of freedom system (MDOF) in structural dynamics is given by
d2X dX M + C + KX = F(t) (9)
dt2 dt where, M, C and K are the global mass, damping and stiffness matrices and F is the external force on the system. Here X represents the collection all degrees of freedoms of the system. We use a 20 DOF system to demonstrate the ) m (t 0.10 n e m e lca 0.05 p s i d ,y9 0.00 x 9 , ,y7 0.05 y 5 , x ,4 0.10 x 1 25 20 ity15 senD10 5 1x 5y 4x 7y 9x 9y 0
2 0 0.1 0.0 0.1 0.2 0.3
Relative Error in Displacement 0.4 0.5 model. Each DOF in this system is either the x direction or the y direction displacement of one of the 10 masses in the system. The external force applied is F(t) = sin(1:25 2 t)N on all masses. The displacement of the masses of the system for the first 10 seconds is given in Figure 5. It is calculated using FEM with generalized alpha time integration scheme. We used a timestep of t = 0:001. Only selected DOFs are plotted in the figure.
Gen-alpha PGML Absolute Error Gen-alpha PGML Absolute Error Gen-alpha PGML Absolute Error Gen-alpha PGML Absolute Error time-steps for training the network. The trained model is used to predict the solutions of the remaining part of the simulation. We used Adamax optimizer with a learning rate of 1e 4, dropout value of 0:3 , 1 = 0:9 and 2 = 0:99. 5 6 9 10 5 6 9 10 4
6
Each curve in the Figure 5 represents one degree of freedom, such as the first mass displacement in the x direction 1x and the first mass displacement in the y direction 1y. Each DOF behave differently according the properties of the system and applied force. It makes the MDOF system more complex in comparison to the SDOF system.
The PGML model used consisted of a three layer LSTM network with 200 hidden units. The training used first 5000 5 6 9
10 7 Time (s) 8
Figure 6 shows the prediction using the trained network for the next 5000 time steps (5-10 seconds). It is compared against the actual solution calculated using FEM with Generalized Alpha time integration scheme. Only selected DOFs from the 20 DOFs are plotted. The results show a good agreement between the model prediction and the FEM solution. The model maintained the prediction accuracy for all the twenty DOFs. Even though the displacements of various DOFs differ in scale and pattern across time, the model is able to capture these variations and make accurate prediction.
The error between the prediction and the FEM solution are also plotted in Figure 6. The prediction error is close to zero for all the DOFs. But, it is observed that the prediction error is high close to the crest and trough of the response for some of the DOFs. We think that the high gradient of 0.10 Gen-alpha PGML displacement at crests and troughs is causing this behavior. This can be solved by taking excitation force also as an input parameter as this is what triggering the change in displacement. MDOF prediction for longer duration The trained network is used to predict the solution of the simulation for longer duration (5-30 seconds). Figure 7 shows the prediction of displacement for 9x. The model maintains the accuracy even for varying amplitude and slope of the displacement curve. The same is observed with other DOFs too. 4
We introduced a generalized physics-guided machine learning workflow to train a neural network for transient simulations. The PGML model developed to achieve this goal takes system matrix and force vector constructed from the numerical method for the training. Since the loss function used directly reflects the residual from the numerical methods, the trained model is physics conforming in comparison to a conventional ”black box” neural network. The proposed model can be easily adapted to different physics and numerical methods. The prediction capacity of the proposed model is demonstrated with the help of two examples from structural dynamics, SDOF and MDOF systems. The results point towards a promising algorithm for training neural networks for transient simulations. Such a trained model can be deployed in a real system and the signals from real system can be compared with prediction for anomalies.
5
Transient simulations for real applications typically consist of large DOF systems. Such systems also involve complex input forces, which vary in time. The proposed algorithm needs to be tested with such complex scenarios for its robustness. In such a case, more parameters such as force and system parameters might be needed at the input side of the neural network. Another drawback of the current implementation is that there is no mechanism to correct the error that accumulates over a long series of recursive predictions. Presently, recursive prediction using the same model diverges due to the accumulation of the error. Algorithms and architectures tailored for recursive prediction also to be tested with our model in the future work.
The proposed algorithm can only be used for predicting the simulation for later timesteps once trained with the simulation done so far. The next step includes training a model that can produce a generalized model, performing a complete simulation given the initial conditions and system parameters.