Graph-Informed Neural Networks Søren Taverniers,1 Eric J. Hall,2 Markos A. Katsoulakis,3 Daniel M. Tartakovsky4 1 Palo Alto Research Center (PARC), 3333 Coyote Hill Road, Palo Alto, CA 94304, USA 2 Division of Mathematics, University of Dundee, Dundee, DD1 4HN, UK 1 Department of Mathematics and Statistics, University of Massachusetts Amherst, Amherst, MA 01003, USA 4 Department of Energy Resources Engineering, Stanford University, Stanford, CA 94305, USA ehall001@dundee.ac.uk (Eric J. Hall), tartakovsky@stanford.edu (Daniel M. Tartakovsky) Abstract where it is deployed to build highly accurate kernel den- sity estimators (KDEs) for the probability density functions Graph-Informed Neural Networks (GINNs) present a strat- egy for incorporating domain knowledge into scientific ma- (PDFs) of relevant output quantities of interest (QoIs). chine learning for complex physical systems. The construc- tion utilizes probabilistic graphical models (PGMs) to incor- GINN (surrogate model) porate expert knowledge, available data, constraints, etc. with EDL Formation Environmental physics-based models such as systems of ordinary and partial Operating Upscaling differential equations (ODEs and PDEs). Computationally in- Macro Diffusion Structure tensive nodes in this hybrid model are replaced by the hid- . . Latent . . den nodes of a neural network (i.e., learned features). Once . . trained, the resulting GINN surrogate can cheaply generate physically-relevant predictions at scale thereby enabling ro- bust sensitivity analysis and uncertainty quantification (UQ). Learned Features r As proof of concept, we build a GINN for a multiscale model of electrical double-layer capacitor dynamics embedded into Domain- lpor Deff + a Bayesian network (BN) PDE hybrid model. aware ω PGM In recent years, several approaches have been proposed T κeff t+ to inform deep neural networks (DNNs) of physical laws λD and constraints to ensure they produce physically sound ϕΓ D−eff predictions. Two main classes of DNNs for building sur- cin rogate representations of physics-based models described ϕEDL Quantities of Interest χ± EDL Formation Control Variables by PDEs have emerged: physics-informed NNs (PINNs) Environmental Operating Computational Bottleneck Upscaling (Raissi, Perdikaris, and Karniadakis 2019) and “data-free” Macro Diffusion Structure . . physics-constrained NNs (Zhu et al. 2019). Our approach . . . . Latent uses the well-known concept of PGMs to embed domain Hybrid model (high delity) knowledge, including correlations between control variables (CVs), into standard DNNs by only modifying their input Figure 1: A domain-aware PGM encoding structured priors r Learned Features layer structure and enabling the use of a standard penalty on CVs serves as input to both the BN PDE (lower route) fi lpor Deff + in the loss function, e.g., `1 (lasso regression) or `2 (ridge and trained GINN (upper route) for a homogenized model ω regression) regularization. This non-intrusive approach per- of ion diffusion in supercapacitors (Taverniers et al. 2020). T κeff t+ λD mits the use of off-the-shelf software like TensorFlow or ϕΓ Deff − PyTorch with minimal effort from the user, while remain- cin Quantities of Interest ing compatible with PINNs and other customized NN archi- Constructing and training a GINN Control Variables ϕEDL χ± tectures which can be used to replace individual computa- Computational Bottleneck Simulation-based decision-making for design tasks involv- tional bottlenecks in the physics-based representation. ing complex multiscale/multiphysics systems requires pre- GINNs are particularly suited to enhance the compu- dicting the impact of tunable CVs on the system’s QoIs. Typ- tational workflow for complex systems featuring intrinsic ically, this is modeled by recasting the problem in a prob- computational bottlenecks and intricate physical relations abilistic framework where CVs and QoIs are represented among input CVs. Hence, to showcase the potential of this as random quantities that can be sampled from their cor- approach, we apply a GINN to simulation-based decision- responding probability distributions. For most real-world making in electrical double-layer (EDL) supercapacitors, applications, these are continuous, non-Gaussian variables Copyright © 2021, for this paper by its authors. Use permitted that need to be characterized by their full PDF rather than under Creative Commons License Attribution 4.0 International through a finite set of moments. (CCBY 4.0). Figure 1 visualizes the construction of a GINN surrogate for a multiscale model of EDL supercapacitor dynamics. intervals for an equivalent computational cost (since learn- A BN, a type of directed acyclic PGM, systematically in- ing the GINN’s parameters and predicting new data with the corporates domain knowledge into the physics-based model GINN carries a negligible computational expense). through structured priors on CVs, resulting in a hybrid BN 5 PDE model for macroscopic diffusion QoIs. The GINN re- 2.5 (Hybrid model) 4 tains the structured priors as inputs but replaces the hybrid 2 (GINN surrogate) 3 model’s computationally intensive nodes, related to upscal- 1.5 (Hybrid model) 2 ing via homogenization, with learned features to speed up 1 (GINN surrogate) 1 0.5 the generation of QoIs while maintaining physical relevance. 0 0 The GINN workflow, summarized in Fig. 2, consists of: 0.1 0.2 0.3 0.4 0.5 0.6 0.15 0.2 0.25 0.3 0.35 0.4 0.45 0.5 0.55 0.6 0.65 0.12 10 (Hybrid model) 1. Data generation: Generate Nsam input-output (io) sam- 0.1 (Hybrid model) 8 ples, divided into Ntrain training and Ntest test samples. 0.08 (GINN surrogate) 6 (GINN surrogate) 0.06 2. Training: Train the GINN with Ntrain training samples. 0.04 4 2 0.02 3. Testing: Test the trained GINN’s ability to handle unseen 0 0 5 10 15 20 25 30 0 0.4 0.45 0.5 0.55 0.6 0.65 data using the Ntest test samples. 4. Repeat steps 1 through 3 (modifying Ntrain ) until both the Figure 3: Estimated marginal densities for the QoIs in the training and test error tolerance are satisfied. supercapacitor testbed based on 8 × 103 samples computed pred with the hybrid BN PDE (solid/blue) or 107 samples com- 5. Prediction: Draw Nsam inputs from the structured pri- puted with the GINN (dashed/red) (Hall et al. 2021). ors on the CVs and predict corresponding QoIs with the trained GINN surrogate. Conclusions PGM PB computations in, domain knowledge structured priors (SP) Physics-based (PB) or surrogate? PB e.g., COMSOL, Our full analysis, in (Hall et al. 2021; Taverniers et al. 2020), SP inputs MATLAB, FENICS outputs suggests that GINNs, which take structured PGMs as inputs, GENERATING DATA SURROGATE estimate QoIs produce physically relevant QoIs that can be used to gener- ate KDEs for robust and reliable sensitivity analysis and fur- Learning completed? NO ther UQ. Trained on a small set of high-fidelity input-output YES SP inputs outputs data from a domain-aware hybrid model, GINNs can quickly PB outputs NO computed computed? outputs generate large amounts of output predictions, yielding an ap- forward propagation SP inputs YES using initial guesses for weights/biases PB outputs computed? NO proach that is orders of magnitude faster than counterparts YES that rely on physics-based models alone. hidden layers from trained GINN GINN outputs (initial) forward backpropagation learned weights/biases SP inputs propagation using learned Acknowledgments weights/biases GINN outputs forward propagation This work was performed while S. T. was employed by Stan- estimate QoIs GINN outputs ford University. cheaply GINN outputs (final) modify PREDICTING YES NO References modify DATA-DRIVEN UQ NN hyperparameters NO TRAINING TESTING YES Hall, E. J.; Taverniers, S.; Katsoulakis, M. A.; and Tar- takovsky, D. M. 2021. GINNs: Graph-Informed Neu- Figure 2: Overview of the global algorithm for GINN-based ral Networks for Multiscale Physics. J. Comput. Phys. training, testing, and predicting (Hall et al. 2021). 433: 110192. doi:10.1016/j.jcp.2021.110192. Share link authors.elsevier.com/a/1ccIO508Hokch valid until 2021-04-10. GINN-based decision-making Raissi, M.; Perdikaris, P.; and Karniadakis, G. 2019. A GINN’s ability to cheaply generate io sample pairs can Physics-informed neural networks: A deep learning frame- be leveraged to construct KDEs for the marginal and joint work for solving forward and inverse problems involving PDFs of QoIs with appropriate confidence intervals. Such nonlinear partial differential equations. J. Comput. Phys. nonparametric estimators form the building blocks for UQ 378: 686–707. tasks such as sensitivity analysis. Taverniers, S.; Hall, E. J.; Katsoulakis, M. A.; and Tar- In Fig. 3, we plot KDEs for QoIs based on 8 × 103 sam- takovsky, D. M. 2020. Mutual Information for Explainable ples simulated using the BN PDE (the minimum amount Deep Learning of Multiscale Systems. ArXiv:2009.04570. of io data needed to train the GINN) and on 107 samples Zhu, Y.; Zabaras, N.; Koutsourelakis, P.-S.; and Perdikaris, predicted with the GINN. We find that the GINN-predicted P. 2019. Physics-constrained deep learning for high- KDEs do not include spurious features observed with the dimensional surrogate modeling and uncertainty quantifica- smaller, expensive-to-compute data set generated with the tion without labeled data. J. Comput. Phys. 394: 56–81. physics-based model, and achieve much tighter confidence