This paper investigates the use of Physics-Informed Neural Networks (PINNs) in active learning cycles. We defined two scenarios: one initially unsupervised and the other initially supervised. PINNs emphasize the integration of physical laws into neural networks to improve the predictive performance of vanilla neural networks and to enhance the eficiency of traditional methods for solving partial diferential equations (PDEs). Key contributions include adapting existing computational frameworks to enable the use of Graph Neural Networks for solving problems that require the calculation of gradients on unstructured triangle meshes, a query strategy focusing on the physical loss, and a comparative analysis of this strategy against random sampling across both defined scenarios. This work establishes a foundation for future research aimed at expanding the application of PhysicsInformed Graph Neural Networks (PIGNN) using active learning and addressing real-world problems in fluid dynamics and electrodynamics.
Solving partial diferential equations (PDEs) is of paramount interest in numerous fields of science
and engineering, as they form the foundation for modeling a wide range of physical phenomena.
PDEs describe the behavior of physical systems over space and time, governing processes such as
heat transfer [
By embedding physical constraints directly into the neural network’s loss function cf. Eq. (1), they utilize both data loss
, cf. Eq. (2) and physics loss , cf. Eq. (3) components. With as the weighting factor for the data component of the total loss and N vertices.
= ⋅
+ , = = ( ) 1 =1 ∑( − , 2 ) , nEvelop-O LGOBE https://www.uni-kassel.de/eecs/ies/ (B. Sick)
PINNs ofer several advantages over traditional methods and ensure that the solutions are not only data-consistent but also physically accurate. Additionally, PINNs can naturally incorporate multiphysics problems and seamlessly handle high-dimensional spaces, providing a flexible and eficient CEUR Workshop Proceedings
ceur-ws.org
ISSN1613-0073
approach to solving complex PDEs. In Fig. 1 an active learning (AL) cycle with a PINN as Model is
depicted. The queries from the Selector are directed towards an Oracle which is in our case a FEM
simulation. The AL cycle uses Eq. (1) where measures the mean squared error between a predicted
and a true solution variable (for instance, the prediction of the electric potential), and is therefore
trained in a supervised manner. While the physics loss Eq. (3) corresponds to the residual ( )
and ensures adherence to the PDE. This loss term operates solely unsupervised on the predicted solution
variable . This integration allows PINNs to handle sparse data efectively, making them particularly
useful in real-world applications where data is limited [
We use this AL cycle to train the PINN starting from two diferent initial states. In Scenario U the model is initially trained completely unsupervised, using the physics-informed loss (3) only, and then data is provided by the oracle to continue the following iterations using on the additional data to support the unsupervised training. In Scenario S, we use ground-truth data for supervised training right from the start and therefore use as a loss function.
The choice of mesh plays a crucial role in the implementation of PINNs, as it defines the discretization
of the problem domain. Structured meshes, with their equidistantly distributed cells, ofer computational
eficiency and simplicity by enabling straightforward application of automatic diferentiation algorithm
to compute spatial gradients [
In summary, PINNs represent a sophisticated method for solving PDEs by integrating neural networks with physical laws. Enhancing these networks through AL by physical loss residuals, rather than explicit uncertainty quantification, allows for a more straightforward yet efective refinement process. Strategic mesh design further augments the model, making PIGNNs a versatile tool for a wide range of applications in science and engineering.
Contributions 1. We adapt an existing TensorFlow implementation to calculate field gradients on twodimensional triangle meshes for our PyTorch model. We use this implementation to build a physical loss function representing the Poisson equation with Dirichlet boundary conditions on an unstructured mesh. 2. We propose a simple yet efective query strategy utilizing the physical loss function. 3. We develop and evaluate two distinct PINN-based active learning scenarios, initially unsupervised and initially supervised, comparing our query strategy with random sampling.
The remainder of this article is structured as follows: in Section 2 we summarize the related work before we introduce our methodology in Section 3. Our preliminary results are presented in Section 4. The article concludes with a summary of our findings and an outlook for future work in Section 5.
In this section, we propose related work to the topics of GNNs and PINNs as well as AL and PINNs.
GNNs and PINNs: Solving mesh-based PDEs with neural networks is an increasingly progressive
topic of research. Typical data-driven solution methods come from the fields of computer vision and
graph-based learning [
Initial studies have demonstrated that combining GNNs and PINNs yields excellent results in various
scientific and engineering applications. GNNs excel at processing data represented as graphs, which
is particularly useful for handling complex relationships in unstructured meshes [
AL and PINNs: AL for regression tasks is highly efective in reducing the computational load
associated with simulating PDEs. By strategically selecting the most informative samples for extensive
simulation, AL can significantly enhance eficiency [
The idea of combining PINNs with AL is gaining increasing attention. Recent works have taken
initial steps in this direction, employing uncertainty sampling via Monte Carlo dropout [
Our methodology is structured as follows: first, we introduce the data derived from the Poisson equation, which is a second-order PDE. Subsequently, we present our model, query strategy, and oracle. Finally, we present our experimental setup.
Data: As dataset we use the charge density input array and the FEM simulated solutions of the Poisson equation together with the mesh, featuring a circular bounded domain (Ω ⊂ ℝ ), and the associated edge indices. As input scalar field we use a random distribution of circular areas with randomly chosen radii. Although the Poisson equation can be applied to a variety of physics problems, our goal is to calculate the electric potential field of a given constant charge density distribution , represented by the circle areas which is expressed in Eq. (4). Here Δ represents the Laplacian operator: −Δ = = 0 in Ω on Ω In this equation, Ω denotes the boundary of the domain Ω. In Fig. 2, the input features (Fig. 2a) and the ground-truth solution (Fig. 2b) of a random sample are exemplarily illustrated.
As illustrated in Fig. 2c, we employ an unstructured triangular mesh to discretize the domain. This type of mesh allows us to accurately capture the geometry and boundary conditions of complex domains. The physical loss of the Poisson equation (Eq. (4)) is defined in Eq. ( 5): = ( ) =
‖Δ + ‖ 2 Ω
{ ‖ ‖2 Ω
To compute this loss, it is necessary to obtain the second spatial derivative, indicated by the Laplace
operator. This computation requires considering the spatial dependencies of the mesh cells. While the
Automatic Diferentiation (AD) algorithm [
Model: We utilize our PIGNN to eficiently handle the intricate geometries of the domain. The
GNN’s structure is particularly well-suited for capturing the relationships and dependencies within
unstructured data. As GNN type we chose six chebyshev spectral graph convolutional (ChebConv)
layers as the main model and two feed-forward layers as encoder and decoder. The ChebConv layers
(4)
(5)
(a) Features
(b) Ground-Truth
(c) Triangular mesh
-hop convolutional operator aggregates information of vertices that are in a radius of -hops from
the central vertex in contrast to the more popular 1-hop graph convolutional layers, which only take
into account directly connected nodes. Using a -factor of six allows our model to recognize bigger
structures and helps to minimize the prediction error. To enhance the model’s capability in dealing with
complex mesh geometries, we integrate it with the MeshGradientPy package [
In Eq. (6) our query strategy is depicted. Let be the set of all samples that are inferred, and be a subset of containing the samples with the highest values. The subset is forwarded to the Oracle for target value acquisition.
= { ∈ ∣ () ∈ Top ( )} (6)
In contrast to uncertainty sampling in AL, this strategy works by evaluating the physical residuals,
which quantify how well the predicted solution variable adheres to the governing physical laws; thus,
no additional uncertainty estimation method is required. By identifying samples where the model’s
predictions are less reliable, we can target specific areas for model improvement. The advantage of this
approach is that we can quantify the physical loss in an unsupervised manner, thereby eliminating the
need for costly epistemic uncertainty quantification methods [
This unsupervised quantification of physical loss simplifies the AL process, allowing the model to autonomously identify and focus on regions with high residuals. These high-residual areas indicate where the model’s predictions are most inaccurate, guiding the addition of new data points or retraining eforts to these critical areas. This method not only streamlines the training process but also ensures robust model enhancement by continuously refining the model based on its internal assessments of physical law adherence. This approach is particularly valuable in scenarios where obtaining groundtruth data is expensive or impractical, as it maximizes the use of available information to improve model performance and reliability.
Oracle: Focusing on samples with high physical residuals, the Oracle generates additional data in these regions, thereby improving the model’s performance. The Model uses its internal physics-based evaluations to guide its learning process, leveraging both the supervised and unsupervised capabilities of the PIGNN to ensure its predictions remain physically consistent. The Selector identifies high-residual samples and the Oracle provides the corresponding true values, which are then included in the training of the Model for fine-tuning. This active interaction between the Oracle and the Model allows for targeted improvements in areas where the model’s predictions are less reliable enhancing the model’s performance cost-efectively.
Experimental Setup: Our experimental setup is designed to evaluate two distinct scenarios and is depicted in Fig. 1:
In Scenario U, we start with a pool of 1500 samples with ground-truth solution data determined from FEM simulations (oracle). The PIGNN is initially trained on 600 randomly selected samples in an unsupervised manner using the physics-based loss function (cf. Eq. (3)) only, therefore, no ground-truth data is provided. After this initial training phase, the model is evaluated on the remaining 900 samples, calculating physics residuals to identify the 60 samples with the highest residuals (cf. Eq.(6)). The ground-truth values for these high-residual samples are then obtained from the Oracle and added to the training set, enabling the use of the total loss Eq. (1) on these additionally acquired samples. This iterative process of identifying and adding 60 high-residual samples continues until the cycle iterates 5 times. This scenario is termed unsupervised since the majority of the training is based on the unsupervised physical loss (cf. Eq. (3)) only, except for the samples added by the oracle.
In Scenario S, we defined a supervised scenario, therefore, using the ground-truth data of the initial 600 randomly selected samples. The total loss (cf. Eq. (1)) is applied to both, the initial samples and the samples acquired over five iterations, which are provided by the oracle.
For both scenarios, after each iteration, the PIGNN is tested on a separate test dataset of 1500 samples to evaluate its prediction performance and adherence to physical laws. Additionally, we compare these methods to a random selection strategy, where 60 random samples are acquired for the training set in each iteration. This comparison assesses the eficiency of the proposed selection strategy guided by the physical loss .
The results of our experiments, are summarized in Fig. 3 and will be discussed in the following: First, we can observe that our proposed query strategy outperforms the random strategy in both scenarios. It is evident that after the first iteration, Scenario S, which was trained supervised to optimize the total loss , surpasses Scenario U, where the model was trained solely using the unsupervised loss . However, after four AL cycles, Scenario U demonstrates superior performance compared to Scenario S. This indicates that the initial unsupervised training is a viable approach for our PIGNN. Considering that substantial resources are saved by determining the ground-truth values for the initial training pool — which in our example involved 600 samples — and given that one FEM simulation in industrial use cases can take days or even weeks of computing time, the advantages become even more apparent. An adaptive approach that only simulates the most valuable samples presents significant benefits.
However, an in-depth analysis of the consistency of multiple runs using various seeds was beyond
the scope of this work. Additionally, we did not conduct any hyper-parameter tuning or investigate
AL parameters such as the initial pool size, the acquisition size, or the total budget. Consequently,
the observed fluctuations in the results may be attributed to these factors. These fluctuations may be
primarily due to the limited number of experiments performed and the non-optimized hyperparameters,
which were not adjusted due to the significant efort required, especially in the context of active
learning [
Other typical AL query strategies were also not considered. Another critical parameter is , which serves as the weighting factor between the two components of the loss function (cf. Eq. (1)). An incorrectly chosen can lead to the optimization being dominated by one part of the loss function, either or , at the expense of the other. These aspects need to be elaborated in future work.
Electric potential u [V]
In Fig. 4 a randomly chosen test sample of the final iteration of Scenario U is depicted. It shows that our AL strategy in combination with the PIGNN is capable of providing high-performing predictions in Fig. 4b. In Fig. 4c the absolute deviation e.g. the L1-error between the prediction and the ground-truth solution is depicted.
Our experiments show that our PIGNN is generally suitable for use in AL scenarios. Our proposed
query strategy is built upon the network’s physical loss, which can be evaluated unsupervised. In
future work, we aim to apply our methodology and model to real-world problems and more complex
datasets from the field of fluid dynamics [
Currently, our PIGNN is validated on a circular problem domain solving the Poisson equation on an
unstructured triangular mesh. In the future, we plan to employ this model for more complex geometries
and physical problems. For the above-mentioned datasets, we intend to solve the Maxwell equations on
an unstructured mesh for modeling an electric motor and address turbulent flow in a U-bend applying
the Navier-Stokes equations on a graded mesh. Another work compares methods from the fields of
computer vision and graph learning on these two datasets [
This research has been funded by the Federal Ministry for Economic Afairs and Climate Action (BMWK) within the project ”KI-basierte Topologieoptimierung elektrischer Maschinen (KITE)” (19I21034C).