ch-PINNModulus.tex

\documentclass[10pt]{article}
\usepackage[utf8]{inputenc}
\usepackage[T1]{fontenc}
\usepackage{amsmath}
\usepackage{amsfonts}
\usepackage{amssymb}
%\usepackage[version=4]{mhchem}
\usepackage{stmaryrd}
\usepackage{bbold}
\usepackage{graphicx}
%\usepackage[export]{adjustbox}
\graphicspath{ {/home/chaztikov/Downloads/Physics Informed Neural Networks in Modulus/2022_12_25_12ad409489955f2c183bg/images/} }

\begin{document}
\section{Basic methodology}
In this section we provide a brief introduction to solving differential equations with neural networks. The idea is to use a neural network to approximate the solution to the given differential equation and boundary conditions. We train this neural network by constructing a loss function for how well the neural network is satisfying the differential equation and boundary conditions. If the network is able to minimize this loss function then it will in effect, solve the given differential equation.

To illustrate this idea we will give an example of solving the following problem,

$$
\mathbf{P}:\left\{\begin{array}{c}
\frac{\delta^{2} u}{\delta x^{2}}(x)=f(x) \\
u(0)=u(1)=0
\end{array}\right.
$$

We start by constructing a neural network $u_{\text {net }}(x)$. The input to this network is a single value $x \in \mathbb{R}$ and its output is also a single value $u_{n e t}(x) \in \mathbb{R}$. We suppose that this neural network is infinitely differentiable, $u_{n e t} \in C^{\infty}$. The typical neural network used is a deep fully connected network where the activation functions are infinitely differentiable.

Next we need to construct a loss function to train this neural network. We easily encode the boundary conditions as a loss in the following way:

$$
L_{B C}=u_{n e t}(0)^{2}+u_{n e t}(1)^{2}
$$

For encoding the equations, we need to compute the derivatives of $u_{n e t}$. Using automatic differentiation we can do so and compute $\frac{\delta^{2} u_{n e t}}{\delta x^{2}}(x)$. This allows us to write a loss function of the form:

$$
L_{\text {residual }}=\frac{1}{N} \sum_{i=0}^{N}\left(\frac{\delta^{2} u_{n e t}}{\delta x^{2}}\left(x_{i}\right)-f\left(x_{i}\right)\right)^{2}
$$

Where the $x_{i}$ 's are a batch of points sampled in the interior, $x_{i} \in(0,1)$. Our total loss is then $L=L_{B C}+L_{\text {residual }}$. Optimizers such as Adam ${ }^{1}$ are used to train this neural network. Given $f(x)=1$, the true solution is $\frac{1}{2}(x-1) x$. Upon solving the problem, you can obtain good agreement between the exact solution and the neural network solution as shown in Fig. 10.

%\begin{center}
%\includegraphics[width=\textwidth]{2022_12_25_12ad409489955f2c183bg-2}
%\end{center}

Fig. 10 Neural Network Solver compared with analytical solution.

Using the PINNs in Modulus, we were able to solve complex problems with intricate geometries and multiple physics. In order to achieve this we have deviated and improved on the current state-of-the-art in several important ways. In this section we will briefly cover some topics related to this.

\section{Monte Carlo integration for loss formulation}
In literature, the losses are often defined as a summation similar to our above equation (7), ${ }^{2}$. We take a different approach and view the losses as integrals. You can instead write $L_{\text {residual }}$ in the form,

$$
L_{r e s i d u a l}=\int_{0}^{1}\left(\frac{\delta^{2} u_{n e t}}{\delta x^{2}}(x)-f(x)\right)^{2} d x
$$

Now there is a question of how we approximate this integral. We can easily see that if we use Monte Carlo integration we arrive at the same summation in equation (7).

$\int_{0}^{1}\left(\frac{\delta^{2} u_{n e t}}{\delta x^{2}}(x)-f(x)\right)^{2} d x \approx\left(\int_{0}^{1} d x\right) \frac{1}{N} \sum_{i=0}^{N}\left(\frac{\delta^{2} u_{n e t}}{\delta x^{2}}\left(x_{i}\right)-f\left(x_{i}\right)\right)^{2}=\frac{1}{N} \sum_{i=0}^{N}\left(\frac{\delta^{2} u_{n e t}}{\delta x^{2}}\left(x_{i}\right)\left(\text { (9) } f\left(x_{i}\right)\right)^{2}\right.$

We note that, this arrives at the exact same summation because $\int_{0}^{1} d x=1$. However, this will scale the loss proportional to the area. We view this as a benefit because it keeps the loss per area consistent across domains. We also note that this opens the door to more efficient integration techniques. In several examples, in this user guide we sample with higher frequency in certain areas of the domain to approximate the integral losses more efficiently.






\section{Integral Equations}
Many PDEs of interest have integral formulations. Take for example the continuity equation for incompressible flow,

$$
\frac{\delta u}{\delta x}+\frac{\delta v}{\delta y}+\frac{\delta w}{\delta z}=0
$$

We can write this in integral form as the following,

$$
\iint_{S}\left(n_{x} u+n_{y} v+n_{z} w\right) d S=0
$$

Where $S$ is any closed surface in the domain and $n_{x}, n_{y}, n_{z}$ are the normals. We can construct a loss function using this integral form and approximate it with Monte Carlo Integration in the following way,

$$
L_{I C}=\left(\iint_{S}\left(n_{x} u+n_{y} v+n_{z} w\right) d S\right)^{2} \approx\left(\left(\iint_{S} d S\right) \frac{1}{N} \sum_{i=0}^{N}\left(n_{x}^{i} u_{i}+n_{y}^{i} v_{i}+n_{z}^{i} w_{i}\right)\right)^{2}
$$

For some problems we have found that integrating such losses significantly speeds up convergence.
\subfile{/home/chaztikov/git/IBPINN/unc-dissertation/sec-GeneralizedPolynomialChaos.tex}
%\subfile{/home/chaztikov/git/IBPINN/unc-dissertation/inverse_problem/Modulus-example.tex}
%\section{Approximation to Parameterized PDE and Geometries}
\subfile{/home/chaztikov/git/IBPINN/unc-dissertation/sec-PINNModulus-Parametrized.tex}
%\section{Weak solution of PDEs using PINNs}

%In previous discussions on PINNs, we aimed at solving the classical solution of the PDEs. However, some physics have no classical (or strong) form but only a variational (or weak) form ${ }^{4}$. 
%This requires handling the PDEs in a different approach other than its original (classical) form, especially for interface problem, concave domain, singular problem, etc. 
%In Modulus, we can solve the PDEs not only in its classical form, but also in it weak form. Before describing the theory for weak solutions of PDEs using PINNs, let's start by the definitions of classical, strong and weak solutions.
%
%Note: The mathematical definitions of the different spaces that are used in the subsequent sections like the $L^{p}$, $C^{k}, W^{k, p}, H$, etc. can be found in the Relative Function Spaces and Integral Identities. For general theory of the partial differential equations and variational approach, we recommend 5,6 . 
%In this section, we introduce the classical solution, strong solution, and weak solution for the Dirichlet problem. Let us consider the following Poisson's equation.
%
%$$
%\left\{\begin{array}{l}
%\Delta u=f \quad \text { in } \Omega \\
%u=0 \quad \text { on } \partial \Omega
%\end{array}\right.
%$$
%
%\section{Definition (Classical Solution):}
%Let $f \in C(\bar{\Omega})$ in (18), then there is a unique solution $u \in C^{2}(\Omega) \cap C_{0}^{1}(\Omega)$ for (18). We call this solution as the classical solution of (18).
%
%\section{Definition (Strong Solution):}
%Let $f \in L^{2}(\Omega)$ in (18), then there is a unique solution $u \in H^{2}(\Omega) \cap H_{0}^{1}(\Omega)$ for (18). We call this solution as the strong solution of (18).
%
%From the definition of strong solution and Sobolev space, we can see that the solution of (18) is actually the solution of the following problem: Finding a $u \in H^{2}(\Omega) \cap H_{0}^{1}(\Omega)$, such that
%
%$$
%\int_{\Omega}(\Delta u+f) v d x=0 \quad \forall v \in C_{0}^{\infty}(\Omega)
%$$
%
%By applying integration by parts and $u=0$, we get
%
%$$
%\int_{\Omega} \nabla u \cdot \nabla v d x=\int_{\Omega} f v d x
%$$
%
%This leads us to the definition of weak solution as the following.
%
%\section{Definition (Weak Solution):}
%Let $f \in L^{2}(\Omega)$ in (18), then there is a unique solution $u \in H_{0}^{1}(\Omega)$ for the following problem: Finding a $u \in H_{0}^{1}(\Omega)$ such that
%
%$$
%\int_{\Omega} \nabla u \cdot \nabla v d x=\int_{\Omega} f v d x \quad \forall v \in H_{0}^{1}(\Omega)
%$$
%
%We call this solution as the weak solution of (18).
%
%In simpler terms, the difference between these three types of solutions can be summarized as below:
%
%The essential difference among classical solution, strong solution and weak solution is their regularity requirements. The classic solution is a solution with 2 nd order continuous derivatives. The strong solution has 2 nd order weak derivatives, while the weak solution has weak 1st order weak derivatives. Obviously, classical solution has highest regularity requirement and the weak solution has lowest one. Now we will discuss how PINNs can be used to handle the PDEs in approaches different than its original (classical) form. In $^{7},^{8}$, the authors introduced the VPINN and hp-VPINN methods to solve PDEs' integral form. This integral form is based on (19). Hence, it is solving a strong solution, which is better than a classical solution.
%
%To further improve the performance of PINNs, we establish the method based on eq:weak i.e., we are solving the weak solution. Let us assume we are solving (18). To seek the weak solution, we may focus on the following variational form:
%
%$$
%\begin{gathered}
%\int_{\Omega} \nabla u \cdot \nabla v d x=\int_{\Omega} f v d x \\
%u=0 \quad \text { on } \partial \Omega
%\end{gathered}
%$$
%
%For (23), we may handle it as the traditional PINNs: take random points $\left\{\mathbf{x}_{\mathbf{i}}^{b}\right\}_{i=1}^{N_{b}} \subset \partial \Omega$, then the boundary loss is
%
%$$
%M S E_{b}=\frac{1}{N_{b}} \sum_{i=1}^{N_{b}}\left(u_{N N}\left(\mathbf{x}_{\mathbf{i}}^{b}\right)-0\right)^{2}
%$$
%
%For (22), we choose a quadrature rule $\left\{\mathbf{x}_{\mathbf{i}}^{q}, w_{i}^{q}\right\}_{i=1}^{N_{q}}$, such that for $u: \Omega \mapsto \mathbb{R}$, we have
%
%$$
%\int_{\Omega} u d x \approx \sum_{i=1}^{N_{q}} w_{i}^{q} u\left(\mathbf{x}_{\mathbf{i}}^{q}\right) .
%$$
%
%For uniform random points or quasi Monte Carlo points, $w_{i}^{q}=1 / N_{q}$ for $i=1, \cdots, N_{q}$. Additionally, we choose a set of test functions $v_{j} \in V_{h}, j=1, \cdots, M$ and then the loss of the integral is
%
%$$
%M S E_{v}=\left[\sum_{i=1}^{N_{q}} w_{i}^{q}\left(\nabla u\left(\mathbf{x}_{\mathbf{i}}^{q}\right) \cdot \nabla v_{j}\left(\mathbf{x}_{\mathbf{i}}^{q}\right)-f\left(\mathbf{x}_{\mathbf{i}}^{q}\right) v_{j}\left(\mathbf{x}_{\mathbf{i}}^{q}\right)\right)\right]^{2} .
%$$
%
%Then, the total loss is
%
%$$
%M S E=\lambda_{v} * M S E_{v}+\lambda_{b} * M S E_{b},
%$$
%
%where the $\lambda_{v}$ and $\lambda_{b}$ are the corresponding weights for each terms.
%
%As we will see in the tutorial example Interface Problem by Variational Method, this scheme is flexible and can handle the interface and Neumann boundary condition easily. We can also use more than one neural networks on different domains by applying the discontinuous Galerkin scheme.
%
%\section{References}
%[1] : Kingma, Diederik P., and Jimmy partial. "Adam: A method for stochastic optimization." arXiv preprint arXiv:1412.6980 (2014).
%
%[2] : Raissi, Maziar, Paris Perdikaris, and George Em Karniadakis. "Physics informed deep learning (part i): Data-driven solutions of nonlinear partial differential equations." arXiv preprint arXiv:1711.10561 (2017).
%
%[3] : Sun, Luning, et al. "Surrogate modeling for fluid flows based on physics-constrained deep learning without simulation data." Computer Methods in Applied Mechanics and Engineering 361 (2020): 112732. [4] : Braess, Dietrich. Finite elements: Theory, fast solvers, and applications in solid mechanics. Cambridge University Press, 2007.
%
%[5] : Gilbarg, David, and Neil S. Trudinger. Elliptic partial differential equations of second order. Vol. 224. springer, 2015.
%
%[6] : Evans, Lawrnce C. "Partial differential equations and Monge-Kantorovich mass transfer." Current developments in mathematics $1997.1$ (1997): 65-126.
%
%[7] : Kharazmi, Ehsan, Zhongqiang Zhang, and George Em Karniadakis. "Variational physics-informed neural networks for solving partial differential equations." arXiv preprint arXiv:1912.00873 (2019).
%
%[8] : Kharazmi, Ehsan, Zhongqiang Zhang, and George Em Karniadakis. "hp-VPINNs: Variational physics-informed neural networks with domain decomposition." Computer Methods in Applied Mechanics and Engineering 374 (2021): 113547.


\end{document}