Automatic Differentiation from Scratch: How PyTorch Computes Gradients in Physics-Informed Neural Networks

Short summary

This paper traces exactly how PyTorch's automatic differentiation engine computes gradients for Physics-Informed Neural Networks, which require two levels of differentiation: physics derivatives through the network and parameter gradients of a loss depending on those derivatives. Using a small MLP on a simple ODE, the authors trace every node in the computational graph and verify all 22 parameter gradients against hand derivations. The work clarifies the graph-on-graph mechanism (create_graph=True) that enables correct second-order differentiation in PINNs.

• •Traces PyTorch autograd pipeline for PINN training with explicit numerical values
• •Demonstrates graph-on-graph mechanism for second-order differentiation in physics-informed residuals
• •Verifies all adjoint values against hand derivations using a 1-3-3-1 MLP

Generated with AI, which can make mistakes.