From 0457de775cd43722b7b01c3dc7c8544e146bc4e4 Mon Sep 17 00:00:00 2001
From: "Documenter.jl"
Date: Mon, 21 Oct 2024 17:53:25 +0000
Subject: [PATCH] build based on 6b5689a
---
dev/.documenter-siteinfo.json | 2 +-
dev/algorithmic_differentiation/index.html | 2 +-
dev/debug_mode/index.html | 4 ++--
dev/debugging_and_mwes/index.html | 4 ++--
dev/index.html | 2 +-
dev/known_limitations/index.html | 2 +-
dev/mathematical_interpretation/index.html | 6 +++---
dev/objects.inv | Bin 1613 -> 1613 bytes
dev/tools_for_rules/index.html | 8 ++++----
dev/understanding_intro/index.html | 2 +-
10 files changed, 16 insertions(+), 16 deletions(-)
diff --git a/dev/.documenter-siteinfo.json b/dev/.documenter-siteinfo.json
index 0fe238140..07d67615a 100644
--- a/dev/.documenter-siteinfo.json
+++ b/dev/.documenter-siteinfo.json
@@ -1 +1 @@
-{"documenter":{"julia_version":"1.11.1","generation_timestamp":"2024-10-21T17:22:57","documenter_version":"1.7.0"}}
\ No newline at end of file
+{"documenter":{"julia_version":"1.11.1","generation_timestamp":"2024-10-21T17:53:19","documenter_version":"1.7.0"}}
\ No newline at end of file
diff --git a/dev/algorithmic_differentiation/index.html b/dev/algorithmic_differentiation/index.html
index 219212e6c..96761b76c 100644
--- a/dev/algorithmic_differentiation/index.html
+++ b/dev/algorithmic_differentiation/index.html
@@ -19,4 +19,4 @@
D f [x] (\dot{x}) &= [(D \mathcal{l} [g(x)]) \circ (D g [x])](\dot{x}) \nonumber \\
&= \langle \bar{y}, D g [x] (\dot{x}) \rangle \nonumber \\
&= \langle D g [x]^\ast (\bar{y}), \dot{x} \rangle, \nonumber
-\end{align}\]
from which we conclude that $D g [x]^\ast (\bar{y})$ is the gradient of the composition $l \circ g$ at $x$.
The consequence is that we can always view the computation performed by reverse-mode AD as computing the gradient of the composition of the function in question and an inner product with the argument to the adjoint.
The above shows that if $\mathcal{Y} = \RR$ and $g$ is the function we wish to compute the gradient of, we can simply set $\bar{y} = 1$ and compute $D g [x]^\ast (\bar{y})$ to obtain the gradient of $g$ at $x$.
This document explains the core mathematical foundations of AD. It explains separately what is does, and how it goes about it. Some basic examples are given which show how these mathematical foundations can be applied to differentiate functions of matrices, and Julia functions.
Subsequent sections will build on these foundations, to provide a more general explanation of what AD looks like for a Julia programme.
Forwards-mode AD achieves this by breaking down $f$ into the composition $f = f_N \circ \dots \circ f_1$, where each $f_n$ is a simple function whose derivative (function) $D f_n [x_n]$ we know for any given $x_n$. By the chain rule, we have that
\[D f [x] (\dot{x}) = D f_N [x_N] \circ \dots \circ D f_1 [x_1] (\dot{x})\]
which suggests the following algorithm:
let $x_1 = x$, $\dot{x}_1 = \dot{x}$, and $n = 1$
let $\dot{x}_{n+1} = D f_n [x_n] (\dot{x}_n)$
let $x_{n+1} = f(x_n)$
let $n = n + 1$
if $n = N+1$ then return $\dot{x}_{N+1}$, otherwise go to 2.
When each function $f_n$ maps between Euclidean spaces, the applications of derivatives $D f_n [x_n] (\dot{x}_n)$ are given by $J_n \dot{x}_n$ where $J_n$ is the Jacobian of $f_n$ at $x_n$.
[1]
M. Giles. An extended collection of matrix derivative results for forward and reverse mode automatic differentiation. Unpublished (2008).
[2]
T. P. Minka. Old and new matrix algebra useful for statistics. See www. stat. cmu. edu/minka/papers/matrix. html 4 (2000).
note_for_geometersin AD we only really need to discuss differentiatiable functions between vector spaces that are isomorphic to Euclidean space. Consequently, a variety of considerations which are usually required in differential geometry are not required here. Notably, the tangent space is assumed to be the same everywhere, and to be the same as the domain of the function. Avoiding these additional considerations helps keep the mathematics as simple as possible.
Settings
This document was generated with Documenter.jl version 1.7.0 on Monday 21 October 2024. Using Julia version 1.11.1.