From e83d30f7fb059f6687674363e3337810d1319471 Mon Sep 17 00:00:00 2001
From: "Documenter.jl"
Date: Tue, 22 Oct 2024 05:37:54 +0000
Subject: [PATCH] build based on 59f2cff
---
previews/PR302/.documenter-siteinfo.json | 2 +-
previews/PR302/algorithmic_differentiation/index.html | 2 +-
previews/PR302/debug_mode/index.html | 4 ++--
previews/PR302/debugging_and_mwes/index.html | 4 ++--
previews/PR302/index.html | 2 +-
previews/PR302/known_limitations/index.html | 2 +-
previews/PR302/mathematical_interpretation/index.html | 6 +++---
previews/PR302/tools_for_rules/index.html | 8 ++++----
previews/PR302/understanding_intro/index.html | 2 +-
9 files changed, 16 insertions(+), 16 deletions(-)
diff --git a/previews/PR302/.documenter-siteinfo.json b/previews/PR302/.documenter-siteinfo.json
index 77aa7857f..1f761fdaa 100644
--- a/previews/PR302/.documenter-siteinfo.json
+++ b/previews/PR302/.documenter-siteinfo.json
@@ -1 +1 @@
-{"documenter":{"julia_version":"1.11.1","generation_timestamp":"2024-10-21T22:19:18","documenter_version":"1.7.0"}}
\ No newline at end of file
+{"documenter":{"julia_version":"1.11.1","generation_timestamp":"2024-10-22T05:37:47","documenter_version":"1.7.0"}}
\ No newline at end of file
diff --git a/previews/PR302/algorithmic_differentiation/index.html b/previews/PR302/algorithmic_differentiation/index.html
index 35b2ef4bb..56e36d121 100644
--- a/previews/PR302/algorithmic_differentiation/index.html
+++ b/previews/PR302/algorithmic_differentiation/index.html
@@ -19,4 +19,4 @@
D f [x] (\dot{x}) &= [(D \mathcal{l} [g(x)]) \circ (D g [x])](\dot{x}) \nonumber \\
&= \langle \bar{y}, D g [x] (\dot{x}) \rangle \nonumber \\
&= \langle D g [x]^\ast (\bar{y}), \dot{x} \rangle, \nonumber
-\end{align}\]
from which we conclude that $D g [x]^\ast (\bar{y})$ is the gradient of the composition $l \circ g$ at $x$.
The consequence is that we can always view the computation performed by reverse-mode AD as computing the gradient of the composition of the function in question and an inner product with the argument to the adjoint.
The above shows that if $\mathcal{Y} = \RR$ and $g$ is the function we wish to compute the gradient of, we can simply set $\bar{y} = 1$ and compute $D g [x]^\ast (\bar{y})$ to obtain the gradient of $g$ at $x$.
This document explains the core mathematical foundations of AD. It explains separately what is does, and how it goes about it. Some basic examples are given which show how these mathematical foundations can be applied to differentiate functions of matrices, and Julia functions.
Subsequent sections will build on these foundations, to provide a more general explanation of what AD looks like for a Julia programme.
Forwards-mode AD achieves this by breaking down $f$ into the composition $f = f_N \circ \dots \circ f_1$, where each $f_n$ is a simple function whose derivative (function) $D f_n [x_n]$ we know for any given $x_n$. By the chain rule, we have that
\[D f [x] (\dot{x}) = D f_N [x_N] \circ \dots \circ D f_1 [x_1] (\dot{x})\]
which suggests the following algorithm:
let $x_1 = x$, $\dot{x}_1 = \dot{x}$, and $n = 1$
let $\dot{x}_{n+1} = D f_n [x_n] (\dot{x}_n)$
let $x_{n+1} = f(x_n)$
let $n = n + 1$
if $n = N+1$ then return $\dot{x}_{N+1}$, otherwise go to 2.
When each function $f_n$ maps between Euclidean spaces, the applications of derivatives $D f_n [x_n] (\dot{x}_n)$ are given by $J_n \dot{x}_n$ where $J_n$ is the Jacobian of $f_n$ at $x_n$.
[1]
M. Giles. An extended collection of matrix derivative results for forward and reverse mode automatic differentiation. Unpublished (2008).
[2]
T. P. Minka. Old and new matrix algebra useful for statistics. See www. stat. cmu. edu/minka/papers/matrix. html 4 (2000).
note_for_geometersin AD we only really need to discuss differentiatiable functions between vector spaces that are isomorphic to Euclidean space. Consequently, a variety of considerations which are usually required in differential geometry are not required here. Notably, the tangent space is assumed to be the same everywhere, and to be the same as the domain of the function. Avoiding these additional considerations helps keep the mathematics as simple as possible.
Settings
This document was generated with Documenter.jl version 1.7.0 on Monday 21 October 2024. Using Julia version 1.11.1.