AbstractDifferentiation.jl

[mateuszbaran]

TODO: check if https://github.com/JuliaDiff/AbstractDifferentiation.jl can be used for specifying differentiation backend (also relevant to Manopt.jl).

[kellertuer]

Yes, this is very interesting. We also have to discuss, where to put AD stuff (here or into Manopt) as well as for Hessians for example how/where to convert from the euclidean hessian to the Riemannian one.

To elaborate: The Matlab package has a lot of these functions, see for example

https://github.com/NicolasBoumal/manopt/blob/fd98d5417c7084f9bfdf8b086fbdd66799fafca5/manopt/manifolds/sphere/spherefactory.m#L122-L127

Where the gradient is fine with me, you have to project the euclidean gradient onto the tangent plane to get the Riemannian one. This might not always be the case, for example for fixed rank matrices, the euclidean one is one matrix, but turning it into the Riemannian one also includes changing the representation accordingly – though this will be done by project, too, for the gradient.

For the hessian we might have to check how to cover this exactly, because it would be nice if one could just take a function from the embedding (say R^{m,n}, the set of all matrices) do Riemannian AD and obtain a Riemannian Hessian.

[mateuszbaran]

Hessians were discussed a while ago here: JuliaManifolds/Manifolds.jl#202 (comment) . I still think Weingarten maps are a good idea (at least as the default mechanism, it should also be possible to implement Riemannian Hessians without Weingarten maps).

I'm not sure where AD support should be put, maybe in a new package?

[kellertuer]

Sure, an extra package would also be possible; keeps the library of manifolds cleaner and makes AD independent of the optimisation.