ENH: add adaptive optimizers for all mappings #315

nghi-truyen · 2024-09-10T14:55:16Z

Suggested reviewer order: FC, PAG.
Adaptive optimizers (i.e. adam, adagrad, sgd, and rmsprop) are now possible for all mappings. This PR results in the following important changes:

Key "epoch" for optimization with ANN is deprecated and replaced by "maxiter" (since the optim process can be stopped before the maximum number of iterations using early stopping method) for the consistency with other optimizers
The verbose of the cost and projected gradient are changed to a scientific presentation (i.e. using the "e" format instead of "f") + update documentation
Fix the display of the cost value at the last iteration in case of SBS. The last iteration was displayed inconsistant value in case of having convergence using SBS optimizer
Split between tools and principal functions in the optimize.py file
Remove smash/fcore/external/lbfgsb.f
Merge regionalization NN parameters into the control vector return
Keys iter_cost and iter_projg are deprecated. The idea is to keep the consistency for all mappings by returning the optimization results only at the last optimization process. To access to intermediate results, the next PR would address this issue Add a callback signature into the optimization methods to get intermediate results #252
Add new tests for optimization with various optimizers and mappings
The baseline is regenerated. The changes for the return of control vector value is due to the fact that the control vector is now directly returned by the Python routine, rather than being passed to the Fortran wrapper that causes precision problems (e.g. ret["control_vector"] = res.x instead of ret["control_vector"] =model._parameters.control.x.copy())

…enh-net-forward-pass

…o enh-combine-optimizers

…pings

…izer

…tead of nan for optimize control info with net

…pings

MAINT: merge branch main into enh-adaptive-opt-for-all-mappings

…zers

…am/main' into enh-adaptive-optimizers

inoelloc

Voila une première passe de relecture. On peut attendre la review de @pag13 pour après échanger sur Skype pour finaliser tout ca !

smash/_constant.py

smash/core/signal_analysis/evaluation/evaluation.py

smash/core/simulation/_doc.py

smash/factory/net/_loss.py

inoelloc · 2024-09-11T12:04:56Z

smash/factory/net/net.py

@@ -532,94 +531,111 @@ def _fit_d2p(
            random_state=random_state,
        )

+        # % First evaluation
+        # calculate the gradient of J wrt rr_parameters (output of the regionalization NN)


pas que rr_parameters en soit ?

# calculate the gradient of J wrt rr_parameters (output of the regionalization NN) # and get the gradient of the parameterization NN if used init_cost_grad, nn_parameters_b = _hcost_prime( self, x_train, model_params_states, instance, parameters, wrap_options, wrap_returns )

J'ai fait pour les deux gradients en fait.

JE voulais dire, aussi rr_initial_states non ?

smash/factory/net/net.py

smash/fcore/routine/mwd_parameters_manipulation.f90

smash/core/simulation/optimize/_tools.py

pag13

Travail impressionnant et très rigoureux Truyen, bravo ! Merci pour la bonne review François.

doc/source/user_guide/quickstart/cance_first_simulation.rst

smash/_constant.py

smash/core/simulation/optimize/optimize.py

smash/factory/net/_loss.py

smash/factory/net/net.py

smash/fcore/routine/mwd_parameters_manipulation.f90

inoelloc

Tu as fait un commit pdt la review. Je sais pas vraiment ce qui va s'afficher ^^"
Le seul point de modification important c'est sur le suffix _reg. On en rediscute

smash/core/simulation/_doc.py

smash/core/simulation/optimize/_tools.py

smash/core/simulation/optimize/optimize.py

smash/factory/net/_loss.py

smash/core/simulation/estimate/_tools.py

inoelloc · 2024-09-12T09:14:10Z

smash/factory/net/net.py

@@ -532,94 +531,111 @@ def _fit_d2p(
            random_state=random_state,
        )

+        # % First evaluation
+        # calculate the gradient of J wrt rr_parameters (output of the regionalization NN)


JE voulais dire, aussi rr_initial_states non ?

inoelloc

Bon pour moi merci !

inoelloc · 2024-09-12T15:29:31Z

smash/factory/net/net.py

+        # calculate the gradient of J wrt rr_parameters and rr_initial_states
+        # that are the output of the descriptors-to-parameters (d2p) NN
+        # and get the gradient of the pmtz NN (pmtz) if used


nghi-truyen added 30 commits July 20, 2024 00:46

Merge remote-tracking branch 'upstream/main' into enh-combine-optimizers

b58ec06

initalize branch combine optimizers

233a8cc

Merge branch 'enh-hybrid-models' into enh-combine-optimizers

90afd60

Merge remote-tracking branch 'upstream/enh-cnn-regionalization' into …

c70db0c

…enh-net-forward-pass

ENH: add set, get and forward_pass methods for Net

30eb4c5

ENH: add test net forward pass + fix doc net

e4b06e8

Merge branch 'enh-hybrid-models' into enh-combine-optimizers

75a847c

Merge branch 'enh-net-forward-pass' into enh-combine-optimizers

91bed2f

Fix indent docstring net

3d71aec

Merge branch 'enh-net-forward-pass' into enh-combine-optimizers

b8dc977

Merge branch 'enh-hybrid-models' into enh-net-forward-pass

63a49b7

add weight/bias_shape into trainable layer

b4c5dfc

add weight/bias_shape into trainable layer

ee7824d

Remove redundant in Python function sbs_optimize

0ffd8ad

Merge remote-tracking branch 'upstream/enh-cnn-regionalization' into …

7f5f9ec

…enh-net-forward-pass

Merge branch 'enh-hybrid-models' into enh-net-forward-pass

41a7aa6

adaptive optimizers for all mappings

91429b2

Merge remote-tracking branch 'upstream/move-optimizers-to-Python' int…

e1a5a64

…o enh-combine-optimizers

finalize optimizer combination

7b5b069

Fix parameter update when using early stopping + dic api doc optimize

268fe5e

Correct comment typo

b14c4af

Fix comment

3b60a04

Merge branch 'enh-net-forward-pass' into enh-adaptive-opt-for-all-map…

9fc544b

…pings

Merge branch 'enh-hybrid-models' into enh-net-forward-pass

873302c

Merge branch 'enh-net-forward-pass' into enh-adaptive-opt-for-all-map…

a371562

…pings

Merge branch 'enh-hybrid-models' into enh-net-forward-pass

a189bc3

Merge branch 'enh-net-forward-pass' into enh-adaptive-opt-for-all-map…

a577e2b

…pings

Change x_train name to x in net

47bda59

Merge branch 'enh-net-forward-pass' into enh-adaptive-opt-for-all-map…

d007323

…pings

retrieve key x in control_info + fix api doc for control prior name

834fa73

nghi-truyen and others added 11 commits August 6, 2024 23:03

Fix api doc cance

700d582

FIX: make check

c1e1bca

FIX/ENH: fix doc default_optimize + generic doc for mapping and optim…

4166e0e

…izer

MAINT: remove see also default_optimize doc

14c6cd2

FIX: returns control with random values depending on random_state ins…

23df534

…tead of nan for optimize control info with net

Merge branch 'enh-hybrid-models' into enh-net-forward-pass

1b40d8d

Merge branch 'enh-net-forward-pass' into enh-adaptive-opt-for-all-map…

fceee65

…pings

MAINT: merge branch main into enh-adaptive-opt-for-all-mappings

ebe3275

Merge pull request #8 from nghi-truyen/merge-adaptive-opt

1f6798e

MAINT: merge branch main into enh-adaptive-opt-for-all-mappings

Re-generate baseline and fix unittest

ab49571

Merge remote-tracking branch 'upstream/main' into enh-adaptive-optimi…

dbcec34

…zers

nghi-truyen requested review from pag13 and inoelloc September 10, 2024 14:55

nghi-truyen added bug Something isn't working documentation Improvements or additions to documentation enhancement New feature or request labels Sep 10, 2024

nghi-truyen added this to the Release v1.1.0 milestone Sep 10, 2024

nghi-truyen linked an issue Sep 10, 2024 that may be closed by this pull request

Combine all optimizers from Fortran and Python #240

Closed

FIX PR: re-generate baseline and Merge remote-tracking branch 'upstre…

5976ba9

…am/main' into enh-adaptive-optimizers

inoelloc reviewed Sep 11, 2024

View reviewed changes

nghi-truyen commented Sep 11, 2024

View reviewed changes

smash/core/simulation/optimize/_tools.py Show resolved Hide resolved

DassHydro deleted a comment from inoelloc Sep 11, 2024

Apply suggestion changes from the first review of FC

73b7ffb

pag13 reviewed Sep 11, 2024

View reviewed changes

Apply suggestion changes from PAG and FC review

1bbb453

inoelloc reviewed Sep 12, 2024

View reviewed changes

Apply suggestion changes from FC second review

c8e3910

inoelloc approved these changes Sep 12, 2024

View reviewed changes

nghi-truyen merged commit 61ad424 into DassHydro:main Sep 12, 2024
22 checks passed

nghi-truyen deleted the enh-adaptive-optimizers branch September 12, 2024 16:39

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

ENH: add adaptive optimizers for all mappings #315

ENH: add adaptive optimizers for all mappings #315

nghi-truyen commented Sep 10, 2024 •

edited

Loading

inoelloc left a comment

inoelloc Sep 11, 2024

nghi-truyen Sep 11, 2024

inoelloc Sep 12, 2024

pag13 left a comment

inoelloc left a comment

inoelloc Sep 12, 2024

inoelloc left a comment

inoelloc Sep 12, 2024

ENH: add adaptive optimizers for all mappings #315

ENH: add adaptive optimizers for all mappings #315

Conversation

nghi-truyen commented Sep 10, 2024 • edited Loading

inoelloc left a comment

Choose a reason for hiding this comment

inoelloc Sep 11, 2024

Choose a reason for hiding this comment

nghi-truyen Sep 11, 2024

Choose a reason for hiding this comment

inoelloc Sep 12, 2024

Choose a reason for hiding this comment

pag13 left a comment

Choose a reason for hiding this comment

inoelloc left a comment

Choose a reason for hiding this comment

inoelloc Sep 12, 2024

Choose a reason for hiding this comment

inoelloc left a comment

Choose a reason for hiding this comment

inoelloc Sep 12, 2024

Choose a reason for hiding this comment

nghi-truyen commented Sep 10, 2024 •

edited

Loading