From 86f30fb32081ae03166e515210b9cfb9e2952e91 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=C5=81ukasz=20Zalewski?= Date: Tue, 28 Sep 2021 20:07:40 +0200 Subject: [PATCH] Release/1.1 (#174) * introduce different running modes: default, debug, experiment * fix pytorch installation in setup_conda.sh * fix incorrect calculation of precision, recall and f1 score in wandb callback * add `_self_` to config.yaml for compatibility with hydra1.1 * fix setting seed in `train.py` so it's skipped when `seed=null` * add exception message when trying to use wandb callbacks with `trainer.fast_dev_run=true` * change `axis=-1` to `dim=-1` in LogImagePredictions callback * add 'Reproducibilty' section to README.md --- README.md | 61 +++++++++++++----------- bash/setup_conda.sh | 37 ++++++++------ configs/config.yaml | 19 ++++---- configs/experiment/example_simple.yaml | 2 +- configs/hparams_search/mnist_optuna.yaml | 1 - configs/hydra/default.yaml | 12 ----- configs/logger/comet.yaml | 4 +- configs/logger/csv.yaml | 2 +- configs/logger/many_loggers.yaml | 1 - configs/logger/mlflow.yaml | 2 +- configs/logger/neptune.yaml | 2 +- configs/logger/tensorboard.yaml | 2 +- configs/logger/wandb.yaml | 2 +- configs/mode/debug.yaml | 15 ++++++ configs/mode/default.yaml | 11 +++++ configs/mode/exp.yaml | 15 ++++++ src/callbacks/wandb_callbacks.py | 13 +++-- src/train.py | 2 +- src/utils/utils.py | 9 +++- 19 files changed, 131 insertions(+), 81 deletions(-) delete mode 100644 configs/hydra/default.yaml create mode 100644 configs/mode/debug.yaml create mode 100644 configs/mode/default.yaml create mode 100644 configs/mode/exp.yaml diff --git a/README.md b/README.md index cbdc7a3c7..1fbd371a4 100644 --- a/README.md +++ b/README.md @@ -74,7 +74,7 @@ The directory structure of new project looks like this: │ ├── datamodule <- Datamodule configs │ ├── experiment <- Experiment configs │ ├── hparams_search <- Hyperparameter search configs -│ ├── hydra <- Hydra related configs +│ ├── mode <- Running mode configs │ ├── logger <- Logger configs │ ├── model <- Model configs │ ├── trainer <- Trainer configs @@ -150,7 +150,6 @@ python run.py trainer.max_epochs=20 model.lr=1e-4 > You can also add new parameters with `+` sign. ```yaml python run.py +model.new_param="uwu" - ``` @@ -217,6 +216,21 @@ python run.py logger=wandb +
+Use different logging modes + +```yaml +# debug mode changes logging folder to `logs/debug/` +python run.py mode=debug + +# experiment mode changes logging folder to `logs/experiments/name_of_your_experiment/` +# also sets custom experiment name in the logger +python run.py mode=exp name='my_new_experiment_253' +``` + +
+ +
Train model with chosen experiment config @@ -269,7 +283,7 @@ python run.py +trainer.max_time="00:12:00:00" ```yaml # run 1 train, val and test loop, using only 1 batch -python run.py debug=true +python run.py trainer.fast_dev_run=true # print full weight summary of all PyTorch modules python run.py trainer.weights_summary="full" @@ -348,12 +362,12 @@ python run.py -m 'experiment=glob(*)'
-
+
@@ -433,7 +447,7 @@ defaults: - callbacks: default.yaml # set this to null if you don't want to use callbacks - logger: null # set logger here or use command line (e.g. `python run.py logger=wandb`) - - hydra: default.yaml + - mode: default.yaml - experiment: null - hparams_search: null @@ -598,13 +612,13 @@ By default, logs have the following structure: │ ``` -You can change this structure by modifying paths in [hydra configuration](configs/hydra/default.yaml). +You can change this structure by modifying paths in [hydra configuration](configs/mode).

### Experiment Tracking PyTorch Lightning supports the most popular logging frameworks:
-**[Weights&Biases](https://www.wandb.com/) · [Neptune](https://neptune.ai/) · [Comet](https://www.comet.ml/) · [MLFlow](https://mlflow.org) · [Aim](https://github.com/aimhubio/aim) · [Tensorboard](https://www.tensorflow.org/tensorboard/)** +**[Weights&Biases](https://www.wandb.com/) · [Neptune](https://neptune.ai/) · [Comet](https://www.comet.ml/) · [MLFlow](https://mlflow.org) · [Tensorboard](https://www.tensorflow.org/tensorboard/)** These tools help you keep track of hyperparameters and output metrics and allow you to compare and visualize results. To use one of them simply complete its configuration in [configs/logger](configs/logger) and run: ```yaml @@ -684,7 +698,7 @@ hydra:
Next, you can execute it with: `python run.py -m hparams_search=mnist_optuna`
-Using this approach doesn't require you to add any boilerplate into your pipeline, everything is defined in a single config file. You can use different optimization frameworks integrated with Hydra, like Optuna, Ax or Nevergrad. +Using this approach doesn't require you to add any boilerplate into your pipeline, everything is defined in a single config file. You can use different optimization frameworks integrated with Hydra, like Optuna, Ax or Nevergrad. The `optimization_results.yaml` will be available under `logs/multirun` folder.

@@ -801,23 +815,14 @@ python run.py trainer.gpus=4 +trainer.accelerator="ddp"

-### Extra Features -List of extra utilities available in the template: -- loading environment variables from [.env](.env.example) file -- pretty printing config with [Rich](https://github.com/willmcgugan/rich) library -- disabling python warnings -- debug mode - - -You can easily remove any of those by modifying [run.py](run.py) and [src/train.py](src/train.py). +### Reproducibility +To reproduce previous experiment, simply load its config from logs: +```yaml +python run.py --config-path /logs/runs/.../.hydra/ --config-name config.yaml +``` +The `config.yaml` from `.hydra` folder contains all overriden parameters and sections.

- - ## Best Practices