Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Added Wandb For Better Training Log Visualization #661

Open
wants to merge 2 commits into
base: master
Choose a base branch
from
Open

Added Wandb For Better Training Log Visualization #661

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
8d6f6b5
added wandb support to train script
Jun 21, 2021
e39ae87
updated readme for wandb installation
Jun 21, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion readme.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -82,7 +82,7 @@ The speed/FPS test includes the time of post-processing with no jit/data precisi
## Demo

# install requirements
pip install pycocotools numpy opencv-python tqdm tensorboard tensorboardX pyyaml webcolors
pip install pycocotools numpy opencv-python tqdm tensorboard tensorboardX pyyaml webcolors wandb
pip install torch==1.4.0
pip install torchvision==0.5.0

Expand Down
16 changes: 15 additions & 1 deletion train.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,7 @@
import numpy as np
import torch
import yaml
import wandb
from tensorboardX import SummaryWriter
from torch import nn
from torch.utils.data import DataLoader
Expand Down Expand Up @@ -173,6 +174,7 @@ def freeze_backbone(m):
else:
use_sync_bn = False

wandb.init() # initialize a wandb run
writer = SummaryWriter(opt.log_path + f'/{datetime.datetime.now().strftime("%Y%m%d-%H%M%S")}/')

# warp the model with loss function, to reduce the memory usage on gpu0 and speedup
Expand Down Expand Up @@ -248,7 +250,13 @@ def freeze_backbone(m):
# log learning_rate
current_lr = optimizer.param_groups[0]['lr']
writer.add_scalar('learning_rate', current_lr, step)

wandb.log({"step" : step,
"epoch": epoch,
"train-loss": loss,
"train-reg-loss": reg_loss,
"train-cls-loss" : cls_loss,
"lr" : current_lr
})
step += 1

if step % opt.save_interval == 0 and step > 0:
Expand Down Expand Up @@ -296,6 +304,12 @@ def freeze_backbone(m):
writer.add_scalars('Regression_loss', {'val': reg_loss}, step)
writer.add_scalars('Classfication_loss', {'val': cls_loss}, step)

wandb.log({"step" : step,
"val-loss" : loss,
"val-reg-loss" : reg_loss,
"val-cls-loss" : cls_loss,
"epoch" : epoch})

if loss + opt.es_min_delta < best_loss:
best_loss = loss
best_epoch = epoch
Expand Down