Deep Learning examples in Pytorch

SimCLR

Simple Pytorch implementation in src/simclr/simclr.py.
An example notebook is provided where we look at a simplistic contrastive learning task, distinguishing points on the unit circle (available here and in Google Colab ).
I wrote this to illustrate my recent blog post about SimCLR.
The original paper is available here.

---

ResNet re-implementations

The code in src/models/resnet.py is a re-implementation of the original ResNet paper.

Next steps are implementing ideas from:

• The "Bag of tricks" paper
• The "revisiting ResNets" paper

Results on CIFAR-10's test set:

Setup	Accuracy
Baseline	0.915
Baseline+ResNet simple tweaks	0.915

Baseline:

• 200 epochs
• learning rate 0.1
• cosine decay
• linear warmup of 5 epochs
• batch size 128
• weight decay 1e-4

ResNet simple tweaks:

• better downsampling, instead of stride 2 1x1 convolutions
• 3 convolutions 3x3 instead of a single 7x7 convolution in the stem

TODOs:

• More training epochs with / without MixUp
• Label smoothing
• Reduce weight decay