Materials/CNNs/transfer_learning_augmentations_oxford_pets.ipynb

{"nbformat":4,"nbformat_minor":0,"metadata":{"colab":{"name":"transfer_learning_augmentations_oxford_pets.ipynb","provenance":[],"collapsed_sections":[]},"kernelspec":{"name":"python3","display_name":"Python 3"},"accelerator":"GPU"},"cells":[{"cell_type":"markdown","metadata":{"id":"dPFybf1EPpLQ"},"source":["# Image Augmentation - tf.data - Transfer Learning\n","\n","This notebook demonstrate 3 concepts:\n","- Image augmentation using the Albumentations library\n","- tf.data API\n","- Transfer learning\n","\n","We will wrap all these concepts to train a pet classifier on the [Oxford pets dataset](https://www.robots.ox.ac.uk/~vgg/data/pets/)."]},{"cell_type":"markdown","metadata":{"id":"CKshwk3Glxb4"},"source":["## Imports "]},{"cell_type":"code","metadata":{"id":"KWeYX0oxJKbG"},"source":["import tensorflow as tf\n","import pathlib\n","import os\n","import glob\n","import matplotlib.pyplot as plt\n","import numpy as np\n","import PIL\n","from functools import partial\n","from sklearn.model_selection import train_test_split\n","from tqdm import tqdm_notebook as tqdm"],"execution_count":null,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"gjlY-fToVkU4"},"source":["We will use the github repo from to load the pre-trained models. Note that there are other sources where you can get pre-trained models. e.g.:\n","- tf.keras.applications has some pre-trained models\n","- https://github.com/qubvel/efficientnet has weights for efficient-net\n","- ...\n","\n","One of the most important thing to look at when doing transfer learning is to know if any pre-processing has been done in the original training. Most libraries will have a `preprocess_input` method or function which you should find!"]},{"cell_type":"code","metadata":{"id":"PvIO41BkOwG8","colab":{"base_uri":"https://localhost:8080/","height":34},"executionInfo":{"status":"ok","timestamp":1603389862769,"user_tz":240,"elapsed":6904,"user":{"displayName":"Yassine Yousfi","photoUrl":"","userId":"06317191099946336363"}},"outputId":"eb2f0910-3970-4f53-eea8-278ab71bd6f2"},"source":["!pip install -U --quiet git+https://github.com/qubvel/efficientnet"],"execution_count":null,"outputs":[{"output_type":"stream","text":["  Building wheel for efficientnet (setup.py) ... \u001b[?25l\u001b[?25hdone\n"],"name":"stdout"}]},{"cell_type":"markdown","metadata":{"id":"HM2LfC2vLY1n"},"source":["We will use the albumentations library which has an extensive collection of augmentation operations"]},{"cell_type":"code","metadata":{"id":"BpgMKqJvlX0F","colab":{"base_uri":"https://localhost:8080/","height":34},"executionInfo":{"status":"ok","timestamp":1603389874048,"user_tz":240,"elapsed":17030,"user":{"displayName":"Yassine Yousfi","photoUrl":"","userId":"06317191099946336363"}},"outputId":"df521575-39a4-419d-cb3c-a88fecceccff"},"source":["!pip install -U --quiet git+https://github.com/albumentations-team/albumentations"],"execution_count":null,"outputs":[{"output_type":"stream","text":["  Building wheel for albumentations (setup.py) ... \u001b[?25l\u001b[?25hdone\n"],"name":"stdout"}]},{"cell_type":"code","metadata":{"id":"sIs25jdfOtXZ"},"source":["import efficientnet.keras as efn \n","from albumentations import (\n","    Compose, RandomBrightnessContrast, OneOf, ShiftScaleRotate, HueSaturationValue, \n","    HorizontalFlip, ToGray, Resize, GaussNoise)"],"execution_count":null,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"Y3TL0H_fOi1Y"},"source":["The dataset we will use in this lecture (oxford pets) is not stored in a nice directory format. This is an opportunity to show how to create a `tf.data.Dataset` object from scratch."]},{"cell_type":"markdown","metadata":{"id":"fM5twhi3mL9L"},"source":["## Data"]},{"cell_type":"code","metadata":{"id":"ZE6_yskdy0eN","colab":{"base_uri":"https://localhost:8080/","height":50},"executionInfo":{"status":"ok","timestamp":1603385350455,"user_tz":240,"elapsed":86590,"user":{"displayName":"Yassine Yousfi","photoUrl":"","userId":"06317191099946336363"}},"outputId":"54a94035-048b-47a6-ffea-0cb417569d41"},"source":["dataset_url =\"https://www.robots.ox.ac.uk/~vgg/data/pets/data/images.tar.gz\"\n","data_dir = tf.keras.utils.get_file(origin=dataset_url, \n","                                   fname='images', \n","                                   untar=True,\n","                                   cache_dir='/content/')\n","\n","pathlib.Path('/content/datasets/images.tar.gz').unlink() "],"execution_count":null,"outputs":[{"output_type":"stream","text":["Downloading data from https://www.robots.ox.ac.uk/~vgg/data/pets/data/images.tar.gz\n","791920640/791918971 [==============================] - 67s 0us/step\n"],"name":"stdout"}]},{"cell_type":"code","metadata":{"id":"24cEpOHMzIOg","colab":{"base_uri":"https://localhost:8080/","height":34},"executionInfo":{"status":"ok","timestamp":1603385350456,"user_tz":240,"elapsed":86417,"user":{"displayName":"Yassine Yousfi","photoUrl":"","userId":"06317191099946336363"}},"outputId":"82597ac4-508d-4d50-b0fd-bc671de42494"},"source":["data_dir = pathlib.Path(data_dir)\n","images_list = data_dir.glob('*.jpg')\n","images_list = [str(x) for x in images_list]\n","np.random.shuffle(images_list)\n","def get_label_from_path(path):\n","  label = '_'.join(path.split(os.sep)[-1].split('_')[:-1]).lower()\n","  return label\n","labels_list = [get_label_from_path(x) for x in images_list]\n","labels = list(set(labels_list))\n","labels_to_num = dict(zip(labels, range(len(labels))))\n","labels_list = [labels_to_num[x] for x in labels_list]\n","print(len(labels))"],"execution_count":null,"outputs":[{"output_type":"stream","text":["37\n"],"name":"stdout"}]},{"cell_type":"code","metadata":{"id":"Vc-L9WCvmuNE","colab":{"base_uri":"https://localhost:8080/","height":118},"executionInfo":{"status":"ok","timestamp":1603385350456,"user_tz":240,"elapsed":86136,"user":{"displayName":"Yassine Yousfi","photoUrl":"","userId":"06317191099946336363"}},"outputId":"c54df0ab-736f-400d-d7b1-a534356961b2"},"source":["images_list[:5], labels_list[:5]"],"execution_count":null,"outputs":[{"output_type":"execute_result","data":{"text/plain":["(['/content/datasets/images/Persian_41.jpg',\n","  '/content/datasets/images/Russian_Blue_94.jpg',\n","  '/content/datasets/images/beagle_30.jpg',\n","  '/content/datasets/images/pomeranian_147.jpg',\n","  '/content/datasets/images/english_setter_116.jpg'],\n"," [34, 16, 28, 14, 23])"]},"metadata":{"tags":[]},"execution_count":54}]},{"cell_type":"code","metadata":{"id":"OCUvCFJk0q4D","colab":{"base_uri":"https://localhost:8080/","height":34},"executionInfo":{"status":"ok","timestamp":1603385350457,"user_tz":240,"elapsed":84987,"user":{"displayName":"Yassine Yousfi","photoUrl":"","userId":"06317191099946336363"}},"outputId":"793ef71b-3143-4bd9-bc82-8f1a7d17d607"},"source":["# Data source = list of paths + labels\n","all_data = tf.data.Dataset.from_tensor_slices((images_list, labels_list))\n","print(len(all_data))"],"execution_count":null,"outputs":[{"output_type":"stream","text":["7390\n"],"name":"stdout"}]},{"cell_type":"code","metadata":{"id":"QmXNUsBh0zB5"},"source":["data_train = all_data.take(5000) # 5000\n","data_val = all_data.skip(5000) # 2390"],"execution_count":null,"outputs":[]},{"cell_type":"code","metadata":{"id":"PJkFxhJ-q761"},"source":["BATCH_SIZE = 32\n","IMG_SIZE = (160, 160)"],"execution_count":null,"outputs":[]},{"cell_type":"code","metadata":{"id":"dOpoVI_qlJLN"},"source":["transforms_train = Compose([\n","            Resize(IMG_SIZE[0], IMG_SIZE[1], p=1),\n","            GaussNoise(p=0.1),\n","            OneOf([RandomBrightnessContrast(), HueSaturationValue()], p=0.7),\n","            HorizontalFlip(p=0.5),\n","            ToGray(p=0.2),\n","            ShiftScaleRotate(shift_limit=0.1, scale_limit=0.2, rotate_limit=30, p=0.5)\n","        ])\n","\n","transforms_val = Compose([\n","            Resize(IMG_SIZE[0], IMG_SIZE[1], p=1)\n","        ])\n","\n","def aug_fn(image, train):\n","    data = {\"image\": image}\n","    if train:\n","      data = transforms_train(**data)\n","    else:\n","      data = transforms_val(**data)\n","    aug_img = data[\"image\"]\n","    aug_img = efn.preprocess_input(aug_img) # Don't forget the preprocess function!\n","    return aug_img\n","\n","def parse(path, label):\n","    image = tf.io.read_file(path)\n","    image = tf.io.decode_jpeg(image, channels=3)\n","    label = tf.one_hot(label, 37)\n","    return image, label\n","\n","def process_data_train(image, label):\n","    aug_img = tf.numpy_function(func=aug_fn, inp=[image, True], \n","                                Tout=tf.float32)\n","    return aug_img, label\n","\n","def process_data_val(image, label):\n","    aug_img = tf.numpy_function(func=aug_fn, inp=[image, False], \n","                                Tout=tf.float32)\n","    return aug_img, label"],"execution_count":null,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"mFy6FdlMWCqw"},"source":["`tf.data.Dataset` have 5 important methods:\n","- shuffle: shuffles the data (not globally, but keeps a buffer_size of shuffled data)\n","- map: maps a function to the dataset, i.e. execute on it every element, but not when you run the cell, but during training. \n","- cache: puts the dataset in a cache during the first epoch, and re-uses the cache for the next epochs. Only use for small datasets ... /!\\ do not cache image augmentation and batching\n","- batch: creates a batch of desired size\n","- prefetch: populates an ordered buffer even while the forward/nackward pass are running (on GPU)\n","\n","https://www.tensorflow.org/guide/data_performance\n","\n","We will use `tf.data.experimental.AUTOTUNE` to auto-tune the prefetching and parllel calls. Tensorflow will try a few values and automatically set the best parameters to optimize performance."]},{"cell_type":"code","metadata":{"id":"wCU08n6qlG_N"},"source":["from tensorflow.data.experimental import AUTOTUNE\n","\n","data_train = data_train.shuffle(buffer_size=BATCH_SIZE*4).map(parse, \n","                  num_parallel_calls=AUTOTUNE).cache().map(process_data_train,\n","                  num_parallel_calls=AUTOTUNE).batch(BATCH_SIZE).prefetch(AUTOTUNE)\n","                  \n","data_val = data_val.shuffle(buffer_size=BATCH_SIZE*4).map(parse, \n","                  num_parallel_calls=AUTOTUNE).cache().map(process_data_val,\n","                  num_parallel_calls=AUTOTUNE).batch(BATCH_SIZE).prefetch(AUTOTUNE)"],"execution_count":null,"outputs":[]},{"cell_type":"code","metadata":{"id":"jA9CjyUscQ1d","colab":{"base_uri":"https://localhost:8080/","height":981,"output_embedded_package_id":"16x6aErlr8JCSkFZONKi3zbrtCR6cCLRh"},"executionInfo":{"status":"ok","timestamp":1603385360133,"user_tz":240,"elapsed":79384,"user":{"displayName":"Yassine Yousfi","photoUrl":"","userId":"06317191099946336363"}},"outputId":"13624418-cc4d-4633-e3c8-c96cc3b416d2"},"source":["def view_image_batch(ds):\n","    mean = [0.485, 0.456, 0.406]\n","    std = [0.229, 0.224, 0.225]\n","    image, _ = next(iter(ds)) # extract 1 batch from the dataset\n","    image = np.clip((image.numpy()*std)+mean,0,1) # Doing the efn.preprocess_input inverse \n","\n","    fig = plt.figure(figsize=(22, 22))\n","    for i in range(20):\n","        ax = fig.add_subplot(4, 5, i+1, xticks=[], yticks=[])\n","        ax.imshow(image[i])\n","\n","# Look at a batch\n","view_image_batch(data_train)"],"execution_count":null,"outputs":[{"output_type":"display_data","data":{"text/plain":"Output hidden; open in https://colab.research.google.com to view."},"metadata":{}}]},{"cell_type":"markdown","metadata":{"id":"B-jiUw3FmSEr"},"source":["## Model"]},{"cell_type":"code","metadata":{"id":"ubd3B4Eh7JBr","colab":{"base_uri":"https://localhost:8080/","height":319},"executionInfo":{"status":"ok","timestamp":1603377250966,"user_tz":240,"elapsed":3858,"user":{"displayName":"Yassine Yousfi","photoUrl":"","userId":"06317191099946336363"}},"outputId":"49d97dde-d01f-4649-a314-f1ab5ecdfd50"},"source":["IMG_SHAPE = IMG_SIZE + (3,)\n","\n","base_model = efn.EfficientNetB0(input_shape=IMG_SHAPE, weights='imagenet', include_top=False)\n","\n","base_model.trainable = False\n","\n","inputs = tf.keras.Input(shape=IMG_SHAPE)\n","x = base_model(inputs, training=False) # BN Will always be in inference mode, i.e. using running stats. Dropout too.\n","x = tf.keras.layers.GlobalAveragePooling2D()(x)\n","x = tf.keras.layers.Dropout(0.2)(x)  \n","outputs = tf.keras.layers.Dense(37)(x)\n","model = tf.keras.Model(inputs, outputs)\n","\n","model.summary()"],"execution_count":null,"outputs":[{"output_type":"stream","text":["Model: \"functional_9\"\n","_________________________________________________________________\n","Layer (type)                 Output Shape              Param #   \n","=================================================================\n","input_10 (InputLayer)        [(None, 160, 160, 3)]     0         \n","_________________________________________________________________\n","efficientnet-b0 (Functional) (None, 5, 5, 1280)        4049564   \n","_________________________________________________________________\n","global_average_pooling2d_4 ( (None, 1280)              0         \n","_________________________________________________________________\n","dropout_4 (Dropout)          (None, 1280)              0         \n","_________________________________________________________________\n","dense_4 (Dense)              (None, 37)                47397     \n","=================================================================\n","Total params: 4,096,961\n","Trainable params: 47,397\n","Non-trainable params: 4,049,564\n","_________________________________________________________________\n"],"name":"stdout"}]},{"cell_type":"code","metadata":{"id":"3l7QDSahnMPd","colab":{"base_uri":"https://localhost:8080/","height":390},"executionInfo":{"status":"ok","timestamp":1603377382582,"user_tz":240,"elapsed":118835,"user":{"displayName":"Yassine Yousfi","photoUrl":"","userId":"06317191099946336363"}},"outputId":"db2dbce1-8b3b-4d41-a201-afb9b85a361c"},"source":["model.compile(optimizer=tf.keras.optimizers.Adam(),\n","              loss=tf.keras.losses.CategoricalCrossentropy(from_logits=True),\n","              metrics=['accuracy'])\n","\n","model.fit(x=data_train, validation_data=data_val, epochs=10)"],"execution_count":null,"outputs":[{"output_type":"stream","text":["Epoch 1/10\n","157/157 [==============================] - 12s 76ms/step - loss: 1.4801 - accuracy: 0.6644 - val_loss: 0.5818 - val_accuracy: 0.8640\n","Epoch 2/10\n","157/157 [==============================] - 11s 70ms/step - loss: 0.5937 - accuracy: 0.8432 - val_loss: 0.4204 - val_accuracy: 0.8845\n","Epoch 3/10\n","157/157 [==============================] - 11s 72ms/step - loss: 0.4600 - accuracy: 0.8646 - val_loss: 0.3755 - val_accuracy: 0.8866\n","Epoch 4/10\n","157/157 [==============================] - 11s 71ms/step - loss: 0.3964 - accuracy: 0.8828 - val_loss: 0.3505 - val_accuracy: 0.8933\n","Epoch 5/10\n","157/157 [==============================] - 11s 71ms/step - loss: 0.3456 - accuracy: 0.8998 - val_loss: 0.3351 - val_accuracy: 0.8975\n","Epoch 6/10\n","157/157 [==============================] - 11s 73ms/step - loss: 0.3215 - accuracy: 0.9044 - val_loss: 0.3304 - val_accuracy: 0.8895\n","Epoch 7/10\n","157/157 [==============================] - 11s 71ms/step - loss: 0.3026 - accuracy: 0.9088 - val_loss: 0.3186 - val_accuracy: 0.8950\n","Epoch 8/10\n","157/157 [==============================] - 11s 71ms/step - loss: 0.2842 - accuracy: 0.9136 - val_loss: 0.3209 - val_accuracy: 0.8950\n","Epoch 9/10\n","157/157 [==============================] - 11s 72ms/step - loss: 0.2613 - accuracy: 0.9230 - val_loss: 0.3153 - val_accuracy: 0.8937\n","Epoch 10/10\n","157/157 [==============================] - 11s 72ms/step - loss: 0.2543 - accuracy: 0.9240 - val_loss: 0.3162 - val_accuracy: 0.8975\n"],"name":"stdout"},{"output_type":"execute_result","data":{"text/plain":["<tensorflow.python.keras.callbacks.History at 0x7f5f15bfdb38>"]},"metadata":{"tags":[]},"execution_count":30}]},{"cell_type":"code","metadata":{"id":"XiNerjxXuw42","colab":{"base_uri":"https://localhost:8080/","height":709},"executionInfo":{"status":"ok","timestamp":1603377624740,"user_tz":240,"elapsed":235427,"user":{"displayName":"Yassine Yousfi","photoUrl":"","userId":"06317191099946336363"}},"outputId":"bb0aed46-07d5-4c16-9d65-3793dc973ccd"},"source":["base_model.trainable = True \n","# BN/Dropout is still in inference mode\n","# Compile here | IMPORTANT\n","model.compile(optimizer=tf.keras.optimizers.Adam(1e-5), # Low LR, but maybe can do better?\n","              loss=tf.keras.losses.CategoricalCrossentropy(from_logits=True),\n","              metrics=['accuracy'])\n","\n","model.summary()\n","model.fit(x=data_train, validation_data=data_val, epochs=10)"],"execution_count":null,"outputs":[{"output_type":"stream","text":["Model: \"functional_9\"\n","_________________________________________________________________\n","Layer (type)                 Output Shape              Param #   \n","=================================================================\n","input_10 (InputLayer)        [(None, 160, 160, 3)]     0         \n","_________________________________________________________________\n","efficientnet-b0 (Functional) (None, 5, 5, 1280)        4049564   \n","_________________________________________________________________\n","global_average_pooling2d_4 ( (None, 1280)              0         \n","_________________________________________________________________\n","dropout_4 (Dropout)          (None, 1280)              0         \n","_________________________________________________________________\n","dense_4 (Dense)              (None, 37)                47397     \n","=================================================================\n","Total params: 4,096,961\n","Trainable params: 4,054,945\n","Non-trainable params: 42,016\n","_________________________________________________________________\n","Epoch 1/10\n","  2/157 [..............................] - ETA: 16s - loss: 0.1637 - accuracy: 0.9688WARNING:tensorflow:Callbacks method `on_train_batch_end` is slow compared to the batch time (batch time: 0.0503s vs `on_train_batch_end` time: 0.0797s). Check your callbacks.\n","157/157 [==============================] - 24s 151ms/step - loss: 0.2065 - accuracy: 0.9400 - val_loss: 0.3014 - val_accuracy: 0.8987\n","Epoch 2/10\n","157/157 [==============================] - 23s 146ms/step - loss: 0.1791 - accuracy: 0.9434 - val_loss: 0.2986 - val_accuracy: 0.9038\n","Epoch 3/10\n","157/157 [==============================] - 22s 143ms/step - loss: 0.1758 - accuracy: 0.9376 - val_loss: 0.3022 - val_accuracy: 0.9017\n","Epoch 4/10\n","157/157 [==============================] - 22s 142ms/step - loss: 0.1534 - accuracy: 0.9498 - val_loss: 0.2984 - val_accuracy: 0.9025\n","Epoch 5/10\n","157/157 [==============================] - 22s 141ms/step - loss: 0.1468 - accuracy: 0.9498 - val_loss: 0.3015 - val_accuracy: 0.9013\n","Epoch 6/10\n","157/157 [==============================] - 23s 145ms/step - loss: 0.1250 - accuracy: 0.9598 - val_loss: 0.3042 - val_accuracy: 0.9008\n","Epoch 7/10\n","157/157 [==============================] - 23s 145ms/step - loss: 0.1437 - accuracy: 0.9538 - val_loss: 0.3019 - val_accuracy: 0.9046\n","Epoch 8/10\n","157/157 [==============================] - 22s 142ms/step - loss: 0.1348 - accuracy: 0.9562 - val_loss: 0.3027 - val_accuracy: 0.9033\n","Epoch 9/10\n","157/157 [==============================] - 23s 145ms/step - loss: 0.1306 - accuracy: 0.9570 - val_loss: 0.3047 - val_accuracy: 0.9079\n","Epoch 10/10\n","157/157 [==============================] - 23s 145ms/step - loss: 0.1155 - accuracy: 0.9646 - val_loss: 0.3054 - val_accuracy: 0.9054\n"],"name":"stdout"},{"output_type":"execute_result","data":{"text/plain":["<tensorflow.python.keras.callbacks.History at 0x7f5f15bfd240>"]},"metadata":{"tags":[]},"execution_count":31}]},{"cell_type":"markdown","metadata":{"id":"J57vubpnj8O4"},"source":["We start to overfit when we unfreeze the entire model, but the validation accuracy is slightly better. We will see later how this can be solved using learning rate schedules for example."]},{"cell_type":"markdown","metadata":{"id":"jP4dL0HY4wyZ"},"source":["## You want to unfreeze BN?"]},{"cell_type":"code","metadata":{"id":"yeGR8YnA4zYV","colab":{"base_uri":"https://localhost:8080/","height":692},"executionInfo":{"status":"ok","timestamp":1603381732179,"user_tz":240,"elapsed":123470,"user":{"displayName":"Yassine Yousfi","photoUrl":"","userId":"06317191099946336363"}},"outputId":"d0a166c6-c2d1-40c2-c163-955b24158274"},"source":["IMG_SHAPE = IMG_SIZE + (3,)\n","\n","base_model = efn.EfficientNetB0(input_shape=IMG_SHAPE, weights='imagenet', include_top=False)\n","\n","base_model.trainable = False\n","\n","inputs = tf.keras.Input(shape=IMG_SHAPE)\n","x = base_model(inputs) # No BN/Dropout inference forcing\n","x = tf.keras.layers.GlobalAveragePooling2D()(x)\n","x = tf.keras.layers.Dropout(0.2)(x)  \n","outputs = tf.keras.layers.Dense(37)(x)\n","model = tf.keras.Model(inputs, outputs)\n","\n","model.summary()\n","\n","model.compile(optimizer=tf.keras.optimizers.Adam(),\n","              loss=tf.keras.losses.CategoricalCrossentropy(from_logits=True),\n","              metrics=['accuracy'])\n","\n","model.fit(x=data_train, validation_data=data_val, epochs=10)"],"execution_count":null,"outputs":[{"output_type":"stream","text":["Model: \"functional_22\"\n","_________________________________________________________________\n","Layer (type)                 Output Shape              Param #   \n","=================================================================\n","input_28 (InputLayer)        [(None, 160, 160, 3)]     0         \n","_________________________________________________________________\n","efficientnet-b0 (Functional) (None, 5, 5, 1280)        4049564   \n","_________________________________________________________________\n","global_average_pooling2d_13  (None, 1280)              0         \n","_________________________________________________________________\n","dropout_14 (Dropout)         (None, 1280)              0         \n","_________________________________________________________________\n","dense_14 (Dense)             (None, 37)                47397     \n","=================================================================\n","Total params: 4,096,961\n","Trainable params: 47,397\n","Non-trainable params: 4,049,564\n","_________________________________________________________________\n","Epoch 1/10\n","157/157 [==============================] - 12s 78ms/step - loss: 1.5022 - accuracy: 0.6634 - val_loss: 0.5867 - val_accuracy: 0.8607\n","Epoch 2/10\n","157/157 [==============================] - 11s 72ms/step - loss: 0.6245 - accuracy: 0.8312 - val_loss: 0.4191 - val_accuracy: 0.8837\n","Epoch 3/10\n","157/157 [==============================] - 11s 73ms/step - loss: 0.4851 - accuracy: 0.8600 - val_loss: 0.3730 - val_accuracy: 0.8908\n","Epoch 4/10\n","157/157 [==============================] - 11s 73ms/step - loss: 0.4284 - accuracy: 0.8674 - val_loss: 0.3451 - val_accuracy: 0.8879\n","Epoch 5/10\n","157/157 [==============================] - 11s 71ms/step - loss: 0.3949 - accuracy: 0.8820 - val_loss: 0.3309 - val_accuracy: 0.8941\n","Epoch 6/10\n","157/157 [==============================] - 12s 76ms/step - loss: 0.3620 - accuracy: 0.8864 - val_loss: 0.3181 - val_accuracy: 0.8992\n","Epoch 7/10\n","157/157 [==============================] - 11s 73ms/step - loss: 0.3302 - accuracy: 0.9016 - val_loss: 0.3169 - val_accuracy: 0.9017\n","Epoch 8/10\n","157/157 [==============================] - 12s 73ms/step - loss: 0.3199 - accuracy: 0.9020 - val_loss: 0.3078 - val_accuracy: 0.9013\n","Epoch 9/10\n","157/157 [==============================] - 11s 73ms/step - loss: 0.2993 - accuracy: 0.9094 - val_loss: 0.3081 - val_accuracy: 0.9004\n","Epoch 10/10\n","157/157 [==============================] - 11s 73ms/step - loss: 0.2854 - accuracy: 0.9112 - val_loss: 0.3035 - val_accuracy: 0.9008\n"],"name":"stdout"},{"output_type":"execute_result","data":{"text/plain":["<tensorflow.python.keras.callbacks.History at 0x7f5ee925fb38>"]},"metadata":{"tags":[]},"execution_count":46}]},{"cell_type":"code","metadata":{"id":"bXomsYxs4324","colab":{"base_uri":"https://localhost:8080/","height":692},"executionInfo":{"status":"ok","timestamp":1603381953975,"user_tz":240,"elapsed":343214,"user":{"displayName":"Yassine Yousfi","photoUrl":"","userId":"06317191099946336363"}},"outputId":"33962eb8-0ba0-4b9c-b425-e16ee274a450"},"source":["base_model.trainable = True \n","# BN/Dropout is in training mode\n","# Compile here | IMPORTANT\n","model.compile(optimizer=tf.keras.optimizers.Adam(1e-4), # This is different! Tuned! Otherwise The perf goes down too much\n","              loss=tf.keras.losses.CategoricalCrossentropy(from_logits=True),\n","              metrics=['accuracy'])\n","\n","model.summary()\n","model.fit(x=data_train, validation_data=data_val, epochs=10)"],"execution_count":null,"outputs":[{"output_type":"stream","text":["Model: \"functional_22\"\n","_________________________________________________________________\n","Layer (type)                 Output Shape              Param #   \n","=================================================================\n","input_28 (InputLayer)        [(None, 160, 160, 3)]     0         \n","_________________________________________________________________\n","efficientnet-b0 (Functional) (None, 5, 5, 1280)        4049564   \n","_________________________________________________________________\n","global_average_pooling2d_13  (None, 1280)              0         \n","_________________________________________________________________\n","dropout_14 (Dropout)         (None, 1280)              0         \n","_________________________________________________________________\n","dense_14 (Dense)             (None, 37)                47397     \n","=================================================================\n","Total params: 4,096,961\n","Trainable params: 4,054,945\n","Non-trainable params: 42,016\n","_________________________________________________________________\n","Epoch 1/10\n","157/157 [==============================] - 22s 141ms/step - loss: 0.8125 - accuracy: 0.7550 - val_loss: 0.3409 - val_accuracy: 0.8858\n","Epoch 2/10\n","157/157 [==============================] - 21s 135ms/step - loss: 0.4468 - accuracy: 0.8576 - val_loss: 0.3452 - val_accuracy: 0.8891\n","Epoch 3/10\n","157/157 [==============================] - 21s 135ms/step - loss: 0.3155 - accuracy: 0.9004 - val_loss: 0.3450 - val_accuracy: 0.8849\n","Epoch 4/10\n","157/157 [==============================] - 21s 135ms/step - loss: 0.2410 - accuracy: 0.9290 - val_loss: 0.3323 - val_accuracy: 0.8929\n","Epoch 5/10\n","157/157 [==============================] - 21s 135ms/step - loss: 0.2119 - accuracy: 0.9366 - val_loss: 0.3272 - val_accuracy: 0.8946\n","Epoch 6/10\n","157/157 [==============================] - 21s 136ms/step - loss: 0.1714 - accuracy: 0.9484 - val_loss: 0.3083 - val_accuracy: 0.9017\n","Epoch 7/10\n","157/157 [==============================] - 21s 135ms/step - loss: 0.1476 - accuracy: 0.9586 - val_loss: 0.3211 - val_accuracy: 0.9004\n","Epoch 8/10\n","157/157 [==============================] - 21s 135ms/step - loss: 0.1287 - accuracy: 0.9628 - val_loss: 0.3139 - val_accuracy: 0.8979\n","Epoch 9/10\n","157/157 [==============================] - 21s 136ms/step - loss: 0.1042 - accuracy: 0.9700 - val_loss: 0.3298 - val_accuracy: 0.8950\n","Epoch 10/10\n","157/157 [==============================] - 21s 134ms/step - loss: 0.0939 - accuracy: 0.9742 - val_loss: 0.3250 - val_accuracy: 0.9017\n"],"name":"stdout"},{"output_type":"execute_result","data":{"text/plain":["<tensorflow.python.keras.callbacks.History at 0x7f5ef7533da0>"]},"metadata":{"tags":[]},"execution_count":47}]},{"cell_type":"markdown","metadata":{"id":"zzjQ1YP-5jKp"},"source":["This is not a good idea, freezing BN in the first stage and releasing it in the next stage will lead to abrupt changes.\n","\n","You need to be very careful with LR!\n","\n","To fix this, one solution is to unfreeze all BN layers in the first stage already.\n","\n","Another trick that also solves this is to use a warm-up phase (one-cycle) and figure out a good LR with LR finder. This usually gives good LR values and avoids those abrubt changes."]},{"cell_type":"markdown","metadata":{"id":"L8ACRgvQ32jh"},"source":["## Unfreeze all BNs from first stage"]},{"cell_type":"code","metadata":{"id":"aTQBQmr7NmyA","colab":{"base_uri":"https://localhost:8080/","height":692},"executionInfo":{"status":"ok","timestamp":1603385538967,"user_tz":240,"elapsed":169428,"user":{"displayName":"Yassine Yousfi","photoUrl":"","userId":"06317191099946336363"}},"outputId":"d47fcc9d-2467-4033-93c9-50fed4a34cc2"},"source":["IMG_SHAPE = IMG_SIZE + (3,)\n","\n","base_model = efn.EfficientNetB0(input_shape=IMG_SHAPE, weights='imagenet', include_top=False)\n","\n","base_model.trainable = False\n","\n","inputs = tf.keras.Input(shape=IMG_SHAPE)\n","x = base_model(inputs) # No BN inference forcing\n","x = tf.keras.layers.GlobalAveragePooling2D()(x)\n","x = tf.keras.layers.Dropout(0.2)(x)  \n","outputs = tf.keras.layers.Dense(37)(x)\n","model = tf.keras.Model(inputs, outputs)\n","\n","base_model.trainable = True\n","for layer in base_model.layers:\n","  if 'bn' in layer.name:\n","    layer.trainable = True\n","  else:\n","    layer.trainable = False\n","\n","model.summary()\n","\n","model.compile(optimizer=tf.keras.optimizers.Adam(),\n","              loss=tf.keras.losses.CategoricalCrossentropy(from_logits=True),\n","              metrics=['accuracy'])\n","\n","model.fit(x=data_train, validation_data=data_val, epochs=10)"],"execution_count":null,"outputs":[{"output_type":"stream","text":["Model: \"functional_24\"\n","_________________________________________________________________\n","Layer (type)                 Output Shape              Param #   \n","=================================================================\n","input_30 (InputLayer)        [(None, 160, 160, 3)]     0         \n","_________________________________________________________________\n","efficientnet-b0 (Functional) (None, 5, 5, 1280)        4049564   \n","_________________________________________________________________\n","global_average_pooling2d_14  (None, 1280)              0         \n","_________________________________________________________________\n","dropout_15 (Dropout)         (None, 1280)              0         \n","_________________________________________________________________\n","dense_15 (Dense)             (None, 37)                47397     \n","=================================================================\n","Total params: 4,096,961\n","Trainable params: 89,413\n","Non-trainable params: 4,007,548\n","_________________________________________________________________\n","Epoch 1/10\n","157/157 [==============================] - 29s 187ms/step - loss: 2.2568 - accuracy: 0.4338 - val_loss: 0.8142 - val_accuracy: 0.8075\n","Epoch 2/10\n","157/157 [==============================] - 15s 94ms/step - loss: 0.9674 - accuracy: 0.7234 - val_loss: 0.4878 - val_accuracy: 0.8657\n","Epoch 3/10\n","157/157 [==============================] - 15s 93ms/step - loss: 0.7000 - accuracy: 0.7916 - val_loss: 0.4099 - val_accuracy: 0.8757\n","Epoch 4/10\n","157/157 [==============================] - 15s 92ms/step - loss: 0.5644 - accuracy: 0.8264 - val_loss: 0.3794 - val_accuracy: 0.8837\n","Epoch 5/10\n","157/157 [==============================] - 14s 92ms/step - loss: 0.4869 - accuracy: 0.8496 - val_loss: 0.3447 - val_accuracy: 0.8874\n","Epoch 6/10\n","157/157 [==============================] - 14s 92ms/step - loss: 0.4550 - accuracy: 0.8582 - val_loss: 0.3323 - val_accuracy: 0.8962\n","Epoch 7/10\n","157/157 [==============================] - 15s 92ms/step - loss: 0.4019 - accuracy: 0.8778 - val_loss: 0.3321 - val_accuracy: 0.8908\n","Epoch 8/10\n","157/157 [==============================] - 14s 92ms/step - loss: 0.3789 - accuracy: 0.8782 - val_loss: 0.3331 - val_accuracy: 0.8879\n","Epoch 9/10\n","157/157 [==============================] - 14s 91ms/step - loss: 0.3334 - accuracy: 0.8932 - val_loss: 0.3225 - val_accuracy: 0.8854\n","Epoch 10/10\n","157/157 [==============================] - 15s 93ms/step - loss: 0.3323 - accuracy: 0.8952 - val_loss: 0.3108 - val_accuracy: 0.8937\n"],"name":"stdout"},{"output_type":"execute_result","data":{"text/plain":["<tensorflow.python.keras.callbacks.History at 0x7f5ee40bb860>"]},"metadata":{"tags":[]},"execution_count":61}]},{"cell_type":"code","metadata":{"id":"Nyq12GJZ3zAP","colab":{"base_uri":"https://localhost:8080/","height":692},"executionInfo":{"status":"ok","timestamp":1603379881475,"user_tz":240,"elapsed":221571,"user":{"displayName":"Yassine Yousfi","photoUrl":"","userId":"06317191099946336363"}},"outputId":"cc7acacd-7aed-4721-bc09-c802dc8a9034"},"source":["base_model.trainable = True \n","# Compile here | IMPORTANT\n","model.compile(optimizer=tf.keras.optimizers.Adam(1e-5), # Low LR, but maybe can do better?\n","              loss=tf.keras.losses.CategoricalCrossentropy(from_logits=True),\n","              metrics=['accuracy'])\n","\n","model.summary()\n","model.fit(x=data_train, validation_data=data_val, epochs=10)"],"execution_count":null,"outputs":[{"output_type":"stream","text":["Model: \"functional_16\"\n","_________________________________________________________________\n","Layer (type)                 Output Shape              Param #   \n","=================================================================\n","input_22 (InputLayer)        [(None, 160, 160, 3)]     0         \n","_________________________________________________________________\n","efficientnet-b0 (Functional) (None, 5, 5, 1280)        4049564   \n","_________________________________________________________________\n","global_average_pooling2d_10  (None, 1280)              0         \n","_________________________________________________________________\n","dropout_11 (Dropout)         (None, 1280)              0         \n","_________________________________________________________________\n","dense_11 (Dense)             (None, 37)                47397     \n","=================================================================\n","Total params: 4,096,961\n","Trainable params: 4,054,945\n","Non-trainable params: 42,016\n","_________________________________________________________________\n","Epoch 1/10\n","157/157 [==============================] - 22s 140ms/step - loss: 0.2679 - accuracy: 0.9180 - val_loss: 0.3122 - val_accuracy: 0.9008\n","Epoch 2/10\n","157/157 [==============================] - 21s 134ms/step - loss: 0.2588 - accuracy: 0.9138 - val_loss: 0.3090 - val_accuracy: 0.8996\n","Epoch 3/10\n","157/157 [==============================] - 21s 135ms/step - loss: 0.2556 - accuracy: 0.9226 - val_loss: 0.3066 - val_accuracy: 0.8996\n","Epoch 4/10\n","157/157 [==============================] - 21s 135ms/step - loss: 0.2441 - accuracy: 0.9246 - val_loss: 0.3052 - val_accuracy: 0.9033\n","Epoch 5/10\n","157/157 [==============================] - 21s 135ms/step - loss: 0.2331 - accuracy: 0.9286 - val_loss: 0.3060 - val_accuracy: 0.9038\n","Epoch 6/10\n","157/157 [==============================] - 21s 133ms/step - loss: 0.2261 - accuracy: 0.9358 - val_loss: 0.3040 - val_accuracy: 0.9059\n","Epoch 7/10\n","157/157 [==============================] - 21s 136ms/step - loss: 0.2234 - accuracy: 0.9328 - val_loss: 0.3040 - val_accuracy: 0.9033\n","Epoch 8/10\n","157/157 [==============================] - 21s 136ms/step - loss: 0.2054 - accuracy: 0.9372 - val_loss: 0.3039 - val_accuracy: 0.9059\n","Epoch 9/10\n","157/157 [==============================] - 21s 133ms/step - loss: 0.1996 - accuracy: 0.9394 - val_loss: 0.3035 - val_accuracy: 0.9067\n","Epoch 10/10\n","157/157 [==============================] - 21s 134ms/step - loss: 0.2084 - accuracy: 0.9392 - val_loss: 0.3033 - val_accuracy: 0.9038\n"],"name":"stdout"},{"output_type":"execute_result","data":{"text/plain":["<tensorflow.python.keras.callbacks.History at 0x7f5ef7533a58>"]},"metadata":{"tags":[]},"execution_count":41}]},{"cell_type":"markdown","metadata":{"id":"_fjAgBL9u6Sa"},"source":["## Helper snipets"]},{"cell_type":"code","metadata":{"id":"CvjM-qGGuxST"},"source":["# Unfreeze only Block7\n","# Because of recursive passing of the trainable attribute\n","# if a child layer is trainable but its parent layer is not trainable\n","# the child will not be trainable\n","base_model.trainable = True # First unfreeze the entire base model\n","for layer in base_model.layers:\n","  if 'block7' in layer.name:\n","    layer.trainable = True # Don't really need this line but makes the code readable\n","  else:\n","    layer.trainable = False # Freeze the other blocks"],"execution_count":null,"outputs":[]},{"cell_type":"code","metadata":{"id":"4yTwBx6wuxoE"},"source":["# Unfreeze Batch Norm Layers all together\n","base_model.trainable = True\n","for layer in base_model.layers:\n","  if 'bn' in layer.name:\n","    layer.trainable = True\n","  else:\n","    layer.trainable = False"],"execution_count":null,"outputs":[]}]}