arj-gan.ipynb

{
 "metadata": {
  "kernelspec": {
   "language": "python",
   "display_name": "Python 3",
   "name": "python3"
  },
  "language_info": {
   "name": "python",
   "version": "3.7.12",
   "mimetype": "text/x-python",
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "pygments_lexer": "ipython3",
   "nbconvert_exporter": "python",
   "file_extension": ".py"
  }
 },
 "nbformat_minor": 4,
 "nbformat": 4,
 "cells": [
  {
   "cell_type": "code",
   "source": [
    "# This Python 3 environment comes with many helpful analytics libraries installed\n",
    "# It is defined by the kaggle/python Docker image: https://github.com/kaggle/docker-python\n",
    "# For example, here's several helpful packages to load\n",
    "\n",
    "import numpy as np # linear algebra\n",
    "import pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\n",
    "\n",
    "# Input data files are available in the read-only \"../input/\" directory\n",
    "# For example, running this (by clicking run or pressing Shift+Enter) will list all files under the input directory\n",
    "\n",
    "import os\n",
    "for dirname, _, filenames in os.walk('/kaggle/input'):\n",
    "    for filename in filenames:\n",
    "        print(os.path.join(dirname, filename))\n",
    "\n",
    "# You can write up to 20GB to the current directory (/kaggle/working/) that gets preserved as output when you create a version using \"Save & Run All\" \n",
    "# You can also write temporary files to /kaggle/temp/, but they won't be saved outside of the current session"
   ],
   "metadata": {
    "_uuid": "8f2839f25d086af736a60e9eeb907d3b93b6e0e5",
    "_cell_guid": "b1076dfc-b9ad-4769-8c92-a6c4dae69d19",
    "execution": {
     "iopub.status.busy": "2022-08-08T14:08:07.162043Z",
     "iopub.execute_input": "2022-08-08T14:08:07.162711Z",
     "iopub.status.idle": "2022-08-08T14:08:07.170484Z",
     "shell.execute_reply.started": "2022-08-08T14:08:07.162676Z",
     "shell.execute_reply": "2022-08-08T14:08:07.169490Z"
    },
    "trusted": true
   },
   "execution_count": 2,
   "outputs": []
  },
  {
   "cell_type": "markdown",
   "source": [
    "## Dog Image Generator - Module 5\n",
    "\n",
    "For week five we have been asked to work through the Kaggle competition on generative Dog images. The competition will draw upon knowledge involving Generative networks, specifically DC adversarial networks.\n",
    "\n",
    "We will produce a unique network that generates pictures of dogs. A training library will be used to help classifiy initial images of dogs. Subsequently, a generator will be trained such that its output is fed to the classifier/discriminator and a feedback function is provided to the generator to create increasingly doglike images.\n",
    "\n",
    "IMPORTANT NOTE: When I download my notebook from Kaggle there cell outputs and impages do not follow the notebook. To view the complete notebook please use my Kaggle link below:\n",
    "\n",
    "https://www.kaggle.com/code/guymcguy/arj-gan/notebook\n",
    "\n",
    "#### Step 1 - Pull in data an perform cleaning/EDA\n",
    "We will import the input data (dog training images). Various pre-processing steps will be coverd (cropping, normalizing and creating random noise photos for the generator. \n",
    "\n",
    "I relied strongly on the following notebook for image pre-processing:\n",
    "\n",
    "https://www.kaggle.com/code/cmalla94/dcgan-generating-dog-images-with-tensorflow"
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "source": "import numpy as np, pandas as pd, os\nimport xml.etree.ElementTree as ET\nimport matplotlib.pyplot as plt, zipfile \nfrom PIL import Image\n\nimport tensorflow as tf\nfrom keras.models import Model\nfrom keras.layers import Input, Dense, Conv2D, MaxPooling2D, UpSampling2D, Reshape, BatchNormalization\nfrom keras.preprocessing.image import ImageDataGenerator\nfrom keras.callbacks import LearningRateScheduler\nfrom keras import optimizers\n\nfrom IPython.display import FileLink, FileLinks\nfrom PIL import Image # image processing library\nimport IPython.display as display # for inspecting the images by displaying them in the kernel\n\n# Input data files are available in the read-only \"../input/\" directory\n# For example, running this (by clicking run or pressing Shift+Enter) will list all files under the input directory",
   "metadata": {
    "execution": {
     "iopub.status.busy": "2022-08-08T14:12:42.986039Z",
     "iopub.execute_input": "2022-08-08T14:12:42.986939Z",
     "iopub.status.idle": "2022-08-08T14:12:42.994934Z",
     "shell.execute_reply.started": "2022-08-08T14:12:42.986890Z",
     "shell.execute_reply": "2022-08-08T14:12:42.993830Z"
    },
    "trusted": true
   },
   "execution_count": 4,
   "outputs": []
  },
  {
   "cell_type": "code",
   "source": "import zipfile\n\nwith zipfile.ZipFile(\"../input/generative-dog-images/all-dogs.zip\",\"r\") as zip_ref:\n    zip_ref.extractall(\"/kaggle/temp/\")\n    \nwith zipfile.ZipFile(\"../input/generative-dog-images/Annotation.zip\",\"r\") as zip_ref:\n    zip_ref.extractall(\"/kaggle/temp/\")",
   "metadata": {
    "execution": {
     "iopub.status.busy": "2022-08-08T14:12:44.447684Z",
     "iopub.execute_input": "2022-08-08T14:12:44.448395Z",
     "iopub.status.idle": "2022-08-08T14:13:11.789919Z",
     "shell.execute_reply.started": "2022-08-08T14:12:44.448355Z",
     "shell.execute_reply": "2022-08-08T14:13:11.788947Z"
    },
    "trusted": true
   },
   "execution_count": 5,
   "outputs": []
  },
  {
   "cell_type": "code",
   "source": "# Variables to be used for network attributes.\n\nSEED = 42\nnp.random.seed(SEED)\nrandom_dim = 128\n\n# Variables needed for the training part\nEPOCHS = 280\nnoise_dim = 100\nnum_examples_to_generate = 16\n\n# This seed will be used later on to produce samples when training\nseed = tf.random.normal([num_examples_to_generate, noise_dim])\n\nROOT = '/kaggle/temp/'\n#if not ComputeLB: ROOT = '/kaggle/working/'\nbreeds = os.listdir(ROOT + 'Annotation') \nIMAGES = os.listdir(ROOT + 'all-dogs')",
   "metadata": {
    "execution": {
     "iopub.status.busy": "2022-08-08T14:13:11.792758Z",
     "iopub.execute_input": "2022-08-08T14:13:11.793473Z",
     "iopub.status.idle": "2022-08-08T14:13:14.705989Z",
     "shell.execute_reply.started": "2022-08-08T14:13:11.793429Z",
     "shell.execute_reply": "2022-08-08T14:13:14.704935Z"
    },
    "trusted": true
   },
   "execution_count": 6,
   "outputs": []
  },
  {
   "cell_type": "markdown",
   "source": "We will need to crop images to effectively use within the model. I have utilized an existing notebook found in the competition directory located here:\nhttps://www.kaggle.com/code/aligsaoud/dog-image-generator",
   "metadata": {}
  },
  {
   "cell_type": "code",
   "source": "idxIn = 0; namesIn = []\nimagesIn = np.zeros((25000,64,64,3))\nDogsOnly = False\n\n# CROP WITH BOUNDING BOXES TO GET DOGS ONLY\nif DogsOnly:\n    for breed in breeds:\n        for dog in os.listdir(ROOT+'annotation/Annotation/'+breed):\n            try: img = Image.open(ROOT+'all-dogs/'+dog+'.jpg') \n            except: continue           \n            tree = ET.parse(ROOT+'annotation/Annotation/'+breed+'/'+dog)\n            root = tree.getroot()\n            objects = root.findall('object')\n            for o in objects:\n                bndbox = o.find('bndbox') \n                xmin = int(bndbox.find('xmin').text)\n                ymin = int(bndbox.find('ymin').text)\n                xmax = int(bndbox.find('xmax').text)\n                ymax = int(bndbox.find('ymax').text)\n                w = np.min((xmax - xmin, ymax - ymin))\n                img2 = img.crop((xmin, ymin, xmin+w, ymin+w))\n                img2 = img2.resize((64,64), Image.ANTIALIAS)\n                imagesIn[idxIn,:,:,:] = np.asarray(img2)\n                #if idxIn%1000==0: print(idxIn)\n                namesIn.append(breed)\n                idxIn += 1\n                \n# RANDOMLY CROP FULL IMAGES\nelse:\n    x = np.random.choice(np.arange(20000),10000)\n    for k in range(len(x)):\n        img = Image.open(ROOT + 'all-dogs/' + IMAGES[x[k]])\n        w = img.size[0]; h = img.size[1];\n        if (k%2==0)|(k%3==0):\n            w2 = 100; h2 = int(h/(w/100))\n            a = 18; b = 0          \n        else:\n            a=0; b=0\n            if w<h:\n                w2 = 64; h2 = int((64/w)*h)\n                b = (h2-64)//2\n            else:\n                h2 = 64; w2 = int((64/h)*w)\n                a = (w2-64)//2\n        img = img.resize((w2,h2), Image.ANTIALIAS)\n        img = img.crop((0+a, 0+b, 64+a, 64+b))  \n        imagesIn[idxIn,:,:,:] = np.asarray(img)\n        namesIn.append(IMAGES[x[k]])\n        #if idxIn%1000==0: print(idxIn)\n        idxIn += 1\n    \n# DISPLAY CROPPED IMAGES\nx = np.random.randint(0,idxIn,25)\nfor k in range(5):\n    plt.figure(figsize=(15,3))\n    for j in range(5):\n        plt.subplot(1,5,j+1)\n        img = Image.fromarray( imagesIn[x[k*5+j],:,:,:].astype('uint8') )\n        plt.axis('off')\n        if not DogsOnly: \n            plt.title(namesIn[x[k*5+j]],fontsize=11)\n        else: \n            plt.title(namesIn[x[k*5+j]].split('-')[1],fontsize=11)\n        plt.imshow(img)\n    plt.show()",
   "metadata": {
    "execution": {
     "iopub.status.busy": "2022-08-08T14:14:42.438200Z",
     "iopub.execute_input": "2022-08-08T14:14:42.438582Z",
     "iopub.status.idle": "2022-08-08T14:15:56.250353Z",
     "shell.execute_reply.started": "2022-08-08T14:14:42.438546Z",
     "shell.execute_reply": "2022-08-08T14:15:56.249437Z"
    },
    "trusted": true
   },
   "execution_count": 7,
   "outputs": []
  },
  {
   "cell_type": "code",
   "source": "imagesIn.shape",
   "metadata": {
    "execution": {
     "iopub.status.busy": "2022-08-08T14:15:56.252536Z",
     "iopub.execute_input": "2022-08-08T14:15:56.252905Z",
     "iopub.status.idle": "2022-08-08T14:15:56.263056Z",
     "shell.execute_reply.started": "2022-08-08T14:15:56.252868Z",
     "shell.execute_reply": "2022-08-08T14:15:56.261871Z"
    },
    "trusted": true
   },
   "execution_count": 8,
   "outputs": []
  },
  {
   "cell_type": "markdown",
   "source": "So we can see that there are 250K sample images with three channels and 64x64 in dimension.\n\nWe can normalize the images around 0 mean",
   "metadata": {}
  },
  {
   "cell_type": "code",
   "source": "# Normalize Images in \n\n# normalize the pixel values\nimagesIn = (imagesIn[:idxIn,:,:,:]-127.5)/127.5",
   "metadata": {
    "execution": {
     "iopub.status.busy": "2022-08-08T14:15:56.264769Z",
     "iopub.execute_input": "2022-08-08T14:15:56.265186Z",
     "iopub.status.idle": "2022-08-08T14:15:56.723927Z",
     "shell.execute_reply.started": "2022-08-08T14:15:56.265151Z",
     "shell.execute_reply": "2022-08-08T14:15:56.722971Z"
    },
    "trusted": true
   },
   "execution_count": 9,
   "outputs": []
  },
  {
   "cell_type": "code",
   "source": "# Plot normalized sample images\n\nplt.figure(figsize=(8,8))\nfor image in range(4):\n    plt.subplot(2,2, image+1)\n    plt.imshow((imagesIn[image]))\n    plt.xlabel('shape: {}'.format(imagesIn[image].shape))",
   "metadata": {
    "execution": {
     "iopub.status.busy": "2022-08-08T14:15:56.726264Z",
     "iopub.execute_input": "2022-08-08T14:15:56.726660Z",
     "iopub.status.idle": "2022-08-08T14:15:57.164305Z",
     "shell.execute_reply.started": "2022-08-08T14:15:56.726620Z",
     "shell.execute_reply": "2022-08-08T14:15:57.163416Z"
    },
    "trusted": true
   },
   "execution_count": 10,
   "outputs": []
  },
  {
   "cell_type": "markdown",
   "source": "We will need a generator to pass in the images to the network. To make this work we will need to cast the images as float32.",
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": "",
   "metadata": {}
  },
  {
   "cell_type": "code",
   "source": "# cast to float 32\nimagesIn = tf.cast(imagesIn, 'float32')\n\nbuffer = 10000\nbatch_size=64\nds = tf.data.Dataset.from_tensor_slices(imagesIn).shuffle(buffer).batch(batch_size)\n",
   "metadata": {
    "execution": {
     "iopub.status.busy": "2022-08-08T14:15:57.165772Z",
     "iopub.execute_input": "2022-08-08T14:15:57.166563Z",
     "iopub.status.idle": "2022-08-08T14:15:58.631370Z",
     "shell.execute_reply.started": "2022-08-08T14:15:57.166508Z",
     "shell.execute_reply": "2022-08-08T14:15:58.630410Z"
    },
    "trusted": true
   },
   "execution_count": 11,
   "outputs": []
  },
  {
   "cell_type": "markdown",
   "source": "### Step 2 - Build the various models (Generator and discriminator/classifier)\n\nI will create two separate networks:\n\n    A. Generator - takes random input noise and produces a model with network weights that produce images that look like dogs for:\n    B: Discriminator - Takes the output from model A and classifies whether or not it is a dog image.\n\nWe will have two separate loss functions, one for each model.",
   "metadata": {}
  },
  {
   "cell_type": "code",
   "source": "# function will return a generator model\n\n# Weight initializers for the Generator network\nWEIGHT_INIT = tf.keras.initializers.RandomNormal(mean=0.0, stddev=0.2)\n\n\ndef make_generator():\n    model = tf.keras.Sequential(\n        [\n            # first layer with 32,768 nodes expecting an input of vector size 100 (random noise)\n            tf.keras.layers.Dense(8*8*512, use_bias=False, input_shape=(100,)),\n            # Normalize the activations of the previous layer at each batch\n            tf.keras.layers.BatchNormalization(),\n            # apply leaky relu activation: f(x) = {x if x > 0 : 0.01*x}\n            tf.keras.layers.LeakyReLU(),\n            # reshape input to (8,8,512)\n            tf.keras.layers.Reshape((8, 8, 512)),\n            \n            # second layer Conv2DTranspose so it is doing the opposite of a convolutional layer\n            tf.keras.layers.Conv2DTranspose(256, (5,5), strides=(2,2), padding='same', use_bias=False,\\\n                kernel_initializer=WEIGHT_INIT),\n            tf.keras.layers.BatchNormalization(),\n            tf.keras.layers.LeakyReLU(),\n            tf.keras.layers.Dropout(0.3),\n            \n            tf.keras.layers.Conv2DTranspose(128, (5,5), strides=(2,2), padding='same', use_bias=False,\\\n                kernel_initializer=WEIGHT_INIT),\n            tf.keras.layers.BatchNormalization(),\n            tf.keras.layers.LeakyReLU(),\n            tf.keras.layers.Dropout(0.3),\n            \n            tf.keras.layers.Conv2DTranspose(64, (5,5), strides=(2,2), padding='same', use_bias=False,\\\n                kernel_initializer=WEIGHT_INIT),\n            tf.keras.layers.BatchNormalization(),\n            tf.keras.layers.LeakyReLU(),\n            \n            tf.keras.layers.Dense(3,activation='tanh', use_bias=False,\\\n                kernel_initializer=WEIGHT_INIT)\n        ]\n    )\n    return model\n# create an instance of the generator model defined\ngenerator = make_generator()\nprint(generator.summary())\n\n# tf.keras.utils.plot_model(\n#     generator,\n#     to_file='/tmp/gen_mdl.png',\n#     show_shapes=True,\n#     show_layer_names=True,\n#     rankdir='TB',\n# )\n\n# random noise vector\nnoise = tf.random.normal([1,100])\n# run the generator model with the noise vector as input\ngenerated_image = generator(noise, training=False)\n# display output\nplt.imshow(generated_image[0, :, :, :])\nprint(generated_image.shape)",
   "metadata": {
    "execution": {
     "iopub.status.busy": "2022-08-08T14:17:52.797744Z",
     "iopub.execute_input": "2022-08-08T14:17:52.798132Z",
     "iopub.status.idle": "2022-08-08T14:18:00.073154Z",
     "shell.execute_reply.started": "2022-08-08T14:17:52.798100Z",
     "shell.execute_reply": "2022-08-08T14:18:00.072199Z"
    },
    "trusted": true
   },
   "execution_count": 12,
   "outputs": []
  },
  {
   "cell_type": "code",
   "source": "# This is helpful for quickly downloading output data and samples\n\nFileLinks('.')",
   "metadata": {
    "execution": {
     "iopub.status.busy": "2022-08-08T14:18:00.075146Z",
     "iopub.execute_input": "2022-08-08T14:18:00.075536Z",
     "iopub.status.idle": "2022-08-08T14:18:00.082677Z",
     "shell.execute_reply.started": "2022-08-08T14:18:00.075486Z",
     "shell.execute_reply": "2022-08-08T14:18:00.081412Z"
    },
    "trusted": true
   },
   "execution_count": 13,
   "outputs": []
  },
  {
   "cell_type": "code",
   "source": "# Create the Discriminator\n\ndef make_discriminator():\n    model = tf.keras.Sequential(\n        [\n            tf.keras.layers.Conv2D(64, (4,4), strides=(2,2), padding='same', input_shape=[64,64,3],\\\n                kernel_initializer=WEIGHT_INIT),\n            tf.keras.layers.BatchNormalization(),\n            tf.keras.layers.LeakyReLU(),\n            tf.keras.layers.Dropout(0.25),\n            \n            tf.keras.layers.Conv2D(128, (4,4), strides=(2,2), padding='same',\\\n                kernel_initializer=WEIGHT_INIT),\n            tf.keras.layers.BatchNormalization(),\n            tf.keras.layers.LeakyReLU(),\n            tf.keras.layers.Dropout(0.25),\n            \n            tf.keras.layers.Conv2D(256, (4,4), strides=(2,2), padding='same',\\\n                kernel_initializer=WEIGHT_INIT),\n            tf.keras.layers.BatchNormalization(),\n            tf.keras.layers.LeakyReLU(),\n            tf.keras.layers.Dropout(0.25),\n            \n            # flatten input into 1-D and output a single a number from the last layer using sigmoid activation\n            tf.keras.layers.Flatten(),\n            tf.keras.layers.Dense(1, activation='sigmoid')\n        ]\n    )\n    return model\n\ndiscriminator = make_discriminator()\n# tf.keras.utils.plot_model(\n#     generator,\n#     to_file='/tmp/dis_mdl.png',\n#     show_shapes=True,\n#     show_layer_names=True,\n#     rankdir='TB',\n# )\nprint(discriminator.summary())\ndecision = discriminator(generated_image)\nprint (decision)",
   "metadata": {
    "execution": {
     "iopub.status.busy": "2022-08-08T14:18:42.384564Z",
     "iopub.execute_input": "2022-08-08T14:18:42.385591Z",
     "iopub.status.idle": "2022-08-08T14:18:42.624094Z",
     "shell.execute_reply.started": "2022-08-08T14:18:42.385542Z",
     "shell.execute_reply": "2022-08-08T14:18:42.623072Z"
    },
    "trusted": true
   },
   "execution_count": 14,
   "outputs": []
  },
  {
   "cell_type": "markdown",
   "source": "\nLoss and optimizer:\n\nTo train the network and individual models of the discriminator and generator we will leverage cross entropy as a starting point. Please see notes below for declaration of references for source material on creating noisy labels (From GAN hacks) and general GAN network architecture (TF tutorial).",
   "metadata": {}
  },
  {
   "cell_type": "code",
   "source": "# Please see cell 17 from: https://www.kaggle.com/code/cmalla94/dcgan-generating-dog-images-with-tensorflow\n# to get a better understanding of whats going on\n\n# The source also refers to GAN hacks: \n# https://github.com/soumith/ganhacks\n\n# Label smoothing -- technique from GAN hacks, instead of assigning 1/0 as class labels, we assign a random integer in range [0.7, 1.0] for positive class\n# and [0.0, 0.3] for negative class\n\ndef smooth_positive_labels(y):\n    return y - 0.3 + (np.random.random(y.shape) * 0.3)\n\ndef smooth_negative_labels(y):\n    return y + np.random.random(y.shape) * 0.5\n\n# Recomended to introduce some noise to the labels, so out of 1000 real labels, approximately 50 should be flipped to 0 (5%)\n# randomly flip some labels\ndef noisy_labels(y, p_flip):\n    # determine the number of labels to flip\n    n_select = int(p_flip * y.shape[0].value)\n    # choose labels to flip\n    flip_ix = choice([i for i in range(y.shape[0].value)], size=n_select)\n    # invert the labels in place\n    y[flip_ix] = 1 - y[flip_ix]\n    return y",
   "metadata": {
    "execution": {
     "iopub.status.busy": "2022-08-08T14:24:10.163997Z",
     "iopub.execute_input": "2022-08-08T14:24:10.164369Z",
     "iopub.status.idle": "2022-08-08T14:24:10.171811Z",
     "shell.execute_reply.started": "2022-08-08T14:24:10.164339Z",
     "shell.execute_reply": "2022-08-08T14:24:10.170742Z"
    },
    "trusted": true
   },
   "execution_count": 15,
   "outputs": []
  },
  {
   "cell_type": "markdown",
   "source": "Training loop",
   "metadata": {}
  },
  {
   "cell_type": "code",
   "source": "# This method returns a helper function to compute cross entropy loss\n# code from tf dcgan tutorial\n# https://www.tensorflow.org/tutorials/generative/dcgan\ncross_entropy = tf.keras.losses.BinaryCrossentropy(from_logits=False)\n\n# The Discriminator loss function\n\ndef discriminator_loss(real_output, fake_output):\n    # tf.ones_like changes all values in the tensor to 1s\n    # similarly tf.zeros_like changes all values in the tensor to 0\n    # then apply label smoothing\n    real_output_smooth = smooth_positive_labels(tf.ones_like(real_output))\n    fake_output_smooth = smooth_negative_labels(tf.zeros_like(fake_output))\n    real_loss = cross_entropy(real_output_smooth, real_output)\n    fake_loss = cross_entropy(fake_output_smooth, fake_output)\n    total_loss = real_loss + fake_loss\n    return total_loss\n\n# The Generator loss function\n\ndef generator_loss(fake_output):\n    fake_output_smooth = smooth_negative_labels(tf.ones_like(fake_output))\n    return cross_entropy(fake_output_smooth, fake_output)\n\n# # optimizers -- Adam\ngenerator_optimizer = tf.keras.optimizers.Adam(learning_rate=.0002,beta_1=.5)\ndiscriminator_optimizer = tf.keras.optimizers.Adam(learning_rate=.0002,beta_1=.5)",
   "metadata": {
    "execution": {
     "iopub.status.busy": "2022-08-08T14:24:14.642155Z",
     "iopub.execute_input": "2022-08-08T14:24:14.643227Z",
     "iopub.status.idle": "2022-08-08T14:24:14.651642Z",
     "shell.execute_reply.started": "2022-08-08T14:24:14.643163Z",
     "shell.execute_reply": "2022-08-08T14:24:14.650565Z"
    },
    "trusted": true
   },
   "execution_count": 16,
   "outputs": []
  },
  {
   "cell_type": "code",
   "source": "# code from tf dcgan tutorial\n# Variables needed for the training part\nEPOCHS = 528\nnoise_dim = 100\nnum_examples_to_generate = 16\n\ndef train_step(images, G_loss_list, D_loss_list):\n    noise = tf.random.normal([64, noise_dim])\n    \n    with tf.GradientTape() as gen_tape, tf.GradientTape() as disc_tape:\n        # the following are the operations recorded onto the \"tape\"\n        generated_images = generator(noise, training=True)\n        \n        real_output = discriminator(images, training=True)\n        fake_output = discriminator(generated_images, training=True)\n        gen_loss = generator_loss(fake_output)\n        disc_loss = discriminator_loss(real_output, fake_output)\n        \n    G_loss_list.append(gen_loss.numpy())\n    D_loss_list.append(disc_loss.numpy())\n    # the following lines are taking the derivatives and applying gradients using Adam optimizer\n    gradients_of_generator = gen_tape.gradient(gen_loss, generator.trainable_variables)\n    gradients_of_discriminator = disc_tape.gradient(disc_loss, discriminator.trainable_variables)\n    \n    generator_optimizer.apply_gradients(zip(gradients_of_generator, generator.trainable_variables))\n    discriminator_optimizer.apply_gradients(zip(gradients_of_discriminator, discriminator.trainable_variables))\n    \n",
   "metadata": {
    "execution": {
     "iopub.status.busy": "2022-08-08T14:24:36.156195Z",
     "iopub.execute_input": "2022-08-08T14:24:36.156611Z",
     "iopub.status.idle": "2022-08-08T14:24:36.165183Z",
     "shell.execute_reply.started": "2022-08-08T14:24:36.156576Z",
     "shell.execute_reply": "2022-08-08T14:24:36.163844Z"
    },
    "trusted": true
   },
   "execution_count": 17,
   "outputs": []
  },
  {
   "cell_type": "code",
   "source": "def train(dataset, epochs):\n    G_loss = []\n    D_loss = []\n    for epoch in range(epochs):\n        start = time.time()\n        for image_batch in dataset:\n            train_step(image_batch, G_loss, D_loss)\n            \n        plot_loss(G_loss, D_loss, epoch)\n        G_loss = []\n        D_loss = []\n        if (epoch % 10 == 0):\n            display.clear_output(wait=True)\n            generate_and_save_images(generator,\n                                 epoch + 1,\n                                 seed)\n        print ('Time for epoch {} is {} sec'.format(epoch + 1, time.time()-start))\n        \n            \n    # Generate after the final epoch\n    print(\"Final Epoch\")\n    generate_and_save_images(generator,\n                           epochs,\n                           seed)",
   "metadata": {
    "execution": {
     "iopub.status.busy": "2022-08-08T14:26:49.028382Z",
     "iopub.execute_input": "2022-08-08T14:26:49.029078Z",
     "iopub.status.idle": "2022-08-08T14:26:49.035802Z",
     "shell.execute_reply.started": "2022-08-08T14:26:49.029034Z",
     "shell.execute_reply": "2022-08-08T14:26:49.034848Z"
    },
    "trusted": true
   },
   "execution_count": 21,
   "outputs": []
  },
  {
   "cell_type": "code",
   "source": "import time\ndef generate_and_save_images(model, epoch, test_input):\n    # Notice `training` is set to False.\n    # This is so all layers run in inference mode (batchnorm).\n    predictions = model(test_input, training=False)    \n    fig = plt.figure(figsize=(8,8))\n    for i in range(predictions.shape[0]):\n        plt.subplot(4, 4, i+1)\n        plt.imshow((predictions[i, :, :, :]+1.)/2.)\n        plt.axis('off')\n    plt.savefig('image_at_epoch_{}.png'.format(epoch+1))\n    plt.show()",
   "metadata": {
    "execution": {
     "iopub.status.busy": "2022-08-08T14:27:00.436393Z",
     "iopub.execute_input": "2022-08-08T14:27:00.437007Z",
     "iopub.status.idle": "2022-08-08T14:27:00.444339Z",
     "shell.execute_reply.started": "2022-08-08T14:27:00.436971Z",
     "shell.execute_reply": "2022-08-08T14:27:00.443195Z"
    },
    "trusted": true
   },
   "execution_count": 22,
   "outputs": []
  },
  {
   "cell_type": "code",
   "source": "# function by Nanashi\n# https://www.kaggle.com/code/jesucristo/introducing-dcgan-dogs-images\n\n\ndef plot_loss (G_losses, D_losses, epoch):\n    plt.figure(figsize=(10,5))\n    plt.title(\"Generator and Discriminator Loss - EPOCH {}\".format(epoch+1))\n    plt.plot(G_losses,label=\"G\")\n    plt.plot(D_losses,label=\"D\")\n    plt.xlabel(\"iterations\")\n    plt.ylabel(\"Loss\")\n    plt.legend()\n    plt.show()\n    ",
   "metadata": {
    "execution": {
     "iopub.status.busy": "2022-08-08T14:27:01.738220Z",
     "iopub.execute_input": "2022-08-08T14:27:01.738636Z",
     "iopub.status.idle": "2022-08-08T14:27:01.744828Z",
     "shell.execute_reply.started": "2022-08-08T14:27:01.738601Z",
     "shell.execute_reply": "2022-08-08T14:27:01.743763Z"
    },
    "trusted": true
   },
   "execution_count": 23,
   "outputs": []
  },
  {
   "cell_type": "code",
   "source": "%%time\nprint('Starting training')\ntrain(ds, EPOCHS)\n",
   "metadata": {
    "execution": {
     "iopub.status.busy": "2022-08-08T14:27:06.790427Z",
     "iopub.execute_input": "2022-08-08T14:27:06.790788Z",
     "iopub.status.idle": "2022-08-08T18:04:54.403384Z",
     "shell.execute_reply.started": "2022-08-08T14:27:06.790758Z",
     "shell.execute_reply": "2022-08-08T18:04:54.402507Z"
    },
    "trusted": true
   },
   "execution_count": 24,
   "outputs": []
  },
  {
   "cell_type": "markdown",
   "source": "Save 10K photos to zip for submission",
   "metadata": {}
  },
  {
   "cell_type": "code",
   "source": "%%time\n# SAVE TO ZIP FILE NAMED IMAGES.ZIP\n\nz = zipfile.PyZipFile('images.zip', mode='w')\n\nfilename = 'gen_model.h5'\ntf.keras.models.save_model(\n    generator,\n    filename,\n    overwrite=True,\n    include_optimizer=True,\n    save_format=None\n)\n\nfor k in range(10000):\n    generated_image = generator(tf.random.normal([1, noise_dim]), training=False)\n    f = str(k)+'.png'\n    img = ((generated_image[0,:,:,:]+1.)/2.).numpy()\n    tf.keras.preprocessing.image.save_img(\n        f,\n        img,\n        scale=True\n    )\n    z.write(f); os.remove(f)\nz.close()\n\n",
   "metadata": {
    "execution": {
     "iopub.status.busy": "2022-08-08T18:04:54.405166Z",
     "iopub.execute_input": "2022-08-08T18:04:54.406168Z",
     "iopub.status.idle": "2022-08-08T18:06:18.506172Z",
     "shell.execute_reply.started": "2022-08-08T18:04:54.406131Z",
     "shell.execute_reply": "2022-08-08T18:06:18.505188Z"
    },
    "trusted": true
   },
   "execution_count": 25,
   "outputs": []
  },
  {
   "cell_type": "code",
   "source": "FileLinks('.')\n",
   "metadata": {
    "execution": {
     "iopub.status.busy": "2022-08-08T18:06:18.521610Z",
     "iopub.execute_input": "2022-08-08T18:06:18.521953Z",
     "iopub.status.idle": "2022-08-08T18:06:18.531480Z",
     "shell.execute_reply.started": "2022-08-08T18:06:18.521919Z",
     "shell.execute_reply": "2022-08-08T18:06:18.530250Z"
    },
    "trusted": true
   },
   "execution_count": 27,
   "outputs": []
  },
  {
   "cell_type": "markdown",
   "source": "## Discussion\n\nWe've built two models that compete against one another. The end result is a network that can produce images of dogs (sort of...).\n\nAs epochs progressed, any intermediate images of dogs were seen to become increasingly more distinctive and pronounced. The training loss for both networks was also seen to evolve over time which produced lower loss scores.\n\nBecause the training time was very long, and my GPU quota on Kaggle was not sufficient, I cut my training epochs to 256 max. Further training would have helped produce better results. Additionally, I believe adding a 1024 neuron layer to the generator and discriminator would have been beneficial. Lastly, I am actually not sure how this would work, but a pre-trained dog classifier could have been used to fast track the progress.\n\n## Conclusion\n\nWe have created a network that spawns hell-like images of anthropomorphically terrifying dogs. The network could certainly be improved to increase the realism of said hell dogs. Improvements would be additional layers to the discriminator and generator or using a pre-trained network like AlexNet. ",
   "metadata": {}
  }
 ]
}