Lab_2_Defenses_Against_Evasion_Attacks.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "provenance": [],
      "toc_visible": true,
      "authorship_tag": "ABX9TyO9e4TOUsG22QfhXFSutLNh",
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "language_info": {
      "name": "python"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/oaramoon/AML/blob/main/Lab_2_Defenses_Against_Evasion_Attacks.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "#**Lab 2: Defnses Against Evasion Attack**\n",
        "\n",
        "Welcome to our hands-on lab where you will actively engage in assessing the robustness of ML models against evasion attacks. This session is an extension of our previous exploration in Lab 1, where we developed a Projected Gradient Descent (PGD) attack. We start by examining seemingly effective defense techniques - pay attention to the significance of \"effective\" within double quotes. These methods include JPEG compression, random model selection, etc. You'll investigate how these strategies enhance adversarial robustness against standard evasion attacks, like the PGD attack.\n",
        "\n",
        "In the second segment, our focus pivots to the critical role of adaptive attacks in evaluating these empirical defenses. Relying solely on standard, off-the-shelf attacks may yield a misleading sense of security. We will dissect two advanced adaptive attack techniques: the Expectation of Transformation attack and the Backward Pass Differentiable Approximation attack. Through practical exercises, you will apply these methods to critically assess the true resilience of these defense strategies. This investigation is crucial in understanding the intricacies of defending machine learning models against evolving adversarial tactics.\n",
        "\n",
        "We conclude our lab with a comprehensive analysis of adversarially trained models. Often celebrated for their resilience to evasion attacks, these models, however, have a propensity to overfit to specific types of threats. Through interactive discussions and experiments, you will explore the multifaceted challenges in achieving genuine robustness in adversarial machine learning contexts. This final section aims to provide you with an in-depth understanding of the strengths and limitations of current defense methods. It's designed to foster a well-rounded perspective on the ongoing endeavors to enhance the security and reliability of machine learning systems in the face of sophisticated adversarial threats."
      ],
      "metadata": {
        "id": "maL2uTepIZlb"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "##**Part 1: Setting up the Enviroment**\n",
        "\n",
        "In this part of the lab, we will establish the groundwork for our experiments. To begin, we will download the lab materials from the course repositoryand then set up the environment necessary for our experiments. We will then load our unprotected target model, which we refer to as the baseline classifier. This model is a convolutional neural network trained for the task of classifying objects in the CIFAR-10 dataset.\n",
        "\n",
        "Following this setup, we will utilize the PGD (Projected Gradient Descent) attack, which we developed in Lab 1, to evaluate the security of our baseline classifier. This exercise provide an understanding of the model's level of vulnerability to evasion (or adversarial example) attacks."
      ],
      "metadata": {
        "id": "7DEnfM4DJCnQ"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "### **Downloading the Lab Material**\n"
      ],
      "metadata": {
        "id": "0qKBqqmLNA7n"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "!git clone https://github.com/oaramoon/AML.git"
      ],
      "metadata": {
        "id": "_3-L3rTLNNio"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "import tensorflow as tf\n",
        "import numpy as np\n",
        "import matplotlib.pyplot as plt\n",
        "from matplotlib.ticker import MaxNLocator\n",
        "from tqdm import tqdm\n",
        "import time\n",
        "import random\n",
        "from AML.TST196 import load_cifar10_model"
      ],
      "metadata": {
        "id": "A4djvQO6JHJF"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "### **Getting Familiar with CIFAR10 Dataset**\n",
        "The CIFAR-10 dataset is a widely-used benchmark dataset in the field of machine learning, particularly for image classification tasks. For those not familiar with it, let's provide a brief introduction.\n",
        "\n",
        "The CIFAR-10 dataset consists of 60,000 color images, each of 32x32 pixels, spread across 10 classes, with 6,000 images per class. It is split into a standard configuration of 50,000 training images and 10,000 test images. Each image is to be classified into one of the 10 different classes, which represent different objects or animals: airplane, automobile, bird, cat, deer, dog, frog, horse, ship, and truck.\n",
        "\n",
        "In the following cell, we will load the CIFAR-10 dataset from Keras datasets and visualize some sample images from the training set."
      ],
      "metadata": {
        "id": "4ksz3OJnIsmt"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "(x_train, y_train), (x_test, y_test) = tf.keras.datasets.cifar10.load_data()\n",
        "num_classes = 10\n",
        "\n",
        "x_train = x_train.astype(\"float32\") /255.0\n",
        "x_test = x_test.astype(\"float32\") /255.0\n",
        "input_shape = x_train[0].shape\n",
        "\n",
        "print(f\"There are {x_train.shape[0]} training and {x_test.shape[0]} test samples in CIFAR10 dataset.\")\n",
        "print(f\"Each sample is of dimension {input_shape}\")\n",
        "\n",
        "# CIFAR-10 dataset labels\n",
        "class_names = ['airplane', 'automobile', 'bird', 'cat', 'deer',\n",
        "               'dog', 'frog', 'horse', 'ship', 'truck']\n",
        "\n",
        "# Select 100 random images from the training set\n",
        "indices = tf.random.uniform([100], 0, len(x_train), dtype=tf.int32)\n",
        "sample_images = tf.gather(x_train, indices)\n",
        "sample_labels = tf.gather(y_train, indices)\n",
        "\n",
        "# Plot the selected images\n",
        "plt.figure(figsize=(15,15))\n",
        "for i in range(100):\n",
        "    plt.subplot(10,10,i+1)\n",
        "    plt.xticks([])\n",
        "    plt.yticks([])\n",
        "    plt.grid(False)\n",
        "    plt.imshow(sample_images[i])\n",
        "    plt.title(class_names[sample_labels[i][0]])\n",
        "plt.tight_layout()\n",
        "plt.show()"
      ],
      "metadata": {
        "id": "LPZFa76jI0oG"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "### **Loading the Baseline Classifier**"
      ],
      "metadata": {
        "id": "PtRinyfEkS12"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "baseline_classifier = load_cifar10_model(model_path=\"./AML/baseline_cifar10\")"
      ],
      "metadata": {
        "id": "fo5y6dgbkY_c"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "### **Untargeted PGD Attack (From Lab 1)**"
      ],
      "metadata": {
        "id": "oKvWb0_uJVhe"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "def get_loss_gradient_wrt_input(x, y, target_model, preprocess_input, num_classes):\n",
        "    # Define the training loss using categorical crossentropy\n",
        "    training_loss = tf.keras.losses.CategoricalCrossentropy()\n",
        "\n",
        "    # Use GradientTape for automatic differentiation\n",
        "    with tf.GradientTape() as tape:\n",
        "        tape.watch(x)  # Instruct tape to watch the input tensor 'x'\n",
        "\n",
        "        # Preprocess the input and then make predictions using the target model\n",
        "        prediction = target_model(preprocess_input(x))\n",
        "\n",
        "        # Compute the loss between the true labels and predictions\n",
        "        loss = training_loss(tf.one_hot(y, num_classes), prediction)\n",
        "\n",
        "    # Compute the gradient of the loss with respect to the input\n",
        "    gradient = tape.gradient(loss, x)\n",
        "    return gradient\n",
        "\n",
        "def untargeted_pgd_attack(target_model, preprocess_input, img_batch, img_labels, epsilon, num_ittr, num_classes):\n",
        "    # Convert image batch to tensors and create a copy for original images\n",
        "    perturbed_images = tf.convert_to_tensor(img_batch)\n",
        "    original_images = tf.identity(perturbed_images)\n",
        "\n",
        "    # Define the step size for the perturbation\n",
        "    step = (epsilon / num_ittr) * 1.25\n",
        "\n",
        "    # Iteratively apply perturbations\n",
        "    for _ in tqdm(range(num_ittr)):\n",
        "        # Calculate gradients of loss w.r.t. the perturbed images\n",
        "        grads = get_loss_gradient_wrt_input(target_model=target_model, x=perturbed_images, y=img_labels, preprocess_input=preprocess_input, num_classes=num_classes)\n",
        "\n",
        "        # Replace any NaNs in gradients with zeros\n",
        "        grads = tf.where(tf.math.is_nan(grads), tf.zeros_like(grads), grads)\n",
        "\n",
        "        # Compute the sign of the gradients\n",
        "        sign_grads = tf.sign(grads)\n",
        "\n",
        "        # Update perturbed images by a step in the direction of the sign of gradients\n",
        "        perturbed_images = perturbed_images + step * sign_grads\n",
        "\n",
        "        # Ensure that the perturbations stay within the epsilon bounds\n",
        "        perturbation = tf.clip_by_value(perturbed_images - original_images, -epsilon, epsilon)\n",
        "        perturbed_images = original_images + perturbation\n",
        "\n",
        "    # Clip the perturbed images to be within valid image range [0, 1]\n",
        "    perturbed_images = tf.clip_by_value(perturbed_images, 0.0, 1.0)\n",
        "\n",
        "    # Return the perturbed images as a numpy array\n",
        "    return perturbed_images.numpy()\n"
      ],
      "metadata": {
        "id": "ejo3dZDIJi0b"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "### **Target Images**\n",
        "\n",
        "To assess the robustness of the baseline classifier, we will utilize the first 100 samples from the CIFAR-10 test set. Why only 100 samples? This limitation is due to the lack of access to GPUs on Google Colab. In real-world evaluations, a larger sample size is often necessary. However, for the purposes of this lab, we will work within these constraints."
      ],
      "metadata": {
        "id": "bbZUxc6RIneV"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "target_samples_x = tf.convert_to_tensor(x_test[:100,::],dtype=tf.float32)\n",
        "target_samples_org_y = y_test[:100].flatten()"
      ],
      "metadata": {
        "id": "Gww3IrcrIX1d"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "### **Baseline Classifier Vs Untargeted PGD Attack**\n",
        "In the following code block, we will evaluate the effectiveness of our PGD attack, developed in Lab 1, against the baseline classifier. We'll measure its success rate using the 100 target samples that were selected earlier."
      ],
      "metadata": {
        "id": "6VrqUulEKAA_"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "adv_imgs_for_baseline_model = untargeted_pgd_attack(target_model=baseline_classifier, preprocess_input=lambda x: x, img_batch=target_samples_x, img_labels=target_samples_org_y, epsilon=4.0/255.0, num_ittr=10, num_classes=num_classes)\n",
        "\n",
        "def untargeted_attack_success_rate(model, preprocess_input, x, y):\n",
        "    predictions = model(preprocess_input(x))\n",
        "    return np.mean(np.argmax(predictions,axis=1) != y)\n",
        "\n",
        "attack_attack_success_on_baseline =  untargeted_attack_success_rate(model=baseline_classifier, preprocess_input=lambda x: x, x=adv_imgs_for_baseline_model, y=target_samples_org_y)\n",
        "print(f\"Attack success rate on baseline model is {100*attack_attack_success_on_baseline:.2f}%.\")"
      ],
      "metadata": {
        "id": "LF6WHQ9PKTUB"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "## **Part 2: Defenses**\n",
        "\n",
        "In the second part of the lab, we'll revisit several defense mechanisms discussed in Lecture 4 of our course. Our objective will be to evaluate the success rate of our off-the-shelf PGD attack against these defenses."
      ],
      "metadata": {
        "id": "WVthUV2vJxVF"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "### **Input Transformation JPEG Compression**\n",
        "One relatively straightforward defense mechanism in the realm of machine learning is to incorporate JPEG compression as an input preprocessing step. The rationale behind this approach is rooted in the nature of the JPEG compression algorithm itself. JPEG, primarily designed for efficient image storage by reducing file size, achieves this through a lossy compression technique. This process inherently filters out certain high-frequency components of the image, which are often imperceptible to the human eye. In the context of adversarial machine learning, this characteristic of JPEG compression can be leveraged to mitigate the impact of adversarial noise. The hypothesis is that during the compression process, some of the deliberately crafted perturbations, which are typically subtle and high-frequency, might be discarded along with other non-essential image details. Consequently, this preprocessing step has the potential to enhance the model's resilience by diminishing the effectiveness of adversarial inputs designed to mislead the model.\n",
        "\n"
      ],
      "metadata": {
        "id": "JeZolu9IxamB"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "from PIL import Image\n",
        "import io\n",
        "\n",
        "\n",
        "def jpeg_compress_batch(input_imgs, quality=50):\n",
        "    \"\"\"\n",
        "    Compresses a batch of images (either TensorFlow tensors or NumPy arrays) using JPEG compression\n",
        "    and returns the compressed images as a TensorFlow tensor.\n",
        "\n",
        "    Args:\n",
        "    - input_imgs (tf.Tensor or np.ndarray): The batch of images with shape (batch_size, height, width, channels).\n",
        "    - quality (int): The quality of the JPEG compression (1 to 100, higher means better quality and less compression).\n",
        "\n",
        "    Returns:\n",
        "    - tf.Tensor: The batch of compressed images as a TensorFlow tensor.\n",
        "    \"\"\"\n",
        "    compressed_imgs = []\n",
        "\n",
        "    # Check if the input is a TensorFlow tensor and convert to a NumPy array if so\n",
        "    if tf.is_tensor(input_imgs):\n",
        "        np_imgs = input_imgs.numpy()\n",
        "    else:\n",
        "        np_imgs = input_imgs\n",
        "\n",
        "    for np_img in np_imgs:\n",
        "        # Ensure the image is in uint8 format\n",
        "        if np_img.dtype != np.uint8:\n",
        "            np_img = (np_img * 255).astype('uint8')\n",
        "\n",
        "        pil_img = Image.fromarray(np_img)\n",
        "\n",
        "        # Save the image to a buffer with JPEG compression\n",
        "        buffer = io.BytesIO()\n",
        "        pil_img.save(buffer, format=\"JPEG\", quality=quality)\n",
        "\n",
        "        # Load the image back from the buffer\n",
        "        compressed_img = Image.open(buffer)\n",
        "\n",
        "        # Convert the PIL Image back to a NumPy array and add to the list\n",
        "        compressed_imgs.append(np.array(compressed_img)/255.0)\n",
        "\n",
        "    # Convert the list of compressed images back to a TensorFlow tensor\n",
        "    return tf.convert_to_tensor(compressed_imgs,dtype=tf.float32)\n",
        "\n",
        "\n"
      ],
      "metadata": {
        "id": "0WMVzCxTxiQU"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "try:\n",
        "  adv_imgs_for_jpeg_defense = untargeted_pgd_attack(target_model=baseline_classifier, preprocess_input=jpeg_compress_batch, img_batch=target_samples_x, img_labels=target_samples_org_y, epsilon=4.0/255.0, num_ittr=10, num_classes=num_classes)\n",
        "\n",
        "  attack_attack_success_on_transform_defense =  untargeted_attack_success_rate(model=baseline_classifier, preprocess_input=jpeg_compress_batch, x=adv_imgs_for_jpeg_defense, y=target_samples_org_y)\n",
        "  print(f\"Attack success rate on transform defense model is {100*attack_attack_success_on_transform_defense:.2f}%.\")\n",
        "\n",
        "except Exception as e:\n",
        "  print(e,\"\\nCould not generate adversarial exapmles!\\n\")\n"
      ],
      "metadata": {
        "id": "0QePT5XPxv85"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "#### **Discussion**\n",
        "Why can't we directly generate adversarial examples for a system using JPEG compression? A partial explanation is that we use the PIL library for JPEG compression, and these operations aren't integrated into TensorFlow's computation graph, hence TensorFlow cannot compute the necessary gradients. But this is not the core issue. Even if we were to replicate all steps of JPEG compression using TensorFlow operations, we would still face challenges in computing the gradients required for creating adversarial examples. This is primarily due to certain steps in the JPEG compression process that are inherently non-differentiable, such as the quantization step and the discarding of high-frequency components following the Discrete Cosine Transform (DCT). These steps, which involve forms of thresholding, introduce non-differentiability. As discussed in our lectures, defenses like JPEG compression aim to enhance robustness against evasion attacks by disrupting the calculation of gradients with respect to the model's input. This disruption makes it difficult for attackers to craft adversarial examples using gradient-based methods, a common tactic in evasion attacks."
      ],
      "metadata": {
        "id": "FMssHOF-qj4v"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "But how about the adversarial examples generated for baseline model? Are they still effective? by running the code block below, you observe for yorslef."
      ],
      "metadata": {
        "id": "yEGqIXtHvtEm"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "attack_attack_success_on_JPEG_defense =  untargeted_attack_success_rate(model=baseline_classifier, preprocess_input=jpeg_compress_batch, x=adv_imgs_for_baseline_model, y=target_samples_org_y)\n",
        "print(f\"Attack success rate of baseline adversarial exmaples on JPEG compression defense is {100*attack_attack_success_on_JPEG_defense:.2f}%.\")"
      ],
      "metadata": {
        "id": "yxs-SW2gvn7K"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        " The success rate of adversarial examples generated against our baseline classifier drops drastically—from 92% in the unprotected scenario to just 49% when JPEG compression is applied. This demonstrates that even a straightforward method like JPEG compression can markedly enhance a model's robustness against a PGD attack."
      ],
      "metadata": {
        "id": "NQGud_Y6v_XH"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "### **Randomization**\n",
        "Another category of defenses against evasion attacks, as discussed in Lecture 4, aims to introduce randomness into the inference process. This strategy is based on the idea that by randomizing the computed gradients used in generating adversarial examples, the system's robustness can be improved, potentially thwarting such attacks."
      ],
      "metadata": {
        "id": "PtfH85LM_RMp"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "#### **Random Neuron Pruning**\n",
        "\n",
        "An example of this technique is the Random Neuron Pruning defense. In this approach, during the inference phase, we don't use the exact network activations computed in the standard manner. Instead, a subset of neurons is randomly dropped. This pruning is biased such that neuron with larger activations are more likely to be removed. The neruons that remain are then scaled up, similar to the dropout technique, to maintain the overall integrity of the network's output.\n",
        "\n",
        "In the upcoming code block, we have implemented this Random Neuron Pruning defense. You are encouraged to test the resilience of this defense mechanism by running the PGD attack against it. This exercise will allow you to evaluate firsthand how effective Random Neuron Pruning is in enhancing the system's robustness against evasion attacks."
      ],
      "metadata": {
        "id": "gbcWwupkNyTv"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "class RandomPruned:\n",
        "    def __init__(self, obj):\n",
        "        self.model = obj\n",
        "\n",
        "    def __call__(self, x, *args, **kwargs):\n",
        "      for layer in self.model.layers:\n",
        "          x = layer(x)\n",
        "          if isinstance(layer, tf.keras.layers.Conv2D):\n",
        "              _, a, b, c = x.shape\n",
        "              p = tf.abs(x) / (tf.reduce_sum(tf.abs(x), axis=(1, 2, 3), keepdims=True) + 1e-8)  # Adding a small constant for numerical stability\n",
        "              p_keep = 1 - tf.exp(-a * b * c / 3 * p)\n",
        "              p_keep = tf.clip_by_value(p_keep, 1e-8, 1.0)  # Ensure p_keep is not zero\n",
        "              keep = tf.random.uniform(p_keep.shape) < p_keep\n",
        "              x = tf.cast(keep, tf.float32) * x / (p_keep + 1e-8)  # Adding a small constant to avoid division by zero\n",
        "\n",
        "      return x\n",
        "\n",
        "classifier_w_random_pruning = RandomPruned(baseline_classifier)\n",
        "\n",
        "adv_imgs_for_RNP_model = untargeted_pgd_attack(target_model=classifier_w_random_pruning, preprocess_input=lambda x: x, img_batch=target_samples_x, img_labels=target_samples_org_y, epsilon=4.0/255.0, num_ittr=10, num_classes=num_classes)"
      ],
      "metadata": {
        "id": "ie4dsRCCxcjf"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "attack_attack_success_on_RNP_defense =  sum([untargeted_attack_success_rate(model=classifier_w_random_pruning, preprocess_input=lambda x:x, x=adv_imgs_for_RNP_model, y=target_samples_org_y) for _ in range(10)])/10\n",
        "print(f\"Attack success rate of baseline adversarial exmaples on random neuron pruning defense is {100*attack_attack_success_on_RNP_defense:.2f}%.\")"
      ],
      "metadata": {
        "id": "OdDkI0VQ2rhM"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "As evidenced by our results, it's clear that Random Neuron Pruning enhances the system's robustness. The PGD attack's effectiveness is notably reduced, achieving only about a 40% success rate against this defense mechanism. This demonstrates the significant impact that Random Neuron Pruning has in bolstering the system's resistance to such attacks."
      ],
      "metadata": {
        "id": "SUw1UVpP3voG"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "#### **Random Model Selection**\n",
        "One notable downside of the Random Neuron Pruning defense is its detrimental effect on the model's test accuracy – you'll have to take my word for it. To address this issue while still leveraging the benefits of randomization in mitigating evasion attacks, we can turn to the Random Model Selection technique. This approach doesn't rely on a single model. Instead, it involves training multiple models, each with a distinct architecture, for the same task. During the inference stage, one of these models is randomly selected to process the test sample, adding an element of unpredictability to counter evasion attacks.\n",
        "\n",
        "In the next code block, we've set up the Random Model Selection defense. You're encouraged to evaluate this defense's effectiveness by launching the PGD attack against it."
      ],
      "metadata": {
        "id": "E-hpZCKw4zOt"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "import random\n",
        "\n",
        "class RandomModelSelection:\n",
        "  def __init__(self, models):\n",
        "    self.models = models\n",
        "    self.num_models = len(models)\n",
        "\n",
        "  def __call__(self, x, *args, **kwargs):\n",
        "    # Randomly select a model\n",
        "    selected_model = random.choice(self.models)\n",
        "\n",
        "    # Feed x to the selected model and return its output\n",
        "    return selected_model(x, *args, **kwargs)\n",
        "\n",
        "model_1 = load_cifar10_model(model_path=\"./AML/baseline_cifar10\")\n",
        "model_2 = load_cifar10_model(model_path=\"./AML/surrogate_cifar10_0\")\n",
        "model_3 = load_cifar10_model(model_path=\"./AML/surrogate_cifar10_1\")\n",
        "model_4 = load_cifar10_model(model_path=\"./AML/surrogate_cifar10_2\")\n",
        "\n",
        "random_select_classifier = RandomModelSelection([model_1,model_2,model_3, model_4])\n",
        "\n",
        "adv_imgs_for_RMS_model = untargeted_pgd_attack(target_model=random_select_classifier, preprocess_input=lambda x: x, img_batch=target_samples_x, img_labels=target_samples_org_y, epsilon=4.0/255.0, num_ittr=10, num_classes=num_classes)"
      ],
      "metadata": {
        "id": "UdD7j7bT_WhM"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "attack_success_on_RMS_defense =  sum([untargeted_attack_success_rate(model=random_select_classifier, preprocess_input=lambda x:x, x=adv_imgs_for_RMS_model, y=target_samples_org_y) for _ in range(10)])/10\n",
        "print(f\"Attack success rate of baseline adversarial exmaples on random model selectiohn defense is {100*attack_success_on_RMS_defense:.2f}%.\")"
      ],
      "metadata": {
        "id": "5DWd_sI3BO75"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "It's evident from our observations that the success rate of the PGD attack on the Random Model Selection (RMS) defense is lower compared to the baseline model. However, it is higher than what we observed with the Random Neuron Pruning defense. This outcome is primarily because the RMS defense introduces a lesser degree of randomization compared to Neuron Pruning. Despite this, the RMS defense still marks an improvement in robustness against evasion attacks, highlighting its potential as a viable defense strategy."
      ],
      "metadata": {
        "id": "8EZ0soMg-1L9"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "## **Part 3: Adaptive Attacks**\n",
        "\n",
        "So far in our lab, we have focused on evaluating the security of various defenses using the standard, off-the-shelf PGD attack. However, as we discussed in our lectures, this approach represents a somewhat naive view of security evaluation. An adversary in a real-world scenario won't limit themselves to these readily available attack methods. Especially when there's a significant incentive to circumvent a system, attackers are likely to get more creative and determined in finding ways to breach defenses.\n",
        "\n",
        "This brings us to the critical concept of adaptive attacks, an essential aspect to consider when evaluating the robustness of any defense mechanism. In an adaptive attack scenario, we assume that the attacker has a white-box level of access, meaning they have complete knowledge of the defense system, including its intricacies and workings. The attacker's goal is to exploit this knowledge to devise strategies specifically tailored to undermine the defense.\n",
        "\n",
        "When considering adaptive attacks, the question we need to ask is: 'How could an adversary potentially break this system?' If it seems there's no feasible way to breach the defense, that's a promising sign. However, it doesn't automatically guarantee the defense is impenetrable. It's important to recognize that the inability to conceive an effective adaptive attack might be due to limitations in the evaluator's expertise or imagination. Therefore, while adaptive attacks are a more rigorous and realistic method for testing defenses, they still might not cover all possible attack vectors an actual adversary might employ.\n",
        "\n",
        "In what follows, we'll explore in more detail the strategies behind adaptive attacks, focusing specifically on two prominent and effective methods: the Expectation over Transformation (EOT) attack and the Backward Pass Differentiable Approximation (BPDA) attack. These sessions will encourage you to adopt an adversary's mindset, understanding not just how these attacks work, but also how they can be creatively applied to challenge defense mechanisms."
      ],
      "metadata": {
        "id": "4WEWMLu_CfMa"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "#### **Expectation over Transformation vs on Random Neuron Pruning Defense**\n",
        "\n",
        "The Expectation over Transformation (EOT) attack** is a sophisticated strategy developed to counter defenses that introduce randomness into model inference. Consider a scenario where a classifier undergoes random transformations—this could involve randomizing the input, selecting models randomly from a pol of models, or employing random neuron pruning. In each case, these transformations represent a set \\( $T$ \\). EOT addresses the challenge posed by these random defenses by not targeting a single static instance of the model. Instead, it focuses on the average behavior over multiple instances or transformations.\n",
        "\n",
        "Crucially, EOT leverages a key insight: the gradient of the expected transformation outcome is equal to the expectation of the gradient following the transformation. In practical terms, EOT iteratively estimates this expectation with each step in the gradient descent process, using sampling. This method allows for the refinement of the adversarial example in response to the defense's randomness.\n",
        "\n",
        "In the following code block, you will find the implementation of this defense strategy. You cal also evaluate its effectiveness against the Random Neuron Pruning defense. however, Note that sinceour code is executing on a CPU, and considering the increased number of gradient computations required, be aware that the attack process may take a significantly longer time to complete."
      ],
      "metadata": {
        "id": "qEYh6QKrDQXZ"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "def adaptive_attack_against_RNP_defense(target_model, preprocess_input, img_batch, img_labels, epsilon, num_ittr, num_classes, num_trials):\n",
        "    # Convert image batch to a tensor and keep a copy of the original images\n",
        "    perturbed_images = tf.convert_to_tensor(img_batch)\n",
        "    original_images = tf.identity(perturbed_images)\n",
        "\n",
        "    # Define the step size for the perturbation\n",
        "    step = (epsilon / num_ittr) * 1.25\n",
        "\n",
        "    # Iteratively apply perturbations over a number of iterations\n",
        "    for _ in tqdm(range(num_ittr)):\n",
        "        # Initialize accumulated gradients as zeros\n",
        "        accumulated_grads = tf.zeros_like(perturbed_images)\n",
        "\n",
        "        # Accumulate gradients over a number of trials to average the effect of random neuron pruning\n",
        "        for _ in tqdm(range(num_trials)):\n",
        "            accumulated_grads += get_loss_gradient_wrt_input(target_model=target_model, x=perturbed_images, y=img_labels, preprocess_input=preprocess_input, num_classes=num_classes)\n",
        "\n",
        "        # Calculate the average of the accumulated gradients\n",
        "        average_grads = accumulated_grads / num_trials\n",
        "        # Replace NaNs in gradients with zeros\n",
        "        average_grads = tf.where(tf.math.is_nan(average_grads), tf.zeros_like(average_grads), average_grads)\n",
        "\n",
        "        # Update perturbed images using the sign of the averaged gradients\n",
        "        sign_grads = tf.sign(average_grads)\n",
        "        perturbed_images = perturbed_images + step * sign_grads\n",
        "\n",
        "        # Ensure that the perturbation stays within the epsilon bounds\n",
        "        perturbation = tf.clip_by_value(perturbed_images - original_images, -epsilon, epsilon)\n",
        "        perturbed_images = original_images + perturbation\n",
        "\n",
        "    # Clip the perturbed images to be within the valid image range [0.0, 1.0]\n",
        "    perturbed_images = tf.clip_by_value(perturbed_images, 0.0, 1.0)\n",
        "\n",
        "    # Return the perturbed images as a numpy array\n",
        "    return perturbed_images.numpy()\n",
        "\n",
        "# Applying the adaptive attack against the Random Neuron Pruning defense\n",
        "adaptive_adv_imgs_for_RNP_defense = adaptive_attack_against_RNP_defense(target_model=classifier_w_random_pruning, preprocess_input=lambda x: x, img_batch=target_samples_x, img_labels=target_samples_org_y, epsilon=4.0/255.0, num_ittr=100, num_classes=num_classes, num_trials=100)\n"
      ],
      "metadata": {
        "id": "uCuxApstZ862"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "attack_success_of_EoT_on_RNP_defense =  untargeted_attack_success_rate(model=classifier_w_random_pruning, preprocess_input=lambda x:x, x=adaptive_adv_imgs_for_RNP_defense, y=target_samples_org_y)\n",
        "print(f\"Attack success rate of baseline adversarial exmaples on random neron pruning defense is {100*attack_success_of_EoT_on_RNP_defense:.2f}%.\")"
      ],
      "metadata": {
        "id": "SbV88k7Ka2vY"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "#### **Backward Pass Differentiable Approximation Attack vs JPEG Compression Defense**\n",
        "\n",
        "\n",
        "\n",
        "The Backward Pass Differentiable Approximation (BPDA) attack, is designed to tackle the challenges posed by defense such as JPEG compression that feature non-differentiable layers. In this approach, the model is represented as \\( f \\), which includes a specific layer \\( $f_i$ \\) characterized by its non-differentiability. BPDA's strategy is centered around the concept of finding a differentiable approximation, denoted as \\( g(x) \\), that closely mirrors the behavior of \\( $f_i(x)$ \\).\n",
        "\n",
        "During the forward pass in the attack process, the model operates as normal, inclusive of the non-differentiable layer \\( $f_i$ \\). However, the innovation of BPDA comes into play during the backward pass, crucial for gradient calculation. In this phase, \\( $f_i$ \\) is substituted with the approximate \\( g(x) \\). This substitution allows for the derivation of gradients that, despite not being perfectly accurate, are approximate enough to assist in crafting effective adversarial examples."
      ],
      "metadata": {
        "id": "PBK15TXR6DNC"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "def adaptive_attack_for_JPEG_compression(target_model, preprocess_input, img_batch, img_labels, epsilon, num_ittr, num_classes):\n",
        "    # Convert image batch to a tensor and keep a copy of original images\n",
        "    perturbed_images = tf.convert_to_tensor(img_batch)\n",
        "    original_images = tf.identity(perturbed_images)\n",
        "\n",
        "    # Define the step size for the perturbation\n",
        "    step = (epsilon / num_ittr) * 1.25\n",
        "\n",
        "    # Iteratively apply perturbations\n",
        "    for _ in tqdm(range(num_ittr)):\n",
        "        # Apply JPEG compression to perturbed images (forward pass)\n",
        "        compressed_pertubed_images = preprocess_input(perturbed_images)\n",
        "\n",
        "        # Compute gradients with respect to the compressed images\n",
        "        grads_for_compressed_images = get_loss_gradient_wrt_input(target_model=target_model, x=compressed_pertubed_images, y=img_labels, preprocess_input=lambda x: x, num_classes=num_classes)\n",
        "\n",
        "        # Replace any NaNs in gradients with zeros\n",
        "        grads_for_compressed_images = tf.where(tf.math.is_nan(grads_for_compressed_images), tf.zeros_like(grads_for_compressed_images), grads_for_compressed_images)\n",
        "\n",
        "        # Directly use the gradients of the compressed images as our gradient estimate\n",
        "        # This corresponds to the backward pass of BPDA\n",
        "        grads = grads_for_compressed_images\n",
        "\n",
        "        # Compute the sign of the gradients\n",
        "        sign_grads = tf.sign(grads)\n",
        "\n",
        "        # Update perturbed images by a step in the direction of the sign of gradients\n",
        "        perturbed_images = perturbed_images + step * sign_grads\n",
        "\n",
        "        # Ensure that the perturbations stay within the epsilon bounds\n",
        "        perturbation = tf.clip_by_value(perturbed_images - original_images, -epsilon, epsilon)\n",
        "        perturbed_images = original_images + perturbation\n",
        "\n",
        "    # Clip the perturbed images to be within the valid image range [0, 1]\n",
        "    perturbed_images = tf.clip_by_value(perturbed_images, 0.0, 1.0)\n",
        "\n",
        "    # Return the perturbed images as a numpy array\n",
        "    return perturbed_images.numpy()\n",
        "\n",
        "# Applying the BPDA attack on the JPEG compression defense\n",
        "adaptive_adv_imgs_for_jpeg_defense = adaptive_attack_for_JPEG_compression(target_model=baseline_classifier, preprocess_input=jpeg_compress_batch, img_batch=target_samples_x, img_labels=target_samples_org_y, epsilon=4.0/255.0, num_ittr=10, num_classes=num_classes)\n"
      ],
      "metadata": {
        "id": "rh5QWlnB6CSq"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "attack_attack_success_of_adaptive_attack_on_JPEG_defense =  untargeted_attack_success_rate(model=baseline_classifier, preprocess_input=jpeg_compress_batch, x=adaptive_adv_imgs_for_jpeg_defense, y=target_samples_org_y)\n",
        "print(f\"Attack success rate of adaptive attack on JPEG compression defense is {100*attack_attack_success_of_adaptive_attack_on_JPEG_defense:.2f}%.\")"
      ],
      "metadata": {
        "id": "Kg14bAVPGuAl"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "## **Exercise: Hands-on Security Evaluation**\n",
        "\n",
        "Imagine you're given the task of assessing the security of the following two systems:\n",
        "\n",
        "- **System 1** uses strong random input augmentation techniques to mitigate the effect of adversarial noise.\n",
        "\n",
        "- **System 2** is designed to defend against evasion attacks through majority voting mechanism.\n",
        "\n",
        "\n",
        " How secure do you think these systems are? what is highest attack success rate you can achieve? As part of your evaluation, note that you have white-box access to the system, meaning you can see and understand all its inner workings. Put on your adversary hat and brainstorm the most effective untargeted attack strategy for this system. Consider the system's strengths and vulnerabilities, and how an attacker could exploit them."
      ],
      "metadata": {
        "id": "vz645OfJm7eD"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "### **System 1: Strong Data Augmentation**\n",
        "\n",
        "In the following code block, we introduce the augment_strong function. This function is specifically designed to apply a sequence of transformations to images. The goal of these transformations is to potentially disrupt adversarial perturbations, thereby serving as an effective defense mechanism against evasion attacks.\n",
        "\n",
        "**Key Features of `augment_strong`:**\n",
        "\n",
        "- **Adjustable Transformation Intensity**: The `strength` parameter in the function controls the intensity of the transformations, affecting the brightness, contrast, saturation, and hue adjustments.\n",
        "\n",
        "- **Diverse Transformations**: The function includes four transformation types: random adjustments to brightness, contrast, saturation, and hue. Each type of transformation is defined as a separate internal function.\n",
        "\n",
        "- **Randomized Transformation Application**: The transformations are applied in a shuffled order, introducing unpredictability that is crucial in thwarting structured adversarial attacks.\n",
        "\n",
        "- **Additional Image Augmentations**: Beyond color adjustments, the function also performs horizontal flipping and random cropping after reflective padding, adding further variability to the transformed images.\n",
        "\n",
        "- **Visualization Tool**: The `display_original_and_transformed_images` function enables a side-by-side visual comparison of the original and augmented images, illustrating the effects of the applied augmentations."
      ],
      "metadata": {
        "id": "1lJ2rgc2BcN-"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "def augment_strong(image, strength=.75):\n",
        "    # Define the intensity of different transformations based on the strength parameter\n",
        "    brightness = 0.8 * strength\n",
        "    contrast = 0.8 * strength\n",
        "    saturation = 0.8 * strength\n",
        "    hue = 0.2 * strength\n",
        "\n",
        "    def apply_transform(i, x):\n",
        "        # Define functions for each type of transformation\n",
        "        def brightness_foo():\n",
        "            return tf.image.random_brightness(x, max_delta=brightness)\n",
        "\n",
        "        def contrast_foo():\n",
        "            return tf.image.random_contrast(x, lower=1-contrast, upper=1+contrast)\n",
        "\n",
        "        def saturation_foo():\n",
        "            return tf.image.random_saturation(x, lower=1-saturation, upper=1+saturation)\n",
        "\n",
        "        def hue_foo():\n",
        "            return tf.image.random_hue(x, max_delta=hue)\n",
        "\n",
        "        # Apply one of the transformations based on the value of i\n",
        "        x = tf.cond(tf.less(i, 2),\n",
        "                    lambda: tf.cond(tf.less(i, 1), brightness_foo, contrast_foo),\n",
        "                    lambda: tf.cond(tf.less(i, 3), saturation_foo, hue_foo))\n",
        "        return x\n",
        "\n",
        "    def augment(x, y):\n",
        "        \"\"\"\n",
        "        Applies flip and shift augmentation to the given example.\n",
        "        Flips the image horizontally and randomly crops it after padding.\n",
        "        \"\"\"\n",
        "        x_shape = tf.shape(x)\n",
        "        x = tf.image.random_flip_left_right(x)\n",
        "        x = tf.pad(x, [[0] * 2, [4] * 2, [4] * 2, [0] * 2], mode='REFLECT')\n",
        "        return tf.image.random_crop(x, x_shape), y\n",
        "\n",
        "    # Shuffle the order in which transformations are applied\n",
        "    perm = tf.random.shuffle(tf.range(4))\n",
        "    for i in range(4):\n",
        "        image = apply_transform(perm[i], image)\n",
        "        image = tf.clip_by_value(image, 0., 1.)  # Clip values to maintain valid image range\n",
        "    return augment(image, None)[0]\n",
        "\n",
        "\n",
        "def display_original_and_transformed_images(original, transformed):\n",
        "    \"\"\"\n",
        "    Displays the original and transformed images side by side for comparison.\n",
        "    \"\"\"\n",
        "    fig, axes = plt.subplots(1, 2, figsize=(5, 10))  # Create subplot for two images\n",
        "\n",
        "    # Display the original image\n",
        "    axes[0].imshow(np.squeeze(original.astype('float32')))\n",
        "    axes[0].set_title('Original')\n",
        "    axes[0].axis('off')\n",
        "\n",
        "    # Display the transformed (augmented) image\n",
        "    axes[1].imshow(np.squeeze(transformed.astype('float32')))\n",
        "    axes[1].set_title('Transformed')\n",
        "    axes[1].axis('off')\n",
        "    plt.show()\n",
        "\n",
        "# Example usage: Apply the strong augmentation to a sample image and display it\n",
        "sample_image = np.expand_dims(target_samples_x[4,::],0)\n",
        "sample_image_transformed = augment_strong(sample_image).numpy()\n",
        "display_original_and_transformed_images(original=sample_image, transformed=sample_image_transformed)"
      ],
      "metadata": {
        "id": "HppdeE_9C08f"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "Below, **please implement your most robust attack strategy, specifically designed to challenge the defense mechanisms of this system.**"
      ],
      "metadata": {
        "id": "3ThzUA1uMeti"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "def your_best_attack_on_strong_augmentation_system(target_model, preprocess_input, img_batch, img_labels, epsilon, num_ittr, num_classes):\n",
        "\n",
        "    ## WRITE YOUR CODE BELOW\n",
        "    #....\n",
        "    #\n",
        "    #return perturbed_images.numpy()\n",
        "\n",
        "\n",
        "adv_imgs_for_strong_aug_defense = your_best_attack_on_strong_augmentation_system(target_model=baseline_classifier, preprocess_input=augment_strong, img_batch=target_samples_x, img_labels=target_samples_org_y, epsilon=4.0/255.0, num_ittr=10, num_classes=num_classes)\n",
        "\n",
        "attack_success_on_strong_aug_defense =  untargeted_attack_success_rate(model=baseline_classifier, preprocess_input=augment_strong, x=adv_imgs_for_strong_aug_defense, y=target_samples_org_y)\n",
        "print(f\"Attack success rate oon majority vote defense is {100*attack_success_on_strong_aug_defense:.2f}%.\")"
      ],
      "metadata": {
        "id": "t229lwlIDkcA"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "### **System 2: Majority Vothing System**\n",
        "\n",
        "This system incorporates three classifiers, each independently evaluating the test sample. The system's final output is based on the majority decision of these classifiers. When the classifiers fail to agree on a common label, the system is designed to select and return the largest label among those predicted by the individual models.\n"
      ],
      "metadata": {
        "id": "z8sdGGmxBQC3"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "class ClassifierWithMajorityVote:\n",
        "    def __init__(self, models):\n",
        "        self.models = models\n",
        "        self.num_models = len(models)\n",
        "\n",
        "    def __call__(self, x, *args, **kwargs):\n",
        "        # Compute predictions using each model\n",
        "        predictions = [tf.argmax(model(x, *args, **kwargs), axis=1) for model in self.models]\n",
        "\n",
        "        # Extract individual predictions\n",
        "        a, b, c = predictions\n",
        "\n",
        "        # Majority vote using TensorFlow operations\n",
        "        # If there's no consensus, return the prediction with the largest label\n",
        "        max_ab = tf.maximum(a, b)\n",
        "        max_abc = tf.maximum(max_ab, c)\n",
        "        majority_vote = tf.where(a == b, a, tf.where(a == c, a, tf.where(b == c, b, max_abc)))\n",
        "\n",
        "        # Convert to one-hot encoding\n",
        "        return tf.one_hot(majority_vote, depth=10)\n",
        "\n",
        "\n",
        "\n",
        "model_1 = load_cifar10_model(model_path=\"./AML/baseline_cifar10\")\n",
        "model_2 = load_cifar10_model(model_path=\"./AML/surrogate_cifar10_1\")\n",
        "model_3 = load_cifar10_model(model_path=\"./AML/surrogate_cifar10_2\")\n",
        "\n",
        "classifier_with_majority_vote = ClassifierWithMajorityVote([model_1,model_2,model_3])"
      ],
      "metadata": {
        "id": "3HJd5La2nB8W"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "Below, **please implement your most robust attack strategy, specifically designed to challenge the defense mechanisms of this system.**"
      ],
      "metadata": {
        "id": "qk3-YAj3M9E2"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "def your_best_attack_on_majority_vote_system(target_model, preprocess_input, img_batch, img_labels, epsilon, num_ittr, num_classes):\n",
        "\n",
        "    ## WRITE YOUR CODE BELOW\n",
        "    #....\n",
        "    #\n",
        "    #return perturbed_images.numpy()\n",
        "\n",
        "\n",
        "adv_imgs_for_maj_vote_defense = your_best_attack_on_majority_vote_system(target_model=classifier_with_majority_vote, preprocess_input=lambda x: x, img_batch=target_samples_x, img_labels=target_samples_org_y, epsilon=4.0/255.0, num_ittr=10, num_classes=num_classes)\n",
        "\n",
        "attack_success_on_majority_vote_defense =  untargeted_attack_success_rate(model=classifier_with_majority_vote, preprocess_input=lambda x: x, x=adv_imgs_for_maj_vote_defense, y=target_samples_org_y)\n",
        "print(f\"Attack success rate oon majority vote defense is {100*attack_success_on_majority_vote_defense:.2f}%.\")"
      ],
      "metadata": {
        "id": "XvmMsBVieg73"
      },
      "execution_count": null,
      "outputs": []
    }
  ]
}