SPritchard_CAP5625_Programming_Assignment_1_10182021.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "SPritchard_CAP5625_Programming_Assignment 1_10182021.ipynb",
      "private_outputs": true,
      "provenance": [],
      "collapsed_sections": [],
      "toc_visible": true,
      "authorship_tag": "ABX9TyP1U4fNk2j13wGueFq9Je9I",
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "language_info": {
      "name": "python"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/shaungt1/-open-source-datasets-for-data-science/blob/main/SPritchard_CAP5625_Programming_Assignment_1_10182021.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "qNcAsfXa9_P5"
      },
      "source": [
        "- **Programmer: Shaun Pritchard**\n",
        "- **Date: 10-10-2021**\n",
        "- **Assignment: 1**\n",
        "- **Prof: Michael DeGiorgio**\n",
        "<hr>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "q-sdUmTK9iFf"
      },
      "source": [
        "# **CAP 52625 COMPUTATIONAL FOUDNATIONS OF AI**\n",
        "\n",
        "## Ridge Regression using Gradient Descent - Assignment 1\n",
        "<hr>\n",
        "\n",
        "*Note: I decided to use symbols to make it easier to view to the implementation of code in contrast to the mathmatical thoeroms learned in class.*\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "tDpfC4jffBuq"
      },
      "source": [
        "## **Deliverables**\n",
        "\n",
        ">\n",
        "*   **Deliverable 1:** build graph of dataset N=9 features tuning parameter effect on inferred Ridge regression\n",
        "*   **Deliverable 2:** Illustrate the effect of the tuning parameter on the cross-validation error\n",
        "*   **Deliverable 3:** Indicate the value of 𝜆that generated the smallest CV(5)error\n",
        "*   **Deliverable 4:** retrain  the  modelof 𝑁=400 observations and provide the estimates of the 𝑝=9 *best-fit* model parameters.\n",
        "*   **Deliverable 4.2:** Alternate less code  way to retrain  the  modelof 𝑁=400 observations and provide the estimates of the 𝑝=9 *best-fit* model parameters*(just for fun).*"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "b9QGfUlEfU0j"
      },
      "source": [
        "## **Import Dataset**"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "rhLYo6EfejRx"
      },
      "source": [
        "#Math libs\n",
        "from math import sqrt\n",
        "from scipy import stats\n",
        "# Data Science libs\n",
        "import numpy as np\n",
        "import pandas as pd\n",
        "# Graphics libs\n",
        "import seaborn as sns\n",
        "import matplotlib.pyplot as plt\n",
        "%matplotlib inline\n"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "8vkdCK8gpDLC"
      },
      "source": [
        "# Mount Google Drive for data access\n",
        "from google.colab import drive\n",
        "drive.mount('/content/drive')\n"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "6TdmqyEMfPKw"
      },
      "source": [
        "# Set up dataframe instance of dataset Credit_N400_p9.csv\n",
        "df = pd.read_csv('/content/drive/MyDrive/Florida_Atlantic_University/Computational_Foundations_of_ AI/Credit_N400_p9.csv')\n",
        "# Set up datafeme for testing\n",
        "# df = pd.read_csv('/content/Credit_N400_p9.csv')"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "lnLh3EJCfPNu"
      },
      "source": [
        "# Check feature (row:col) shape of dataframe\n",
        "df.shape"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "GgUMKzHSfPQS"
      },
      "source": [
        "# Build copy of dataset for Pre-proccessing\n",
        "df1 = df.copy()\n",
        "# Validate new dataframe\n",
        "df1.head(3)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "DdC23kva_q1Y"
      },
      "source": [
        "## **Preproccess Data**"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "lc1Zu2hTfPSw"
      },
      "source": [
        "# Assign dummy variables to catigorical feature attributes\n",
        "df1 = df1.replace({'Male': 0, 'Female':1, 'No': 0, 'Yes': 1})"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "dOzlmzpofPVU"
      },
      "source": [
        "# Validate new trianing dataframe with dummy variables\n",
        "df1.head(3)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ohm-7lVmfPX9"
      },
      "source": [
        "# Separate independent n X 1 feature X  and convert to numpy array\n",
        "X = np.array(df1.iloc[:,:-1], dtype='float64')\n",
        "# Test print X feature data conversion results\n",
        "print('Matrix shape:{X}\\nValidate array:(row:col)'  .format(X = X.shape), '\\n', X)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Fa7oojbYfPaF"
      },
      "source": [
        "# Seperate dependant n X 1 feature Y and reshape to (400 x 1) vector numpy array\n",
        "Y = np.array(df1.iloc[:,-1], dtype='float64').reshape([-1,1])\n",
        "# Test print Y feature data conversion results\n",
        "print('Dependant Feature:{Y}\\n \\nValidate array:(row:col)\\n' .format(Y = Y.shape))\n",
        "for i in Y:\n",
        "    print(i, end = ' ')"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "dFPoGyzDCA68"
      },
      "source": [
        "## **Center response variables and standarize features**\n",
        "- Convert dataframe objects to numpy arrays\n",
        "- Center the response variable Y (subtracting the mean)\n",
        "- Standardizing input features X to a Z score\n",
        "<!-- - Dimension of Y is will be (num_samples,1) and dim of X is assumed to be (num_samples, num_features) -->"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "bjNpqVh02mB7"
      },
      "source": [
        "# Center Y response variable(subtract the mean)\n",
        "Y_p  = Y- np.mean(Y, axis=0)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "7lj5yDnBEcfD"
      },
      "source": [
        "# Validate Y response vairables - mean of y_\n",
        "y_ = np.mean(Y, axis=0)\n",
        "print('Mean of Y:', y_)\n",
        "print('Matrix Shape:', Y_p.shape)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "1x8XjsysWC1Q"
      },
      "source": [
        "# Split centered (row:col) of Y feature\n",
        "Y_row, Y_col = Y_p.shape\n",
        "# validate feature split of Y\n",
        "print('(Y_p) Row x Col:=',Y_row, Y_col)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "p4fB1gU79Sbi"
      },
      "source": [
        "# Standardized X feature n x 1 matrix  as X_p array and reshape\n",
        "mean_X = np.mean(X, axis=0).reshape([1,-1])\n",
        "# Center X\n",
        "X_p = X - mean_X  \n",
        "# Apply standard deviation to new shape[1,-1]\n",
        "std_X  = np.std(X_p, axis=0).reshape([1,-1])  \n",
        "# Caluate centered features divided by standard deviation\n",
        "X_p = X_p / std_X "
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "kMWAgC-B9UWT"
      },
      "source": [
        "# Validate X_p feature\n",
        "print('Matrix Shape:', X_p.shape, '\\n', '\\n', 'Mean of X:' , '\\n', mean_X, '\\n', '\\n', 'Standard deviation of X:', '\\n', std_X)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Atz4ktf3VJXM"
      },
      "source": [
        "#Store and seperate (row:col) in variable for X_p training/test set\n",
        "X_row , X_col = X.shape \n",
        "print('(X_p) Row x Col:=', X_row, X_col)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "E-U2-bGJVx4W"
      },
      "source": [
        "## **Assign local variables**"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "vCYxo2FfYRdS"
      },
      "source": [
        "# Local Variables\n",
        "\n",
        "# Tuning Parms\n",
        "λ  = np.array([1e-2, 1e-1, 1e0, 1e1, 1e2, 1e3, 1e4])\n",
        "\n",
        "# Learning Rate\n",
        "α = 1e-5 # learning & convergence rate\n",
        "\n",
        "# K-folds\n",
        "k = 5\n",
        "\n",
        "#Iterations\n",
        "q = 1000 # itterations\n",
        "\n",
        "#log base of lambda\n",
        "λ_log = np.log10(λ) \n",
        "\n",
        "#Standerrdized X features\n",
        "X_p = X_p\n",
        "\n",
        "#Centered y features\n",
        "Y_p = Y_p"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ljAcbQutG0ZV"
      },
      "source": [
        "## **Ridge Regression Batch Gradient Decent**\n",
        "\n",
        "- Implement(inferred) Batch Gradient Descent for Ridge Regression on standardizing feature X_p and centered feature Y_p numpy arrays. Where p is n x p matrix.\n",
        "- α = the learning and convergence rate.\n",
        "- λ = L2 regularization tuning parameters.\n",
        "- q = max number of iterations (1000) as specified.\n",
        "- 𝛽x  = Ridge regression beta coefficients parms.\n",
        "- MSE = is a storage list to contain the mean squared errors for each iteration of the Batch Gradient Descent algorithm.\n",
        "\n",
        "*Note: I decided to use symbols to make it easier to view to the implementation of code in contrast to the mathmatical thoeroms learned in class.*"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ea28TCdCVPzx"
      },
      "source": [
        "## **Ridge Regression Batch Gradient Decent Algorithm**\n",
        "![image.png](data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAQIAAAAaCAYAAABcvVISAAAGs0lEQVR4Ae1bW5LjKBDUGR3RF+rQVdxH8RxkPvoYbFAPVKDkIZBsz5qN2LCMoB5ZWQnSuBc3/5sITAQ+HoHl4xGYAEwEJgJuCsEkwRMR+HX3r8UtC/p/dY8nRjJdxQiUheDPGhXt9vMbr37Vt793d7Nk+v6HKVTLxdyv4v+u9Qo8ebh1kYanvMz1191l2SUYrH+CoX/wIhVBI3xUt5u7/8Vp/f7coj68AoesELBzE5yQ7IogcPqZ0TQOIUm1STLmjg2nxVzcMipCRZI/3KoN4vPWaxD0S+olsUc7vODx+La7viG9xE7xhnwebs3gGM8DiT9xKM7J5yf9keCw56IXQMsV4VHI3ychYxkcKM0iV8aAwEIAHXIy+yTHAji2GsXQAOAxJ3i2FtsUj5tvcUOYQKxBCDRv31A0E9pAWAG7g0OKAYmBwcY5Ib82S+SHa1bD7Z1EgMPXnFjk7KYYcACNTAISYeMcz0/rWcEF1jkCtvsLFAIY5IVBNEcPj1DPIbyj/M0JiYIWYiRFbs7HT2zG1ftKicOeXlsvEWJ5VAvNkZ7cIlB8LimW0QSnQhLspbdf9V3yioVPMIA8wPxE4kApQZ5Jss1cOQ4OEAJJKlE2PhZhIh5327cCgcdNUCNVn7/6qjoB+Nis8QFsW4tL8xD+wKZvIzqao/n1rA7PkBw4V+9TBDLhULBLzVSODYqb2jXvh/hUIRjoeM5vCGDsgrHlU4H3X+Qg2rwEL3wi4lzgvVaudKQHhICLuCmxglwunPpmkJT4OnrG5x6g63w1xpsraDruv/vdokCKDW/ku1SDsXohbz1j3AzmvQDcHdkyzS02q+SL5gi2JDr2vo4X/PbkhdeI0KnwLIvL1Y9ytTHJiQI2ujjbrdEgJMecL53W87kXAkNWLe4Rx7Y5dT3vFIYkBkC9V/VhQWgAsweMo2soV1tkMiAkjsY9cVa3foP3CTavTAABx8imTB6sV8Zlx7CKla9zYSOQ2vm655uBGw3fN34sHg04diSVX2LyyL8wtoIm4mFjzlkn22DjvTDHnRAw6WwQpaLkMrlg3BCerVuQy/5YnMpCxIJUIHDqIlcsIUgsbEIC9IxfK264b/4FwcTSXy/TUECYVaD50/LBOE8ug2B5ey2ET9ZvXyucE4yt4JDvks+wps4DLEBbdHylNWV7cb117j6Pfb10rvmkWAEXAxfM3JMud0JATWOPXN5RLrCTgmgxAwt9ITDFmAp44ELvCRHsF3MwYufnAaK/Tb2k0W5f2795tzVUQGK7KGLip21NyD74O27Gzex5V1KXr5v5PQsQS8ojbWhT01xAOX5VcckZrI8nQpAhbC4wYJ93X06em6KuwH7XKRcxA96FwIDUeAgWd5tN+ScNyzikhJA1hRwiUfHzErvaELuGO1CvLfKRK8sbqRWdMjI5V11Ze3gy80xOHpQvaES8dHg0qgv5Fo4nG2g0L3jNcDncL2y8Ba7Y5T3XsRBkHDHobUBbIegJCK/BxCg2GDY0OMpxxKLlC7thsxMCwXRBjwU+mgzmfAozAunnpUKQWXukXoOAbD+EsbGFnHsfEXC9o1htAxbfN0Srxr+IX8sBxnv/iLDjgvcu2OzE20aWE7ZMve3S3utYCCRJGyQ3W+nFTq/rA+uUWFZxlQh27IDJnqlU2NQfxbYJQbwLeEKv7mEK+/hOdklYXG6E6Dnb+Ak23qBe2gSWM+FXcvruIcWsCn7DrmkeD7IiW/VzcILyMBV15WJ0CtKT0cYNPcHV4iUOWWHVMCFX9ObYZyQEGsA9+nmo2ZXGfPWvlka6J7+5jsnXb75pZSABetRBxfbzZNystTsJ+QXF5eZKBMMQX228ul4seorHFm88vt8pW/BmGxbX/SoVofxb+/2a/hER5524JeP+PjUxj68/yd/FoAaPgmIBgdwGXImWDnwxQtCiwgOeBpZSwasADjh45dLu4r5vvc6BUxqp9IdGhN0mQOf4PckKbV4dsZVy6uZKPScjBA3A1+1dMKOgkBd4e7rJ7uK+a73OQ1BPPLm/SqT7hx87zouvZKkWO15b4Xo3V7A3O7oJQUmJ7IqnX//PCd9b3Let17kESRvKnw7p0ejN86dT7CGRajjh9XKloSRBCAhwfaZtWPi0Kb1HrKcFOOhIiht+wNNInret1yAccLlpgPBeoPTrRWjkmYO8ecHnfBRGheNca30Xc807uyAEKL45NhGYCHwGAlMIPqPOM8uJQBGBKQRFeObNicBnIDCF4DPqPLOcCBQRmEJQhGfenAh8BgJTCD6jzjPLiUARgf8AU9aXoK7NgCAAAAAASUVORK5CYII=)"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "FFPiGtG5ma95"
      },
      "source": [
        "def RidgeRegression_BGD(X, Y, α, λ ) : \n",
        "    # Empty list to hold MSE errors\n",
        "    MSE = []\n",
        "    # Randomly initialize the parameter vector 𝛽 = [𝛽1, 𝛽2, … , 𝛽𝑝]\n",
        "    𝛽x = np.random.uniform(-1,1,(X_col,1)) \n",
        "    # Instantiate loop to update parameter vectors\n",
        "    for i in range(q) : \n",
        "        # Store 𝛽x coefficients in temp variable\n",
        "        𝛽x_temp = 𝛽x\n",
        "        # Update 𝛽x parameter vector as 𝛽x ≔ 𝛽 − 2𝛼[𝜆𝛽 − 𝐗𝑇(𝐲 − 𝐗𝛽)]\n",
        "        𝛽x = 𝛽x - 2*α*( λ *𝛽x -  np.dot( X.T, Y - np.dot(X,𝛽x) ) )\n",
        "        # Calcualte vector direction of response variables\n",
        "        ŷ = np.dot(X,𝛽x)\n",
        "        # Instantiate temp var \"MSE_temp\"square-root of real Y variables\n",
        "        # minus the calculated ŷ response\n",
        "        MSE_temp = np.mean(np.square(Y - ŷ))\n",
        "        # Append updated MSE_temp caluation to MSE list\n",
        "        MSE.append(MSE_temp)\n",
        "        # Caluate the divided absolute values of 𝛽x coefficients minus 𝛽x\n",
        "        𝛽t = np.abs((𝛽x - 𝛽x_temp)/𝛽x_temp)\n",
        "        # Calualte the max value of 𝛽t and store in 𝛽m\n",
        "        𝛽m = np.max(𝛽t)\n",
        "# Console log to test my code:\n",
        "#-------------- Feature Testing Output----------------------------------- \n",
        "        if (𝛽m < α):\n",
        "            print(\"Testing:\\nBatch Gradient Descent(RR_BGD) breaks on: {i} iteration\".format(i=i))\n",
        "            break      \n",
        "        # Test for convergance error \n",
        "    if (MSE[-1] < MSE[0]): #Check MSE is lower than the initial value\n",
        "        pass \n",
        "    else :\n",
        "        print(\"Testing:\\n Error not converging with lambda = {λ}param\".format(λ=λ))\n",
        "    # Output updated coefficients and MSE\n",
        "    return 𝛽x, MSE"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "tGovZyqpk5Oa"
      },
      "source": [
        "# Test tunning paremters inferred on RidgeRegression_BGD()\n",
        "# Create emtpy list to store updated coefficients\n",
        "𝛽_lst = []\n",
        "# Create counter for test ouput\n",
        "count = 0\n",
        "# Itterate through RidgeRegression_BGD oupt: 𝛽x, MSE    \n",
        "for i, l in enumerate(λ):\n",
        "    # counter\n",
        "    count += 1 \n",
        "    # print('Tuning parameters {} \\n', lmbdas)\n",
        "    print('Tuning parameter converged at = #{c}λ {} \\n'.format(l, c=count))\n",
        "    # run RidgeRegression_BGD\n",
        "    𝛽x, MSE = RidgeRegression_BGD(X_p, Y_p, α, l)\n",
        "    # Append 𝛽x beta coefficients to empty list\n",
        "    𝛽_lst.append(𝛽x)\n"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "BaYnEF04htPu"
      },
      "source": [
        "## **Deliverable 1:**\n",
        "> Build graph of dataset N=9 features tuning parameter effect on inferred Ridge regression"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "yoqvIzr8sQIx"
      },
      "source": [
        "# Output Deviverable 1: inferred tuning parmeters of ridge regresion\n",
        "sns.set_theme()\n",
        "sns.set_style(\"darkgrid\", {\"grid.color\": \".5\",\"image.cmap\": \"mako\", \"grid.linestyle\": \":\" })\n",
        "plt.figure()\n",
        "plt.figure(figsize=(16, 10), dpi=70)\n",
        "plt.xlabel('λ Tuning Params')\n",
        "plt.ylabel('p=9 features')\n",
        "plt.title('Deliverable 1: Inferred Ridge Regression Coefficients')\n",
        "for i in range(X_col) :\n",
        "    𝛽j = [ 𝛽x[i,0] for 𝛽x in 𝛽_lst  ]\n",
        "    legend = 'Beta_λ_{}'.format(i)\n",
        "    sns.lineplot(x=λ_log,  y=𝛽j , label=legend, )\n",
        "# Output Deliverable1.jpg to file\n",
        "plt.savefig('SPritchard_CAP5625_Assignment1_Deliverable1.jpg')\n",
        "plt.show()\n",
        "\n"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "AA5tsXYgJwN6"
      },
      "source": [
        "## **(5)K-Fold Grid Search Cross Validation Algorithm**\n",
        "- (5)K-fold Grid Cross Validation calualted Batch Gradient Decent Ridge Regression with hyperparameter tuning\n",
        "- Use grid search CV to trian and test 5 k folds\n",
        "- Calculate test and trainging errors\n",
        "- Test MSE on tuning params"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "v7kAwhVlQ2hE"
      },
      "source": [
        "# Implement start and end of test k-fold data split\n",
        "X_row = X.shape[0]\n",
        "# Divide absolute row features by k = 5\n",
        "X_row_test = X_row // k\n",
        "# Store data in k-fold array (Kfold/Kfold_ )\n",
        "Kfold = [ X_row_test * ind for ind in range(k)] # initial k-folds \n",
        "Kfold_ = [ ind + X_row_test for ind in Kfold ]  # End k-folds "
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "YiV5XTicwhke"
      },
      "source": [
        "# Grid Cross Validation for Batch Gradient Decent Ridge Regression with hyperparameter tuning\n",
        "# Instantiate empty list to hold Cross vlaidations errors\n",
        "CV = [] \n",
        "# Add a counter to iterate tuning params and errors\n",
        "for i, l in enumerate(λ) :\n",
        "    # print('(5)K-fold CV tuning parameter error = {}'.format(l))\n",
        "    MSE = [] \n",
        "    # Loop through K trianing and test vectors\n",
        "    for i in range(k):\n",
        "        #Hold-out 5 k-folds arrays (80 x 9)\n",
        "        CV_fold = Kfold[i]\n",
        "        CV_fold_ = Kfold_[i]\n",
        "        # Seperate training feature variables\n",
        "        X_train = np.row_stack(( X[0:CV_fold,:] , X[CV_fold_:, :] ))\n",
        "        Y_train = np.row_stack(( Y[0:CV_fold,:] , Y[CV_fold_:, :] ))\n",
        "        # Seperate testing feature variables\n",
        "        X_test   = X[CV_fold:CV_fold_, :]\n",
        "        Y_test   = Y[CV_fold:CV_fold_, :]\n",
        "        # Standardize X test set\n",
        "        X_test_ = (X_test - np.mean(X_test, axis=0))/np.std(X_test, axis=0) \n",
        "        # Center Y test set\n",
        "        Y_test_ =  Y_test - np.mean(Y_test, axis=0)\n",
        "        # Implement ridge regressionand MSE on test data\n",
        "        𝛽x, _ = RidgeRegression_BGD(X_p, Y_p, α, l)\n",
        "        # Product transofrmation of test data on trining set\n",
        "        ŷ = np.dot(X_test_, 𝛽x) \n",
        "        # Claulate average squareroot of Y(test)- ŷ variables\n",
        "        err = np.mean(np.square(Y_test_ - ŷ))\n",
        "        # Append calualtion to MSE list\n",
        "        MSE.append(err)\n",
        "    #Caluate average of updated MSE    \n",
        "    CV_err = np.mean(MSE)\n",
        "    # Append averaged MSE variables to CV list\n",
        "    CV.append(CV_err)  "
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "d7v8DaKkYXxn"
      },
      "source": [
        "# console log  to test CV code\n",
        "#-------------- Feature Testing -----------------------------------\n",
        "print('Initial (5)K-folds', Kfold, '\\n')\n",
        "print('Ending  (5)K-folds', Kfold_, '\\n')\n",
        "print(X_test.shape, Y_test.shape, X_train.shape, Y_train.shape, '\\n')\n",
        "print(X_test[0,0], Y_test[0,0], X_train[0,0], Y_train[0,0], '\\n')  \n",
        "print(\"Mean Square Error\",err, '\\n')\n",
        "print(\"MSE value\",MSE, '\\n')\n",
        "print(\"CV_error\", CV_err, '\\n')\n"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "tyQc9CX0h2Bp"
      },
      "source": [
        "## **Deliverable 2**\n",
        "- Illustrate the effect of the tuning parameter on the cross validation error by generating a plot.\n",
        "- Grid Cross Validation tuning errors for each tuning parameter value, perform five-fold cross validation and choose the value of 𝜆 that gives the smallest value.\n",
        "\n",
        "![image.png](data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAQgAAABnCAYAAAD8I9zEAAAgAElEQVR4Ae2dB3hUxdfGyaZ3AoFUkgABDL13ASlSFaTYQEQ+qoA06R0bKiJYQBFUCEj7UxUpgpRAICA1kFADpPdett7f92waCdmEZLNLsnDzPPPs5u6dmTPvmfvemTNnzlRB/BMREBEQESgGgSrFXBcviwiICIgIIBKE2AlEBEQEikVAJIhioRF/EBEQERAJQuwDIgIiAsUiIBJEsdCIP4gIiAiIBCH2AREBEYFiERAJolhoxB80IZAadZ+gGzcIDAzMT9cvHmPN3Jms+DOINLmmXOI1Q0VAJAhD1VyFyC3nwIwO1LS1wMzUFNMCydjMk/HbLpGqqBDBxEr1hIBIEHoC9vksVs6+qW3x9HqJLn1fY9CgQY/T4IlsvPCALNXz2fIXtVUiQbyomteq3XL2TetI54EzOHwrkoSkJJLyUwpZCiWCVuWKmSorAiJBVFbNVEq51ATRiS5D53ImMgOVyAaVUku6FEokCF2i+dyXlTuCGDSDw8HhxMbGEhMbR1JKOnKlOLd4HtUvEsTzqFW9tUnOwTldeKlhawYMHcprfbrTqVMXBo2cyo+7/YhJk4pTDL1hXzEFiwRRMbgbaK0C4f8dYMO6n9i8bQe7dm1n/VezebWpOzXrdGTBrquky8V5h4EqV6PYIkFohEW8WBwCgqBCqVCiEgQEQUCZlcDxjdNp4miDU78viEzMLC6reN0AERAJwgCVVnEi55BC4TGCivunN9HXxw7zJhMIj0mtOPHEmnWOgEgQOof0+S1QFXONfTt3ceFuIor8JQwFwcfW0qOuPTX7fi6OIJ4z9YsE8ZwpVJ/NkfmvoJWnO/2m/ERgVBqCALLke2yYMxCvWk2Z/Ns5UmXiaoY+dfCsyxYJ4lkjbsD1KYK3Mqxtbao5ONGwdVdeGziArq18qNe4E1PX7OdRUpa4imHA+tUkukgQmlAxsGuCSkFmerr+PRlVKuRZ6UTcucKJQ/vZvm0HB/45R0h0KjLRD8LAek3pxBUJonQ4Vdq75OnRXDy0mUXTFrH9ehjSwhZE/cgtCKhUqtwkiKMG/aBcKUoVCaJSqEELIQQp4ZcPsuKjN2nt7YSVbSs+ORVM5rMgCC3EFbMYJgIiQRig3qSJD9j99SQ6NHDF2swYI6MqGFm35gu/WyJBGKA+K7PIIkFUZu1okC3myk4mvtaWl5p3Y9yilSwf0xlHSxMktm34yu82WeIIQgNq4iVtERAJQlvkKiSfiqDdy5m5cDXHroWSKVMS7vcJrapbYWLbViSICtHJ812pSBAGpl+lLJXUdFm2q7Na9KiAFbQtM0EoSU+K5dGDEO7fv18BKYTQ6ERkCtFnorJ3P5EgKruGniKfVgShSuDId9No37AuXl5eFZDq0Hn019yIyHhK68SfKxoBkSAqWgPlrF8rghDSubhpLs2crTGqUoUquclIYoGTV318fBrSsKGOko8P3l7u1HSwwURilFuXhNoD5nPpUXo5Wy9m1zcCIkHoG2E9l68VQQCqlDv8Pr0vTlYm+SRhZOzI64t3cTs2DalMhkxHKS0hjHN71vBWey8sjNUkIRKEnruFzooXCUJnUFZMQdoSBAhkhvuzZFBDbM0kuW92I6zd2rJ420WSZEqdNkhQZRLy9zLaOVtjLBKETrHVZ2EiQegT3WdQtvYEoRZOSdS1HbzVvCZmucN/I4kpXp1Hsvu/yAI7NnXTEEEeweoh3tiaGYtTDN1AqvdSRILQO8T6raB8BAGCIo2zvnNo5WaLxCjHHmFk6kDXCWu4GZOpYzdqgeCNI/GwsxAJQr/dQmeliwShMygrpqDyEoRaaln8bTZ8PBAXW7P8qYZJVR/G/nCU2CyFTkki5sr3dHW1x1s0UlZMhyljrSJBlBGwyna7LghCPdVIuneMj/v6YGXyeKXB2qMrK/6+S4Zcd/4K6XEXWPnRKKZ88QchcVmVDU5RnicQEAniCUAM7V/dEAQgyEm4spH+datiKsld+jQyxr3t++wPSkRXHCGo5KQlJZCUmoEyPyqVoaH+4sgrEoSB6zrS/3PaqD0pbVrz5ely7sUQFARum0ZzZxuM8+wRZnZ0H/89gTHiQTkG3lW0El8kCK1gqyyZVNzaPpF6VS2QmNdnxr6rpJdzdVKWFsrvcwfiWdU81z/CCEuXJkxde5S4DPFk3sqi+Wclh0gQzwppHdWjUmQSeusq50//w55N3zDq5dpYmUioYmSJd68xrN60jxP+/3HjfmT2Zq4yVysoib9znNmDWmJrmusfYWRCjSYD+cXvAdJyElCZ5REzVCgCIkFUKPxlrzw97hpzX2+Kl7szjg42mOa7L1fByNgcO4cauLjWwuf1OVwNSSh7BeocKimhp9bSq34NjPNcsSUW1Oo+B/+odJTilnLtcDXAXCJBGJrSBHVcyEwyMzLIKCFlSmWoyrH4IKjS+O/nsdSvbpHvH1FFYsXLU//gXmIW5Sja0BB/oeXVKUEoMpIID7nDzcBrXLt2nRvBt3kQFk1Kplyna+kvtMaeYeNVWY/YMrMPtWxN8/drWFZvyPRfThKdrlv/iGfYrKJVCert7zGE3A7i+rVrXL9xk9v3HhAVn4r8BV9p0QlByFIjObPnJ2aNG06/nl3p2L4tbdu0pUOnl+nZ53XeHTOL349dJ1Uhw/+3RUyfMokPP/wwP02esYxdZ2+RWWDsKntwhpVLZuXfk3P/JJb6BpAqFY1lRXu5Pq4IpD48y4IhLbA3M85xojIyxa3VUH49eYcsvT88mfj9uoQZHz3uLx/NXMje/6KQFegrRVuuIDRgH5/Mnfq4/0yaw69+98lUFJ4fSRPucuCXz5k4YjA9u3aiXbs2tOvQkS7dejJg8DtMnv8tZx6kkRO6QsGFrZ8yc+pjeQr2Y03fZy3fzM3kTAzVdFNugpAm32XttNdo4FId52YDWLJ+L+cuX+XcPztY8GYbqpobIzF14c3PdxGVpSDUfwtje9bH1swEiUSCpYM3E9Yc5F5cGgX7myotHL9dK+nnUw0LE2MkxlZ4vzqZ7QHhT+kcRbuLeEV7BNR+C/dPbmRQC+f87dpGJjY0fWs5l8NT9TwyVBJ7+1+WvtcBRytTjCUSTKwcGbJ4N5EZxY9KhaxQfOcNwdXaNLuPGVu70H3Uck7fi8990HPwUMUFsHRYOzzda/PKu7PZetif69cvc8j3Swa3r4uliQRbl9asORVNVvY7SSDh/hlWjOmGs7UZxhITvNq/wfujxzJu3LjsNHbsaIb17YynozXGxhJq+IzmcEQKhvpKKxdBqDLD2TS5MzVsbKjdfRJHgqOzz0dQn7gkCEoUmff5cUQz7C2cGfLJdiIzlaCSEfzPWnrVs8+e21av/z6Ho9J4gtjVnjuoVHLOf9GXalbmVG0+ggOBES/8kE/7R137nII8geM/TsGnplX+VENi7sIbX/xFlL6nGkIWlw98QQdXG8zNTDCSmOHz6hSO3U8p9EJ53DqBpKAjjOvujYWpBKMqRji1GMQW/yc2n8nD2TqlOzVsbGn39nLOh6Xkb05T97uI0+vp3bAmdk7NWHUijyDU3VLGjcPf8Upt9d4VE/p8cpro5Czkcnl+kqVHc3T1ROo7WuDYYBSHwl9EghDSuLJxPPXsLbHz6syKPVdIL/KUC8Rc/oX+DZsw8qvdRGXlmLZkkRdZNKw1VsZGOHi9we6QVA0EoVa5gpOLu1KtqjtDl24nIs1Qefhx9zXUb8rEIH6e3IualnnxI4wwdWzNwp1XSJQq9TiSUHBLffantzNturfHxcYS29qdWbk/sNCUNB9XZSYB2xbS2tuHVi85YmlmQu2XR/HnzeRCqy9pV7fyWhMXLGydGf/jWZKzCk8CBHkk26b1psFLnfnhdEyB5d2cw4r7+9hjbGRC38/9ScgonFctiyziFDP6NqN+qwn8E/kCEoQ89BjjOtfGzMSSVkOXcCEyQ2MnkWfGEXDkIGdvPnqsUEUc+z8djruNCZbVffj04EOkRchFvRv5Pp/19sDZpwc/HLmNtOAcJL9HiF+eDQIC6Y9OMa9ffazz/COqSHBuMZTfz4aSVaJNoDwSKrh9/Cd61XOi97wvGd7cGSsrZwYv/oOw1KIvDFlqGN+P7UiDbqOZPbgpDpYm1Ok6moPBBUccKu7t/4xWHjaY2bky7berpBU5U1RFQvAZ/jxwhFsxWQXIRUXIGV8GNCqZIBDSueN/mAMHzxOZKTfYVR8tpxgKgnYuorGzJcZWjoz48ihJJXjQCIL62PiCnURJ+N9f0sLDFmPL6rz1yV/EazA8ygPX0dm1Oi0HzsU/Ir2YIWXBcsXvekVAUBJxaQuDG1fP36+htke0e2cp50ML25B0J0ceQVSj26K/2T3vVWramOPafRrn7icVeSnF39rKQJ8GDFn4I6v/rz3VLDQTxN29n9KyljUScwcGLdhJRIasSFnqTpvddws1Rk0QW55OEOo8ufkLZTewf7QjCGUUOxYNw9nSGItqdVh64BFZmkYAJYChjDrG6DaemEosaP3mci7FPXHwq5DF2c9fo6ajM+99eYSEJ4aAJRT9DH9SIU1PJSUlReuUlqmOUP0MRS5nVYI8lVMbZ2Tv18iLH2FiV5fRX+8jKl1eztI1Zc8jCAc6zjlEkP86ennaY+bQms//CSKzkEOGnDMrXse7SQ++3X+aTRM64GCuiSAg8/p2BrZwxdjIGKemA/jpyE3SShVFq2SCkKaFsXvtav7nd4O05yBqt1YEoUq+wVejXsY228rbkV/+02K3nzKG7RM7YWdmjHOrYey8GFNgGAeqxABm9axPDa9ubAiIpVS609S/9HlNGca2pZP4YOQIRozQLk1bfZQ4A/MTyYoJYt20/jjZmOYHobXz7sU3B28WWUYsP/yPCaL9x38SGnqbz99sjI25De1m7CQy7TEpqRLPMrmjF+3emINfSAibx3WgajEEgTyKvz57F++q5hgbW+Deoj+Lf97HtYcJyEpk7JIIQslDv2/p4dOc8WsOEFMpO23ZNKIVQSgjApj7RjPMJUZUrdWfHXdTijEyliSMisg9U6ljb46ZU0uWbb9Aev48VkXEsW/p4OWAz7DvuJUkLTr8K6noZ/Wb4g5fv/MyzRv54OOjXerx0RbC0zQMb59VG7SpR1CScOsw03rWxzI7CG0VqhiZ4zH4G+4nPTES1Kb8QnnyCKIqrafvIyoxiWPrxlO/qgWW9Udw8EES8uwRmIL7O6bS2MOT//vmGLHp4fw+pj32xRFEdqCcIDYvGU5jZ2tMjI2xcPCgQ//3+dL3KA8SMp+YFucJVZAgjKnXawxzFy5h2bKlLFkwg2Ev18POrg4frNr/4hKE4uEZpvX3yd4H4OAxmL2PiluFyANV86eQeJC31PEHTGswaMk2IvKswcpY9n0yHDc7dz7ceY+0nB6guZAKvSojMTKUhw8e8EDLFBaTmr+8VqFNKWvlgozYgLX09LLDJHdruFm7WdyIS9exQe4xQbSc8j+iUmWEnvuDwc1rYGxak7Gbg0mRqiArmG+GtcLdewBbrsciU4Sz8YN22JtpnmLkNFdAlhrN+b3fMqRtbSxMjDCSmGBT3ZNuI5fhF5JQaFSbk6cgQUhwatSZ3n37M2DAAPp2b4u7nTnGFp6MEgmiYTZBVPUYVMIy5VN6nZDCbyMbYGtmSt3XF3EtPOecBFnYeea+0ZyqDcfwb2SaBiXllKuUZZAYF01covqe0k/k5VlpxMXEkJQuNYj5/549e6hRowbW1tZlTvXq1SM+Pv4pitDyZ1UGx5a+Sg21Q5J5Tfos/TN7S3jpNVGaeh8TRLNJO4hKVaCMu8YX73fG1tQY78GrCE5IJ/bsT3Rv4EyrcZt5lCJFUIXxy/ttsCuRIHLqF5QKMqKD2PnlONrWdcLKzBiJiRkuHcaz/2bsE6tnBQnChN7LTxKZmIFUKkWalclV3w9p6PYSo0siCJWU+PD7BF4NJDRBVmz/Lg06+r5HqymGKiKAeYObZ08x7Nx6sDkwWcuIQwK3to6ljq0Z5rUHsONKGApBwb0Tv9DHpxbdFhwgTu1c9eSfoCDx4RV2freYieMn8M22syTkzvfk6SkkJsRnPxTqByM7xYRy1d+PSw+TkKtUxASf4us5E/lw9lfsPXObZJmqck5hctu9a9eubGLIO+CmLJ/u7u56IgglCYE7ea+VMxYW9rQYtgj/h4l6INwCBPHh9myCQJXC6bVTqF/dHDv3bvx8+io7lr+Hp2srPv3nERlqg3kZCCIHZgGlQkbY5QMsGPEK7up4GMZmNB78GRciCu5gLUwQT/pBpMUEsHzsaL7c4UdiMUZKVfx1vhrzCo72Piw4HEoBM8qTPb3C/9eKIIQCRkrrmk1ZdTJOayNiwp3dDKpvh6m5JxO2XiEtI5lDq0ZSx/sVvj91jyLL06iIv3uaT8YMZsDYZezxu0l8vtutkoANH/Pe20MZMmRIfho8sA+tG7Xnox3XUPtaCUopsfcvs2vlh/TuOYSVuy+SWIkNSurzMzds2MBPP/1U5uTr60tWlu5jP2aFn2P5W22xMzfDrcVgfvn3NhnFPBDl6+UaCAKB1Ot/MKiJC+bWNRk8YSEjezakbv/PuBabkTPFKTNB5NKESkHCvZMsersD9mYSrBwb8+mBu2Tmx9wrmSBUShkxoQ+Jyh7Vam55ZthlVk1/m659p3FI/dLS7ZBLc6VaXtWKIFBEsm3+EJwsjDG1c2f65kCtA5tKkx6wYpgPlqbmNBjjy6Ow2ywb9BKN3ljE5bCiR7Mp0qP4bWZ/OgyZxfHg2Cfm73L2TmmBvYUxRkZGhZNJXaYduP444pIgoMiM5+iXQ2nXfQTbz4U+UVYpEFXc47vRvems3pzWVrv0+pwdRDzFSJnjR5K7Jp+3Nl+Gz1K0pEy3CJnh/O+T9/C0N8fc3ospP/sRm6Gv3Z2aCEJtYXzEz++3xd7cFPtqTlSrXo9xm86RLMt92rQkiGwgVFKubJlPY2crjK1rMvqbf0nL2YyhDpZRej+IYlBVKaQkxUUTEZVAlrLQOm0xOSrusnYEgZzrW+fiU9MCI7Oq9J+1lfD8t3jhxsgzYvl3x2YOXbitwRVb7dqezMGVw3G3MsW0/mh2HVvFK3V9+GDlPmKK+D4IhJ3+ki5NOjB/kx8pRd5YcvZMaUu7vhPZcfYawbducSsv3Q4hPkNWxICWGnmeyd2bMHDurzxMfrxkVrgVxfynCGZ5v8Z4ubng4qJdaj16I6HqOXMxVVS2y4Iqi2t/rqR7PQdMTK3pOOE37ibpMz5EMQSBioi9M3PjVUiwbzGSQzeiHuu3JIIQZDy6fJzfNu7lZoKmnZYCcSe+p6N3NYytajBq5bGyE4SQxr9fjqBbjzHsv5s3SpASdvMCp08c5/jxUwSGJxfaPFbZdK2WR0uCAGnIIUa198BUYkrdbuM4GBSvwdgiEHFevbGlHu98vjNns9aTKAhygo6soZuXLRLTuvR/ox0ezfqx/vi9ohuzVHH8OqoFTV4Zxf5ATfWpCaIdLw+dj39UTtTkQm/fJ+vODp6UwO5Ph9KgzXD234gs47ZcKTEPbhMUdJObN7VLd8PUPiQGQw/Zy5uTe9XH2tyaOq9+zL8h2tqfNChD4yX1Xox19PS2p+nEbTk2iNz7VEl+fNjGBUuL6vSe+QshSbLHJahCWf9eG+xMTajTZTQHgwq4WgspHF8zkfpODZi+9zapeaOO/NxKHh38kjaedphX9WLO1kAy8ue6KkL8NtO/Ycmu1orYM8zo8RKudYaxJ28pVkji9KZP6dfUGRv7Rszcc4XUMjoY5ov4jL5oTRBqQ9GF9WOpqx5mOtTl/RV7iH7C2iJk3mfdB61xsHR5vJuzSMMEUm8fY3yPuphKTLCwsqPV0MUEaNjboYrax5AGbrzywTfc1OgbkUcQCzgXVcpToQQpV/Z/TvvajZiy86rGUU4RkQteKMNQvxBZ5ecrWFjl/i6kBfHd/3XCycocx6YDWX/yrh73YORiIcgIPLSKLh521Bu5gfCCozwhi3O/L2HytEXs8CscTwTFHVYOaY6NiYRaHd5l79XExy8wIYVj346jTlULGg7+gv8iCxohQciKYs8nw/GwM8Ox6XD2BSUUMMIruHX8J3p622Xv5nzSSJkttTKNy38sopWbHTUajOJwRHK+n1BmfAjLBtXDrnZvfj8fQgFKq5TK154g1HupMiLYuaAvbtbmWNdsyDuLfuVKRCpKpZz4+/6smdQbNzv1nvyaJRAECBn3WDvpVRzMJNlzvg++Palxb0f6Pwuo5+LCkMV7icufExbENWeK0bzzIOZ8spw50yczaepsVvse4l5sejGOLwKh/tsZ2MSJTvP+JiGjcs8JC7ZW83e1NV6OQqlh9UdzhtJdVSVybMXbeFU1w7xGI6auU3uAFt0sVbrCSn+XIj2SrYsGU8vGDIs6A9l0IbwQKSmy0khOSSVLXqC9goKYi5sY2MQFkypVsHRtyfzfz5Cct18onyBMMbVxoe+ErzgZFJkdZ0Sa9JAD38+kk7cjZnY+TN9yqZCbvyIjmp2fvkNtO3XUb2MaD/qYr779jh9++CEnfb+aZTM/oJOPC+bGRtRsNYVTUan5I9O0mIt82MmVWt0m4XcnvtJPLctFEGo1y9Nj+eeXefRu3QBXF2dcXN3x8PTE3dWFmjVrUNOlFj4t+7Fs2xkS84dpT3QQIYsLv86imUc1XOv3ZfNVTa7bKu5tGImrkyuj15wmVeOqg5Iz375Dq8ZNaNelO6/26EKz+h44Vq2OT5+pHA0u7M6dI4VA/LW/eK9jLbze/ZnY1DLaIZ5oir7/VWYlERUWSmio5nTr4gEmvNab+b+fICnf8l4+qdR2hzt/fUXnOg6YmNnzyphVXI1+FudkpLFrTj9a+tTF09MTD8/aNG7Xh3X/hmve/ZvdTCWhF3czvl8rvGt74eHhgaenF94Nu7Job2CO052QRdDhnxnZuxW1a7ng7OSEi6sbnur7a7nj5u5Jky5v8s2Bq9nbwB9PAOX8tXQgrRt546WWx8MDr9p18fb2LpDqUrdO7fzfWw9ZwdX4jHyCiL+1kd6eTrQfvYqgGGn5FPMMcpebINQyqhQykqNDCPj3IDv+8GXz5i1s27Gbvw6f4FJwKIlpWchLtNYKZIRdZ/8OX7btP0+MxvMglVxbMxQXJzcmrDtf7MYapSydxIREMrKysp1Xkh5d5OvxPXC2rUaHmTtJSC/61ku+cZgPunjhNPAbYpIrt9Ii/llMW3dH7GxssNGQrC3NMavRlHmb/HSzWUhQEnPjb6b0bYK1qSmencaw50o00ny3eH32UgGlXJa9TKteqs1JUhRPqVtQKZFJ8+7P+5Rm98G8h12lvkeWQezDYM6fOMzenX/g67uF7bv2cyIgkMiEtEL357VSJZchzZclr+ziP6UyRQHfEIE72yZQ36ku73y2kwhNPj55FVWST50QRE5bBASVCpVSiVKd1N9VquztsqVqq6COIKXMzqP5fiU3f3grmyDGfn+2WIIoklc9h/17NV08rbFqM5PohIwnbhFIDPyb9zt74Dr4O2JTKvesMOzIPBo7WGNpY4djjRrZHpZqL8uCyalRT1btu6KTh1iWEsLP0/pmuxBb1OrOysO3SC1uJPgEsgbxr9oWlNtXs/utUoVKfU0vwks5MKMDNT1bscj3LOlFVuH0Umm5CtUhQZRLjlJkFojZMRE3Zxfe/vwQSRriR6BSkCWVPuHPoOL+qd/p08AWi7YfE6WBICIv7mFYSycajf+D+EoetSqbIKq702fqKvyuB3Pnzp2i6X4oCWm62DSl4NLGsTRyssbYui5jfzpJbKY+o0eVohsY8i2Kuyzr5Yl70z5s+PfhE/20cjbMgAgC5NfW0M7Nmb4fbeRBetEdkLJzX9H+pcbM2HCB5FwjpqBIxc93Ns2dHen48W4Sihwfp+DWP2vp7u3GoG/9SZbq592hK/XnEIQnQ5dvI1yfQ1RBQXzAj/Sq44C5uQMvj1tNUHQpV4Z01djnrZzMY4zycaJex3fZeSGMR/572eMfWmAJtfI12KAIgqzLfNzJk9avz+R0aNEIU/L/1tCpjhPebQayfN1W/vzrT7b/8gVv92hNx0HT+etGdP5yU74qFGn8u34iPnW68e2pu6g3Blbmv2dDEEqS7hxl+qsvYW1mTp0uH7DrwqOnxEmozKhVEtlkl5jTxQsXr2a8+cEERrw1iq/2XidZo8G9cshsWASBjLOrh9GoRR/WHrn1xC47EJLucvR/m1n9+RLmzpnNrFmzmD13Mas3/I+AW5Eao15Jk0L4ekRbWg5ZyH9h+g7jXn6lPwuCkMbfYv3M13GxM8PauQ3LdlwkQeOycvnb80KVIGRx8/B65k+dzKxla9jnf5sUvQb8LT+6BkYQkPrIj8k9mjFg1kYeFPScy8Yix+Akl2aQmpJMYmIiKanpyBTKYnwgIPT4Z3Rp1pEFatdtAzC+6ZsgBEU6/r5zaO1mh0TiwMDPDhGWUnQ6V/6u92KWIChlpKemkJYpRWkAHrQGRxCCMpPLe1fwartXmLPZj+hU7WI6qhRZxAftZUyXxgz86EduxBrG/DqHIGoxYNYP+J0/xYFdvqxft5YNvns4c/0e0Unp5XDdVhETuIt3WjhhYWbFS299w424zALLdLp4qNVLl9Lss0XVzk2V2+Kji/YadhkGRxBquJUZMZzZ/jUjBg9j0tJ1HL8W9jik/lP1IZARH8IR368Z++YgRs//iYuh6qhOT81YKW6IOf89/ZrWo0Hj5rRt35727dvTqqkP7jUcsKlWi5eHTWfryWCStTCmSCMDWD64MfaWVtTq8D67Lofr3O4gyDO4dugXFsyaw7p/7pFRuf3SKoXOK1IIgySI7FO35OmEBp5ht+8v7D0ZVIY1ZYGk0ED2bN3E3uOXiUqVPvbRr0hNlLJueVokAcf/Yv9fRzjtH8ClS5c473ecXeuX8VozVyzNrU9h2wIAAAc4SURBVPDuMort5x6U6eEW0u6zZfZruNmYU9WrA0u2+JNUZDdtKYUs4TZFWgwbJ3Wmmo0zw74/S1IlXzUqoSkvxE8GShC5ulE7VynlyORli0UgKJXZedQOMYb4p2nTl1KezoODC2jmaImJWQ1em+9LqIaDZTS2V5XK+d/n0sLVFlPLagycu5X7ibrwoyhaW0ZcELN71cLCzJ0PNl0kRRxBFAWpEl0xbIKoREBWBlEE+V2W9HDDysQY5+7TuBCS9HSxBAXhF7YzvKMnZiZm1Ok1m1OPHu8+fHoBZbsj7uav9Pa0x8S0PjMPXCXNQKZ2ZWvl83O3SBDPjy4BKdsnNMXeXIJZ43c4djPqqUbAzOgrrBjVCUdLc6o1HMzGcxF6ONsiF2QhnRNfDMLNxgyJZXM+ORFEhmEO4p6rXlNSY0SCKAkdg/tNzp7JLalqIcG8yXD+DYoumSCU8fz56VvUr26JlWtb5v+hDtlWYNu0DtuvzIzj0r6v6NOgGmbGVZBU78z3Z+8imiB0CLIeihIJQg+g6qtIVUYEV86d5drtcDI17Y5VJfHjsNrYmElwe/Vj/nuQUoIoCu7vnkcrVxtMJEZYOnrRrmtP+vbrRz8dp759XuXlts3wcrLDNPegHYl7b3wvPkA0QZSgokrwk0gQlUAJpRVBEbye7l4utOg3jWP3Cx9nr17ZyQrfw1DvapibO/HGIs2nX2fXpT4Z69oO3mpZ6/EhvEZGSCQSPSV1AOEqucf05XyaNBjKwesRj2NIlhYE8b5nioBIEM8U7vJVpri5jk7OtlhWr8fIpb9z9ZE6FJra8SidiFtn+WF8VxxtrPDuNpqd5x4Wu8wpjb3Bt+N7UN3CuNBDW5bzNsp3rxFWLd/nZFBMyVOg8sEl5tYBAiJB6ADEZ1WEMvwIUwe0xa2qFbY169Cp35uM/XAyE8e8x4AuzXBx9qTzkGlsOXFTY8i+HDkF4q7+xfuda2FhaoKJSUUkU5y6TeHcvUSRIJ5V59GyHpEgtASuQrIpM4mLeMiN/06xc+MaFn78EePH/B/jJk5l8dc/c/DsVR5GJyPTZJ8oILAiK5Wwe8EEBgZWWAoOiSRDTwbRAk0Vv5YTAZEgyglghWQXBJRKBXKZLOdMSKkMuaJgaLMKkUqs9DlEQCSI51Cp2jVJQKGOtyiV8ZQBiHbFi7kMEgGRIAxSbboXWki5yW+fzGLS1MUcCU4xmM1rukdCLLEgAiJBFETjBf6edW0rfRq5UKP2G+y6k3dUnC4BUZAUEUmSTCkubeoSVj2XJRKEngE2jOIFkoOOs3LJbBau2sfDNKnOHmKVPIOouxdYP384Db368HNwPFmie7VhdIvynM1pMC0UBS0RAUFQkpmWQlJiAnFxcaRkynUTIEaRQcStAHZ8v5C3e7XGxd4SiXlnfr4TJ7pXl6iRyvWjOIKoXPp45tJI0x6wY9Vcxox8l7eHT2dXUJJOAvfKQk+xdOJoxs1awcYfVzC4nRdmll3ZcC9eJIhnrmXtKxQJQnvsnoucKmUmwUc3MrCFE2aOfdh6L7HQAxwf8AfLF85j9uzZT0nzWHdUHSEqZ/6gkiYScu8h0UmZSB+dY+7g5lhYdxMJwsB6jUgQBqYw3YsrEB6wi8EtHLHtPI/A2PRC9ocHWz6kqbcnLi4uJSdXD974+gyJmUUNDMrwAOYNEQlC97rTf4kiQegf48pdgyDj+t8r6ezhQIvJO4h+4mSxrOjbXDh3ljNnzpSczvoTqA40oyEAjEgQlbsLlCSdSBAlofMi/KZI5djaMdSt6sWoXwNIyZ0i5Dc9+8zUnHNW1WetlpTUofA0/YkEoQkVw7gmEoRh6ElvUirTYvh18ss4uHThO7+iAVwyw65y/Ohh/v7776ekw5y/k4BcQ7wZkSD0pj69FywShN4hrtwVZMQHZweRdWw7in+DYwrZH9SSP/AdT0MvVxwdHUtONVx4bYUfCaINonIrvIzSiQRRRsCet9uTHu5jSH1HGg9bzvWITKRJkcSlPD6MKOXmEX79eS0//PBDyenHtey5EK7xeENF6DlmD2qKudXL/HQ7TnSUMqBOJBKEASlLH6LG3/2VXm7VaTZ0IUeO/cWqWZPZ8M9dMvKsjYIKhUJRqlT4KDkl6YkxPLx7k+O+X9CniRPGJm68u/J/nL9xl7CYJGR5deijYWKZOkFAJAidwGi4haTHXGDaK57YO9TA66UOjP9qD7dj08t/mJAyHN9pb9CsQR3cajpgYSKhShUjrByc8ajjTau3FnP9YSnC8hsutM+F5CJBPBdq1L4RgkpBzP3L/L3vT/yDIkhTnzateTGijJWokGWmk5qSQoqGlJqeaRCH15ax0c/d7SJBPHcqLXuDBEGFUqnEUE8aK3uLxRylRUAkiNIiJd4nIvACIiASxAuodLHJIgKlRUAkiNIiJd4nIvACIiASxAuodLHJIgKlRUAkiNIiJd4nIvACIvD/5XQq9GkO9DMAAAAASUVORK5CYII=)"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "VOwEial_Jc84"
      },
      "source": [
        "# Illustrate the effect of the tuning parameter on the cross validation error by  generating a plot\n",
        "sns.set_theme()\n",
        "sns.set_style(\"darkgrid\", {\"grid.color\": \".5\",\"image.cmap\": \"mako\", \"grid.linestyle\": \":\" })\n",
        "plt.figure()\n",
        "plt.figure(figsize=(16, 10), dpi=70)\n",
        "plt.title('Deviverable 2: Effect of tuning parameter on the Cross Validation Error')\n",
        "plt.xlabel('λ Tuning Params')\n",
        "plt.ylabel('CV Error')\n",
        "sns.lineplot(x=λ_log, y=CV , color='purple', markersize=12)\n",
        "plt.savefig('SPritchard_CAP5625_Assignment1_Deliverable2.jpg')\n",
        "plt.show()"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "9St_DEmGiIpl"
      },
      "source": [
        "## **Deliverable 3**\n",
        "\n",
        "- Indicate the value of 𝜆 that generated the smallest CV(5) error"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "uFZ7Fit7hRqT"
      },
      "source": [
        "#Find minimum MSE error\n",
        "err = min(MSE)\n",
        "# index MSE error \n",
        "i = MSE.index(err)\n",
        "# Itereate to find MSe from λ list\n",
        "l = λ[i]\n",
        "# Output final results of lowest λ tuning param\n",
        "print(\"Best CV error of λ = {e}\\nBest tuning param of λ = {l}\".format(e=err, l=l))"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "NsvhzUEziVvI"
      },
      "source": [
        "## **Deliverable 4**\n",
        "- Given the optimal 𝜆, retrain your model on the entire dataset of 𝑁 = 400\n",
        "observations and provide the estimates of the 𝑝 = 9 best-fit model parameters."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "sUKzgawFhRsu"
      },
      "source": [
        "# Retrain model based on λ = 10.0\n",
        "𝛽x, _dh = RidgeRegression_BGD(X_p, Y_p, α, l)\n",
        "# Output best fit model params of  𝛽x based on on λ = 10.0 tuning param\n",
        "print('Best fit model parameters', '\\n', 𝛽x)"
      ],
      "execution_count": null,
      "outputs": []
    }
  ]
}