notebooks/tutorial4.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "50755a59",
   "metadata": {},
   "source": [
    "May 2023\n",
    "\n",
    "# Law 3027 Tutorial 4: Correlation, Covariance and Cross-Tabulations\n",
    "\n",
    "#### Intended Learning Outcomes:\n",
    "By the end of this notebook you will know how to:\n",
    "- compute the Pearson Correlation Coefficient\n",
    "- generate cross-tabulation/contingency tables\n",
    "- interpret the results of exploratory data analysis with respect to a target variable\n",
    "\n",
    "#### Libraries to be used:\n",
    "You can activate your previously used environment, though you will not use most packages from that environment. In this tutorial, we will use only the most commonly used python libraries such as: `pandas`, `numpy`, `matplotlib`, `scipy`, `seaborn` etc. "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a75901d2",
   "metadata": {},
   "source": [
    "### 1. Cross Tabulation\n",
    "\n",
    "A researcher collected the following hypothetical dataset from a sample taken in **Maastricht**, where **each observation (row) is a different neighbourhood**. In this task, you will create a cross-tabulation of the churches-pubs toy data.\n",
    "\n",
    "|Pub Type  | Church Type  |\n",
    "|:---:|:---:|\n",
    "|  Lounge | Catholic  |\n",
    "|  Beerhouse |  Protestant |\n",
    "|  Lounge |  Protestant |\n",
    "|  Beerhouse | Catholic  |\n",
    "|  Beerhouse |  Protestant |\n",
    "|  Lounge | Catholic  |\n",
    "|  Beerhouse |  Protestant |\n",
    "|  Lounge |  Catholic |\n",
    "|  Lounge |  Protestant |\n",
    "|  Beerhouse | Protestant  |\n",
    "|  Lounge |  Catholic |\n",
    "\n",
    "\n",
    "\n",
    "> Here are some examples of how to create tables in a notebook text cell: [creating-a-table.com](https://riptutorial.com/markdown/example/1741/creating-a-table)\n",
    "\n",
    "> You can always confirm the findings and create the table using code instead of text, here is a guide: [pandas-crosstab-function-in-python/](https://pandas.pydata.org/docs/reference/api/pandas.crosstab.html)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "490c3f1b",
   "metadata": {},
   "source": [
    "### **2. Pearson's Correlation Test**\n",
    "\n",
    "- Pearson’s correlation coefficient, $\\rho$, is a measure of the linear correlation (dependence) between two variables (e.g. X and Y)\n",
    "- Its value is between +1 and −1 inclusive, where 1 is a total positive correlation, 0 is no correlation, and −1 is a negative correlation\n",
    "- The hypothesis test explores whether if the value of the correlation coefficient $\\rho$ is significantly distant from 0 (the closer the value of the correlation coefficient to 0 the less correlated X and Y will be).\n",
    "- If the test concludes that the correlation coefficient is significantly distant from 0, then we are able to say that \"there is statistical evidence that X and Y are correlated\" (this is not a causal result, only association is proven)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "617d3d9e",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(0.8660254037844386, 0.011724811003954649)"
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from scipy import stats\n",
    "import numpy as np\n",
    "\n",
    "# Having X and Y variables\n",
    "X = np.array([0, 0, 0, 1, 1, 1, 1])\n",
    "Y = np.arange(7)\n",
    "\n",
    "# perform a pearson correlation test\n",
    "stats.pearsonr(X, Y)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d0907da8",
   "metadata": {},
   "source": [
    "As you noticed, the Pearson correlation coefficient function produces two outputs.\n",
    "- The first output corresponds to the $\\rho$ value\n",
    "- You would interpret that X and Y are **highly positive** correlated since 0.86 is close to 1\n",
    "- The second output refers to the p-value\n",
    "- By convention, if the p-value is less than 0.05 then the test is statistically significant\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "244ff73f",
   "metadata": {},
   "source": [
    "A good practice is to save the values in new Python objects, in this case since it's producing two outputs, therefore two objects can be assigned at the same time on the left side of the equals sign"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "6a98d147",
   "metadata": {},
   "outputs": [],
   "source": [
    "# perform a pearson correlation test\n",
    "rho, p_value = stats.pearsonr(X, Y)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "673f9231",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "correlation coefficient: 0.866\n",
      "p value: 0.0117\n"
     ]
    }
   ],
   "source": [
    "print(f'correlation coefficient: {round(rho,4)}')\n",
    "print(f'p value: {round(p_value,4)}')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8fafd743",
   "metadata": {},
   "source": [
    "It is possible then to verify the linear relationship by plotting it"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "914a9bb6",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.collections.PathCollection at 0x7ff521fc0860>"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAWoAAAD4CAYAAADFAawfAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuNCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8QVMy6AAAACXBIWXMAAAsTAAALEwEAmpwYAAAPBklEQVR4nO3db2xleVnA8e9jp6sXWLzEqUjvggNGq4YRu9aViG5gUbsgkTrZF4hCgiYToxJMTIXywn3hi8E0MWD8QybDikRkY6BUNEAlIetKYFc6dNnCrjXryp+5xUxXrMB6I53u44u2k5nZzt5zZ8+9/d3p95NMtj339J7nl06+c/f03J7ITCRJ5fqOgx5AkvTUDLUkFc5QS1LhDLUkFc5QS1LhjvTjSY8ePZrHjh3rx1NL0nXp7Nmzj2Xm2H6P9SXUx44dY3l5uR9PLUnXpYj48tUe89SHJBXOUEtS4Qy1JBXOUEtS4Qy1JBWu0lUfEdEEzgAvBhL49cz8TB/nkqShsbjSZn5pjfXNDuPNBrPTE8xMtmp7/qqX570L+Hhm3hERNwDPqG0CSRpiiytt5hZW6WxtA9De7DC3sApQW6y7nvqIiGcDtwLvAcjMb2fmZi1Hl6QhN7+0djHSezpb28wvrdV2jCrnqF8EbAB/GRErEXEmIp555U4RcTIiliNieWNjo7YBJalk65udnrZfiyqhPgLcDPxFZk4CjwNvu3KnzDydmVOZOTU2tu+7ICXpujPebPS0/VpUCfU54Fxm3r/7+QfZCbckHXqz0xM0Rkcu29YYHWF2eqK2Y3QNdWb+J/DViNg76iuBh2qbQJKG2Mxki1MnjtNqNgig1Wxw6sTxA7nq483A+3ev+HgUeFNtE0jSkJuZbNUa5itVCnVmPgBM9W0KSdJV+c5ESSqcoZakwhlqSSqcoZakwhlqSSqcoZakwhlqSSqcoZakwhlqSSqcoZakwhlqSSqcoZakwhlqSSqcoZakwhlqSSqcoZakwhlqSSqcoZakwhlqSSqcoZakwhlqSSqcoZakwhlqSSqcoZakwh2pslNEfAn4JrANXMjMqX4OJUnDZHGlzfzSGuubHcabDWanJ5iZbNX2/JVCvesVmflYbUeWpOvA4kqbuYVVOlvbALQ3O8wtrALUFmtPfUjS0zC/tHYx0ns6W9vML63VdoyqoU7gHyPibESc3G+HiDgZEcsRsbyxsVHbgJJUsvXNTk/br0XVUL8sM28GXgX8dkTceuUOmXk6M6cyc2psbKy2ASWpZOPNRk/br0WlUGfm+u5/zwMfBm6pbQJJGmKz0xM0Rkcu29YYHWF2eqK2Y3QNdUQ8MyJu3PsY+AXgC7VNIElDbGayxakTx2k1GwTQajY4deL4wK/6eC7w4YjY2/9vMvPjtU0gSUNuZrJVa5iv1DXUmfko8JK+TSBJekpenidJhTPUklQ4Qy1JhTPUklQ4Qy1JhTPUklQ4Qy1JhTPUklQ4Qy1JhTPUklQ4Qy1JhTPUklQ4Qy1JhTPUklQ4Qy1JhTPUklQ4Qy1JhTPUklQ4Qy1JhTPUklQ4Qy1JhTPUklQ4Qy1JhTPUklS4I1V3jIgRYBloZ+Zr+jeSJA2XxZU280trrG92GG82mJ2eYGayVdvzVw418BbgYeDZtR1dkobc4kqbuYVVOlvbALQ3O8wtrALUFutKpz4i4ibgF4EztRxVkq4T80trFyO9p7O1zfzSWm3HqHqO+p3A7wNPXG2HiDgZEcsRsbyxsVHHbJJUvPXNTk/br0XXUEfEa4DzmXn2qfbLzNOZOZWZU2NjY7UNKEklG282etp+Laq8on4Z8EsR8SXgbuC2iPjr2iaQpCE2Oz1BY3Tksm2N0RFmpydqO0bXUGfmXGbelJnHgNcBn8zMX6ttAkkaYjOTLU6dOE6r2SCAVrPBqRPHD+yqD0nSPmYmW7WG+Uo9hToz7wHu6cskkqR9+c5ESSqcoZakwhlqSSqcoZakwhlqSSqcoZakwhlqSSqcoZakwhlqSSqcoZakwhlqSSqcoZakwhlqSSqcoZakwhlqSSqcoZakwhlqSSqcoZakwhlqSSqcoZakwhlqSSqcoZakwhlqSSqcoZakwh3ptkNEfBdwL/Cdu/t/MDPv7PdgkjQsFlfazC+tsb7ZYbzZYHZ6gpnJVm3P3zXUwP8Bt2XmtyJiFPhURHwsM++rbQpJGlKLK23mFlbpbG0D0N7sMLewClBbrLue+sgd39r9dHT3T9ZydEkacvNLaxcjvaeztc380lptx6h0jjoiRiLiAeA88InMvH+ffU5GxHJELG9sbNQ2oCSVbH2z09P2a1Ep1Jm5nZk/DtwE3BIRL95nn9OZOZWZU2NjY7UNKEklG282etp+LXq66iMzN4F7gNtrm0CShtjs9ASN0ZHLtjVGR5idnqjtGF1DHRFjEdHc/bgB/Bzwr7VNIElDbGayxakTx2k1GwTQajY4deL4wK/6eB7wVxExwk7Y/zYz/6G2CSRpyM1MtmoN85W6hjozHwQm+zaBJOkp+c5ESSqcoZakwhlqSSqcoZakwhlqSSqcoZakwhlqSSqcoZakwhlqSSqcoZakwhlqSSqcoZakwhlqSSqcoZakwhlqSSqcoZakwhlqSSqcoZakwhlqSSqcoZakwhlqSSqcoZakwhlqSSqcoZakwh3ptkNEPB94H/B9wBPA6cx8V92DLK60mV9aY32zw3izwez0BDOTrboPI0lDp2uogQvA72Xm5yLiRuBsRHwiMx+qa4jFlTZzC6t0trYBaG92mFtYBTDWkg69rqc+MvNrmfm53Y+/CTwM1FrP+aW1i5He09naZn5prc7DSNJQ6ukcdUQcAyaB+/d57GRELEfE8sbGRk9DrG92etouSYdJ5VBHxLOADwG/m5nfuPLxzDydmVOZOTU2NtbTEOPNRk/bJekwqRTqiBhlJ9Lvz8yFuoeYnZ6gMTpy2bbG6Aiz0xN1H0qShk6Vqz4CeA/wcGb+cT+G2PuBoVd9SNKTVbnq42XAG4DViHhgd9vbM/OjdQ4yM9kyzJK0j66hzsxPATGAWSRJ+/CdiZJUOEMtSYUz1JJUOEMtSYUz1JJUOEMtSYUz1JJUOEMtSYUz1JJUOEMtSYUz1JJUOEMtSYUz1JJUOEMtSYUz1JJUOEMtSYUz1JJUOEMtSYUz1JJUOEMtSYUz1JJUOEMtSYUz1JJUOEMtSYU70m2HiLgLeA1wPjNf3K9BFlfazC+tsb7ZYbzZYHZ6gpnJVr8OJ0lDo8or6vcCt/dziMWVNnMLq7Q3OyTQ3uwwt7DK4kq7n4eVpKHQNdSZeS/w9X4OMb+0Rmdr+7Jtna1t5pfW+nlYSRoKtZ2jjoiTEbEcEcsbGxs9fe36Zqen7ZJ0mNQW6sw8nZlTmTk1NjbW09eONxs9bZekw6SIqz5mpydojI5ctq0xOsLs9MQBTSRJ5eh61ccg7F3d4VUfkvRkVS7P+wDwcuBoRJwD7szM99Q9yMxkyzBL0j66hjozf2UQg0iS9lfEOWpJ0tUZakkqnKGWpMIZakkqnKGWpMIZakkqnKGWpMIZakkqnKGWpMIZakkqnKGWpMIZakkqnKGWpMIZakkqnKGWpMIZakkqnKGWpMIZakkqnKGWpMIZakkqnKGWpMIZakkqnKGWpMIZakkq3JEqO0XE7cC7gBHgTGa+o+5BFlfazC+tsb7ZYbzZYHZ6gpnJVt2HkaSh0zXUETEC/Bnw88A54LMR8ZHMfKiuIRZX2swtrNLZ2gagvdlhbmEVwFhLOvSqnPq4BXgkMx/NzG8DdwOvrXOI+aW1i5He09naZn5prc7DSNJQqhLqFvDVSz4/t7vtMhFxMiKWI2J5Y2OjpyHWNzs9bZekw6RKqGOfbfmkDZmnM3MqM6fGxsZ6GmK82ehpuyQdJlVCfQ54/iWf3wSs1znE7PQEjdGRy7Y1RkeYnZ6o8zCSNJSqXPXxWeAHI+KFQBt4HfD6OofY+4GhV31I0pN1DXVmXoiI3wGW2Lk8767M/GLdg8xMtgyzJO2j0nXUmflR4KN9nkWStA/fmShJhTPUklQ4Qy1JhTPUklS4yHzSe1ee/pNGbABfvsYvPwo8VuM4w8A1X/8O23rBNffq+zNz33cL9iXUT0dELGfm1EHPMUiu+fp32NYLrrlOnvqQpMIZakkqXImhPn3QAxwA13z9O2zrBddcm+LOUUuSLlfiK2pJ0iUMtSQV7kBCHRG3R8RaRDwSEW/b5/GIiD/ZffzBiLj5IOasU4U1/+ruWh+MiE9HxEsOYs46dVvzJfv9ZERsR8Qdg5yvH6qsOSJeHhEPRMQXI+KfBj1j3Sr83f7uiPj7iPj87prfdBBz1iUi7oqI8xHxhas8Xn+/MnOgf9j5Van/DrwIuAH4PPCjV+zzauBj7Nxd5qXA/YOe8wDW/NPAc3Y/ftVhWPMl+32Snd/OeMdBzz2A73MTeAh4we7n33vQcw9gzW8H/mj34zHg68ANBz3701jzrcDNwBeu8njt/TqIV9RVbpb7WuB9ueM+oBkRzxv0oDXquubM/HRm/vfup/excyedYVb1pshvBj4EnB/kcH1SZc2vBxYy8ysAmTns666y5gRujIgAnsVOqC8Mdsz6ZOa97Kzhamrv10GEusrNcivdUHeI9Lqe32DnX+Rh1nXNEdECfhl49wDn6qcq3+cfAp4TEfdExNmIeOPApuuPKmv+U+BH2LmF3yrwlsx8YjDjHYja+1XpxgE1q3Kz3Eo31B0ildcTEa9gJ9Q/09eJ+q/Kmt8JvDUzt3debA29Kms+AvwE8EqgAXwmIu7LzH/r93B9UmXN08ADwG3ADwCfiIh/zsxv9Hm2g1J7vw4i1FVultv3G+oOWKX1RMSPAWeAV2Xmfw1otn6psuYp4O7dSB8FXh0RFzJzcSAT1q/q3+3HMvNx4PGIuBd4CTCsoa6y5jcB78idE7iPRMR/AD8M/MtgRhy42vt1EKc+Lt4sNyJuYOdmuR+5Yp+PAG/c/enpS4H/ycyvDXrQGnVdc0S8AFgA3jDEr64u1XXNmfnCzDyWmceADwK/NcSRhmp/t/8O+NmIOBIRzwB+Cnh4wHPWqcqav8LO/0EQEc8FJoBHBzrlYNXer4G/os6r3Cw3In5z9/F3s3MFwKuBR4D/Zedf5KFVcc1/AHwP8Oe7rzAv5BD/5rGKa76uVFlzZj4cER8HHgSeAM5k5r6XeQ2Dit/nPwTeGxGr7JwWeGtmDu2vP42IDwAvB45GxDngTmAU+tcv30IuSYXznYmSVDhDLUmFM9SSVDhDLUmFM9SSVDhDLUmFM9SSVLj/B9GJIQFDvTxKAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "plt.scatter(X,Y)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4afa94bd",
   "metadata": {},
   "source": [
    "### 2.1\n",
    "\n",
    "A toy dataset of Dutch cities featuring the number of pubs and churches. This small data is perfect to practice some statistical concepts. The objective of this task is to calculate and interpret the covariance and correlation coefficient.\n",
    "\n",
    "|Dutch City   | Number of Pubs  | Number of Churches  |\n",
    "|:---:|:---:|:---:|\n",
    "|  Leiden | 23  |16  |\n",
    "|  Maastricht |  25 |26 |\n",
    "|  Haarlem |17  |12  |\n",
    "|  Nijmegen |  38 |32 |\n",
    "|  Eindhoven | 40  |35  |\n",
    "|  Tilburg |  30 |26 |\n",
    "|  Groningen |  36 |36 |\n",
    "|  Den Bosch |  29 |32 |\n",
    "|  Scheveningen | 15  |8  |\n",
    "|  Venlo | 11  |14  |\n",
    "\n",
    "\n",
    "#### 2.1.1: Use the above data to: compute the i) Covariance and ii) Correlation between the two variables (the number of pubs and the number of churches)\n",
    "\n",
    "\n",
    "You can use `np.cov()` from the `numpy` library to compute the covariance.\n",
    "\n",
    "⚠️ **Note:** The Covariance is the value that appears twice, - (the second and third number from left to right); This is because Python is actually calculating the covariance matrix, $cov(x)$, $cov(x,y)$, $cov(y,x)$, and $cov(y)$ in that order, we only need $cov(x,y)$, which happen to be the same as $cov(y,x)$\n",
    "\n",
    "\n",
    "\n",
    "#### 2.1.2: Create a scatter plot of both variables."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ecd88c0b",
   "metadata": {},
   "source": [
    "## 3. Exploratory Data Analysis on a Crime Dataset from a State X\n",
    "\n",
    "The dataset consists of observations from the year 1987 for the crime rate in a particular State `X`. The State consists of counties. The dataset is available here: https://raw.githubusercontent.com/maastrichtlawtech/law3027-advanced-legal-analytics/main/data/crime_1987.csv . The data is aggregated by county. A brief description of various variables in the dataset is presented below:\n",
    "\n",
    "\n",
    "\n",
    "county\n",
    "\n",
    "    county identifier\n",
    "year\n",
    "\n",
    "    year = 1987\n",
    "crmrte\n",
    "\n",
    "    crimes committed per person\n",
    "prbarr\n",
    "\n",
    "    'probability' of arrest\n",
    "prbconv\n",
    "\n",
    "    'probability' of conviction\n",
    "prbpris\n",
    "\n",
    "    'probability' of prison sentence\n",
    "avgsen\n",
    "\n",
    "    average sentence, days\n",
    "polpc\n",
    "\n",
    "    police per capita\n",
    "density\n",
    "\n",
    "    hundreds of people per square mile\n",
    "taxpc\n",
    "\n",
    "    tax revenue per capita\n",
    "west\n",
    "\n",
    "    'west' = 1, if region is west for the State \n",
    "    \n",
    "central\n",
    "\n",
    "    'central' = 1, if region is central for the State\n",
    "\n",
    "urban\n",
    "\n",
    "    'urban' = 1 if in SMSA (Standard Metropolitan Statistical Area)\n",
    "    \n",
    "pctmin80\n",
    "\n",
    "    percentage minority in 1980\n",
    "    \n",
    "wcon\n",
    "\n",
    "    weekly wage in construction\n",
    "wtuc\n",
    "\n",
    "    weekly wage in trns, util, commun\n",
    "wtrd\n",
    "\n",
    "    weekly wage in whole sales and retail trade\n",
    "wfir\n",
    "\n",
    "    weekly wage in finance, insurance and real estate\n",
    "wser\n",
    "\n",
    "    weekly wage in service industry\n",
    "wmfg\n",
    "\n",
    "    weekly wage in manufacturing\n",
    "wfed\n",
    "\n",
    "    weekly wage of federal employees\n",
    "wsta\n",
    "\n",
    "    weekly wage of state employees\n",
    "wloc\n",
    "\n",
    "    weekly wage of local governments employees\n",
    "mix\n",
    "\n",
    "    offense mix: face-to-face/other\n",
    "pctymle\n",
    "\n",
    "    percentage of young males\n",
    "\n",
    "\n",
    "\n",
    "#### Objectives\n",
    "\n",
    "The target (or dependent) variable of interest is crime rate which is represented by `crmrte` (in theory/future we would like to use a machine learning algorithm like <strong>Linear Regression</strong> to predict the the crime rate, `crmrte`. Your goal is to find out which are the key variables (features) with most strong association with `crmrte`. This is called as feature selection phase in machine learning. You want to take only the most relevant features to predict the value of `crmrte`. Therefore, you must\n",
    "\n",
    "\n",
    "- Do an exploratory numerical and visual data analysis on the data variables\n",
    "- Explore and plot the relationships between various independent variables and the target variable: `crmrte`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "9fd2ced9",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Import the libraries\n",
    "import pandas as pd\n",
    "import seaborn as sns\n",
    "import matplotlib.pyplot as plt\n",
    "pd.set_option('display.max_columns', None)# Set no limit to display pandas rows and columns\n",
    "pd.set_option('display.max_rows', None)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f0583169",
   "metadata": {},
   "source": [
    "#### 3.1 Read the crime dataset for the year 1987"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "86a9ace8",
   "metadata": {},
   "source": [
    "#### 3.2 Inspect the data\n",
    "\n",
    "In the Legal Analytics course, you learned and performed  Numerical (Exploratory) and Visual Data Analaysis. In this tutorial, you will use that knowledge to explore the given Crime dataset. Use the `info()` function to check the datatypes and get a concise summary of the DataFrame."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0ee15662",
   "metadata": {},
   "source": [
    "#### 3.3 Numerical EDA\n",
    "\n",
    "Use the `describe()` function to get the summary statistics for the DataFrame. "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5481ed35",
   "metadata": {},
   "source": [
    "#### 3.4 Fix the data anomalies\n",
    "From the analysis of the `info()` and `describe()` function did you notice any anomalies in the dataset ? If so, please take approproate measures to fix these data anomalies."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5991ee15",
   "metadata": {},
   "source": [
    "#### 3.5 Univariate Visual Exploratory Data Analysis\n",
    "\n",
    "Conduct a univariate visual exploratory data analysis for the different columns (variables) in the crime dataset (you can plot the histograms, cumulative/probability distribution plots etc (feel free to recall the tutorial on Visual Exploratory Analysis from Legal Analytics Course).\n",
    "\n",
    "What are your observations. Did you find some outliers? If so, you may decide to drop those particular rows from the dataset"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "98956361",
   "metadata": {},
   "source": [
    "#### 3.6 Correlation Analysis\n",
    "\n",
    "The target variable of interest is crime rate which is represented by `crmrte` in the `data_crime` DataFrame. Therefore, we are interested in computing the correlation coefficient between `crmrte` and other variables in the dataset.\n",
    "\n",
    "Follow the tutorial here: https://www.sfu.ca/~mjbrydon/tutorials/BAinPy/08_correlation.html and \n",
    "\n",
    "- a) plot the relationship between `crmrte` and all the other independent variables in the dataset. \n",
    "- b) Compute the correlation matrix between all the variables of the `data_crime` DataFrame. Before doing this, drop the column `year` as its a constant.\n",
    "- c) Plot a heatmap of this correlation matrix using the `seaborn` library\n",
    "- d) Identify the top 5 most correlated variables (features) with `crmrte` \n",
    "- e) Plot a heatmap matrix of `crmrte` with the 5 most correlated variables (features)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}