F0C-00-ExploratoryDA.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# FLIP (00): Data Science \n",
    "**(Module 02: Data Visualization)**\n",
    "\n",
    "---\n",
    "- Materials in this module include resources collected from various open-source online repositories.\n",
    "- You are free to use,but NOT allowed to change and distribute this package.\n",
    "\n",
    "Prepared by and for \n",
    "**Student Members** |\n",
    "2006-2023 [TULIP Lab](http://www.tulip.org.au), Australia\n",
    "\n",
    "---\n",
    "\n",
    "\n",
    "# Session A - Exploratory Data Analysis\n",
    "\n",
    "\n",
    "This practical session will show you how to use packages for data exploration.\n",
    "\n",
    "\n",
    "## Content\n",
    "\n",
    "### Part 1 Matplotlib Module\n",
    "\n",
    "\n",
    "### Part 2 Plotting a Histogram\n",
    "\n",
    "2.1 [Dataset](#ds)\n",
    "\n",
    "2.2 [Histogram](#hist)\n",
    "\n",
    "2.3 [Boxplot](#boxplot)\n",
    "\n",
    "\n",
    "### Part 3: Data Understanding\n",
    "\n",
    "1.1 [Pie Chart](#pie)\n",
    "\n",
    "1.2 [Bar Chart](#bar)\n",
    "\n",
    "1.3 [Word Cloud](#wordcloud)\n",
    "\n",
    "1.4 [Step Plot](#stepplot)\n",
    "\n",
    "1.5 [Histogram](#histogram)\n",
    "\n",
    "1.6 [Box Plot](#box)\n",
    "\n",
    "1.7 [Scatter Plot](#scatter)\n",
    "\n",
    "### Part 4: Exercise\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## <span style=\"color:#0b486b\">1. Matplotlib</span>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "matplotlib is a python plotting library which produces publication quality figures in a variety of hardcopy formats and interactive environments across platforms. You can generate plots, histograms, power spectra, bar charts, errorcharts, scatterplots, etc, with just a few lines of code. \n",
    "\n",
    "For simple plotting the pyplot interface provides a MATLAB-like interface, particularly when combined with IPython. You have full control of line styles, font properties, axes properties, etc, via an object oriented interface or via a set of functions familiar to MATLAB users."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### <span style=\"color:#0b486b\">1.1 Get started</span>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To get started with `'matplotlib'` you can either execute:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from pylab import *"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "or"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import matplotlib.pyplot"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In fact it is a convention to import it under the name of `'plt'`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import matplotlib.pyplot as plt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**note: The second method is preferred.**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "import numpy as np"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Regardless of the method you use, it is better to configure matplotlib to embed figures in the notebook instead of opening them in a new window for each figure. To do this use the magic function:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### <span style=\"color:#0b486b\">1.2 `plot`</span>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "By using `'subplots()'` you have access to both figure and axes objects. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "x = np.linspace(0, 10)\n",
    "y = np.sin(x)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "fig, ax = plt.subplots()\n",
    "ax.plot(x, y)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### <span style=\"color:#0b486b\">1.3 title and labels</span>\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ax.set_title('title here!')\n",
    "ax.set_xlabel('x')\n",
    "ax.set_ylabel('sin(x)')\n",
    "fig"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can also use $\\LaTeX$ in title or labels, or change the font size or font family."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "x = np.linspace(-10, 10)\n",
    "fig, ax = plt.subplots(figsize=(8, 5), dpi=300)\n",
    "ax.plot(x, x**3-x**2, 'bo', lw=3)\n",
    "\n",
    "ax.set_title('$x^3-x^2$', fontsize=18)\n",
    "ax.set_xlabel('$x$', fontsize=18)\n",
    "ax.set_ylabel('$y$', fontsize=18)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### <span style=\"color:#0b486b\">1.4 Subplots</span>\n",
    "\n",
    "You can pass the number of subplots to `'subplots()'`. In this case, `'axes'` will be an array that each of its elements associates with one of the subgraphs. You can set properties of each `'ax'` object separately like the cell below. \n",
    "\n",
    "Obviously you caould use a loop to iterate over `'axes'`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "fig, axes = plt.subplots(nrows=1, ncols=2)\n",
    "\n",
    "x = np.linspace(0, 10)\n",
    "\n",
    "axes[0].plot(x, np.sin(x))\n",
    "axes[0].set_xlabel('x')\n",
    "axes[0].set_ylabel('sin(x)')\n",
    "\n",
    "axes[1].plot(x, np.cos(x))\n",
    "axes[1].set_xlabel('xx')\n",
    "axes[1].set_ylabel('cos(x)')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`'cos(x)'` label is overlapping with the `'sin'` graph. You can adjust the size of the graph or space between the subplots to fix it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "fig, axes = plt.subplots(nrows=1, ncols=2, figsize=(10, 5))\n",
    "fig.subplots_adjust(wspace=0.4)\n",
    "x = np.linspace(0, 10)\n",
    "\n",
    "axes[0].plot(x, np.sin(x))\n",
    "axes[0].set_xlabel('x')\n",
    "axes[0].set_ylabel('sin(x)')\n",
    "\n",
    "axes[1].plot(x, np.cos(x))\n",
    "axes[1].set_xlabel('xx')\n",
    "axes[1].set_ylabel('cos(x)')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### <span style=\"color:#0b486b\">1.5 Legend</span>\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "x = np.linspace(0, 10)\n",
    "fig, ax = plt.subplots(figsize=(7, 5))\n",
    "ax.plot(x, np.sin(x), label='$sin(x)$')\n",
    "ax.plot(x, np.cos(x), label='$cos(x)$')\n",
    "# ax.legend(fontsize=16, loc=3)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### <span style=\"color:#0b486b\">1.6 Customizing ticks</span>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In many cases you want to customize the ticks and their labels on x or y axis. First draw a simple graph and look at the ticks on x-axis. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "x = np.linspace(0, 10, num=100)\n",
    "\n",
    "fig, ax = plt.subplots(figsize=(10, 5))\n",
    "ax.plot(x, np.sin(x), x, np.cos(x), lw=2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can change the ticks easily with passing a list (or array) to `'set_xticks()'` or `'set_yticks()'`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "xticks = [0, 1, 2, 5, 8, 8.5, 10]\n",
    "ax.set_xticks(xticks)\n",
    "fig"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "Or even you can change the labels:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "xticklabels = ['$\\gamma$', '$\\delta$', 'apple', 'b', '', 'c',''] \n",
    "ax.set_xticklabels(xticklabels, fontsize=18)\n",
    "fig"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### <span style=\"color:#0b486b\">1.7 Saving figures</span>\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "x = np.linspace(0, 10)\n",
    "fig, ax = plt.subplots(figsize=(7, 5))\n",
    "ax.plot(x, np.sin(x), label='$sin(x)$')\n",
    "ax.plot(x, np.cos(x), label='$cos(x)$')\n",
    "ax.legend(fontsize=16, loc=3)\n",
    "fig.savefig('P03Saved.pdf', format='PDF', dpi=300)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### <span style=\"color:#0b486b\">1.8 Other plot styles</span>\n",
    "\n",
    "There are many other plot types in addition to simple `'plot'` supported by `'matplotlib'`. You will find a complete list of them on [matplotlib gallery](http://matplotlib.org/gallery.html)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### <span style=\"color:#0b486b\">1.8.1 Scatter plot</span>\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "fig, ax = plt.subplots()\n",
    "\n",
    "x = np.linspace(-0.75, 1., 100)\n",
    "ax.scatter(x, np.random.randn(x.shape[0]), \n",
    "                   s = 250*np.abs(np.random.randn(x.shape[0])), \n",
    "                   alpha=0.4,\n",
    "                   edgecolor='none')\n",
    "ax.set_title('scatter')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### <span style=\"color:#0b486b\">1.8.2 Bar plot</span>\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "fig, ax = plt.subplots()\n",
    "\n",
    "x = np.arange(1, 6)\n",
    "ax.bar(x, x**2, align=\"center\")\n",
    "ax.set_title('bar')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "## <span style=\"color:#0b486b\">2. Plotting a histogram</span>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id = \"ds\"></a>\n",
    "\n",
    "\n",
    "### <span style=\"color:#0b486b\">2.1 Dataset</span>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You are provided with a dataset of percentage of body fat and 10 simple body measurements recoreded for 252 men (courtesy of Journal of Statistics Education - JSE). You can read about this and other [JSE datasets here](http://www.amstat.org/publications/jse/jse_data_archive.htm).\n",
    "\n",
    "First load the data set into an array:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!pip3 install wget"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import wget\n",
    "link_to_data = 'https://github.com/tulip-lab/mds/blob/master/Jupyter/data/fat.dat.txt'\n",
    "#link_to_data = 'https://github.com/tuliplab/mds/raw/master/Jupyter/data/fat.dat.txt' 这个路径改变了\n",
    "DataSet = wget.download(link_to_data)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "data = np.genfromtxt(\"./data/Data2/fat.dat.txt\")\n",
    "data.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Based on the [dataset description](http://www.amstat.org/publications/jse/datasets/fat.txt), 5th column represents the weight in lbs. Index the weight column and call it `'weights'`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "weights = data[:, 5]\n",
    "weights"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Use array operators to convert the weigts into kg. 1 lb equals to 0.453592 kg."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "a = 3\n",
    "a = a + 4\n",
    "a += 4"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "weights *= 0.453592\n",
    "weights = weights.round(2)\n",
    "weights"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id = \"hist\"></a>\n",
    "\n",
    "### <span style=\"color:#0b486b\">2.2 Histogram</span>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A histogtram is a bar plot that shows you the statistical distribution of the data over a variable. The bars represent the frequency of occurenve by classess of data. We use the package `'matplotlib'` and the function `'hist()'` for plotting the histogram. To learn more about `'matplotlib'` make sure you have read tutorial.\n",
    "\n",
    "The first line of the cell below if for showing the figure in the notebook and not opening it in a separate window."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%matplotlib inline\n",
    "import matplotlib.pyplot as plt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "fig, ax = plt.subplots()\n",
    "ax.hist(weights)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `'hist()'` functions automatically group the data over 10 bins. Usually you need to tweek the number of bins to obtain a more expressive histogram."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "fig, ax = plt.subplots(figsize=(7, 5))\n",
    "ax.hist(weights, bins=20)\n",
    "# title\n",
    "# label"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id = \"boxplot\"></a>\n",
    "\n",
    "### <span style=\"color:#0b486b\">2.3 Boxplot</span>\n",
    "\n",
    "A `Boxplot` is a convenient way to graphically display numerical data. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import matplotlib\n",
    "fig, ax = matplotlib.pyplot.subplots(figsize=(7, 5))\n",
    "matplotlib.rcParams.update({'font.size': 14})\n",
    "ax.boxplot(weights, 0, labels=['group1'])\n",
    "ax.set_ylabel('weight (kg)', fontsize=16)\n",
    "ax.set_title('Weights BoxPlot', fontsize=16)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You have already been thought about different sorts of plots, how they help to get a better understanding of the data, and when to use which. In this practical session we will work with `matplotlib` package to learn more about plotting in Python."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "## 3. Data Understanding \n",
    "\n",
    "\n",
    "You have already been thought about different sorts of plots, how they help to get a better understanding of the data, and when to use which. In this practical session we will work with `matplotlib` package to learn more about plotting in Python."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import csv\n",
    "\n",
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id = \"pie\"></a>\n",
    "\n",
    "### <span style=\"color:#0b486b\">3.1 Pie Chart</span>\n",
    "\n",
    "Suppose you have the frequency count of a variable (e.g. hair_colour). Draw a pie chart to explain it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "labels = 'Black', 'Red', 'Brown'\n",
    "\n",
    "# frequency count\n",
    "hair_colour_freq = [5, 3, 2]  # Black, Red, Brown\n",
    "\n",
    "# colors\n",
    "colors = ['yellowgreen', 'gold', 'lightskyblue']\n",
    "\n",
    "# explode the third one\n",
    "explode = (0, 0, 0.1)\n",
    "\n",
    "fig, ax = plt.subplots(figsize=(5, 5))\n",
    "ax.pie(hair_colour_freq, labels=labels, explode=explode, colors=colors, \n",
    "       autopct='%1.1f', shadow=True, startangle=90);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "What if we have too many tags and sectors?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Excellence in Reasearch Australia\n",
    "labels = ['HEALTH', 'ENGINEERING', 'COMPUTER SCIENCES', 'HUMAN SOCIETY', \n",
    "          'TOURISM SERVICES', 'EDUCATION', 'CHEMISTRY', 'BIOLOGY', 'PSYCHOLOGY', \n",
    "          'CREATIVE ARTS', 'LINGUISTICS', 'BUILT ENVIRONMENT', 'HISTORY', \n",
    "          'ECONOMICS', 'PHILOSOPHY', 'AGRICULTURE', 'ENVIRONMENT', 'TECHNOLOGY', \n",
    "          'LAW', 'MATHS', 'EARTH SCIENCES', 'PHYSICS']\n",
    "\n",
    "\n",
    "# frequency count\n",
    "xx = [2625.179999, 1306.259999, 1187.039999, 1166.04, 980.8599997, 810.5999998,\n",
    "      725.6399996, 678.7899998, 436.5999997, 404.3299999, 348.01, 304.33, 294.19, \n",
    "      293.02, 282.31, 228.21, 197.3399999, 164.0599998, 157, 50.49999998, 49.60999999, 48.08000005]\n",
    "\n",
    "fig, ax = plt.subplots(figsize=(10, 10))\n",
    "ax.pie(xx, labels=labels, autopct=\"%1.1f\");"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id = \"bar\"></a>\n",
    "\n",
    "### <span style=\"color:#0b486b\">3.2 Bar Chart</span>\n",
    "\n",
    "Use the hair colour data to draw a bar chart."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "labels = ['Black', 'Red', 'Brown']\n",
    "hair_colour_freq = [5, 3, 2]\n",
    "\n",
    "fig, ax = plt.subplots(figsize=(7, 5), dpi=300)\n",
    "\n",
    "x_pos = np.arange(len(hair_colour_freq))\n",
    "colors = ['black', 'red', 'brown']\n",
    "\n",
    "ax.bar(x_pos, hair_colour_freq, align='center', color=colors)\n",
    "\n",
    "ax.set_xlabel(\"Hair Colour\")\n",
    "ax.set_ylabel(\"Number of participants\")\n",
    "ax.set_title(\"Hair Colour Distribution\")\n",
    "\n",
    "ax.set_xticks(x_pos)\n",
    "ax.set_xticklabels(labels)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now suppose we have the hair colour distribution across genders, so we can plot grouped bar charts. Plot a grouped bar chart to show the distribution of colours acros genders."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "\"\"\"\n",
    "        black  red  brown\n",
    "Male      4     1     3\n",
    "Female    1     2     2\n",
    "\n",
    "\"\"\"\n",
    "\n",
    "data = np.array([[4, 1, 3], \n",
    "                 [1, 2, 3]])\n",
    "\n",
    "x_pos = np.arange(2)\n",
    "width = 0.2\n",
    "fig, ax = plt.subplots(figsize=(7, 5), dpi=300)\n",
    "ax.bar(x_pos, data[:, 0], width=width, color='black', label='Black', align='center')\n",
    "ax.bar(x_pos+width, data[:, 1], width=width, color='red', label='Red', align='center')\n",
    "ax.bar(x_pos+2*width, data[:, 2], width=width, color='brown', label='Brown', align='center')\n",
    "\n",
    "ax.legend()\n",
    "\n",
    "ax.set_xlabel(\"Gender\")\n",
    "ax.set_ylabel(\"Frequency\")\n",
    "ax.set_title(\"Distribution of hair colour amongst genders\")\n",
    "\n",
    "ax.set_xticks(x_pos+width)\n",
    "ax.set_xticklabels(['Male', 'Female'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Can we plot it more intelligently? We are doing the same thing multiple times! Is it a good idea to use a loop?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "\"\"\"\n",
    "        black  red  brown\n",
    "Male      4     1     3\n",
    "Female    1     2     2\n",
    "\n",
    "\"\"\"\n",
    "\n",
    "data = np.array([[4, 1, 3], \n",
    "                 [1, 2, 3]])\n",
    "\n",
    "n_groups, n_colours = data.shape\n",
    "\n",
    "x_pos = np.arange(n_groups)\n",
    "width = 0.2\n",
    "fig, ax = plt.subplots(figsize=(7, 5), dpi=300)\n",
    "\n",
    "colours = ['black', 'red', 'brown']\n",
    "labels = ['Black', 'Red', 'Brown']\n",
    "for i in range(n_colours):\n",
    "    ax.bar(x_pos + i*width, data[:, i], width=width, color=colours[i], label=labels[i], align='center')\n",
    "    \n",
    "ax.legend()\n",
    "\n",
    "ax.set_xlabel(\"Gender\")\n",
    "ax.set_ylabel(\"Frequency\")\n",
    "ax.set_title(\"Distribution of hair colour amongst genders\")\n",
    "\n",
    "ax.set_xticks(x_pos+width)\n",
    "ax.set_xticklabels(['Male', 'Female'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "What if we want to group the bar charts based on the hair colour?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "\"\"\"\n",
    "        black  red  brown\n",
    "Male      4     1     3\n",
    "Female    1     2     2\n",
    "\n",
    "\"\"\"\n",
    "\n",
    "labels = ['Black', 'Red', 'Brown']\n",
    "colours = ['r', 'y']\n",
    "data = np.array([[4, 1, 3], \n",
    "                 [1, 2, 3]])\n",
    "\n",
    "n_groups, n_colours = data.shape\n",
    "width = 0.2\n",
    "x_pos = np.arange(n_colours)\n",
    "\n",
    "fig, ax = plt.subplots(figsize=(7, 5), dpi=300)\n",
    "for i in range(n_groups):\n",
    "    ax.bar(x_pos + i*width, data[i, :], width, align='center', label=labels[i], color=colours[i])\n",
    "ax.set_xlabel(\"Hair Colour\")\n",
    "ax.set_ylabel(\"Frequency\")\n",
    "ax.set_title(\"Distribution of gender amongst hair colours\")\n",
    "\n",
    "ax.set_xticks(x_pos+width/2)\n",
    "ax.set_xticklabels(labels)\n",
    "\n",
    "ax.legend()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Stacked bar chart\n",
    "\n",
    "The other type of bar chart is stacked bar chart. draw a stacked bar plot of the hair colour data grouped on hair colours."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "\"\"\"\n",
    "        black  red  brown\n",
    "Male      4     1     3\n",
    "Female    1     2     2\n",
    "\n",
    "\"\"\"\n",
    "\n",
    "labels = ['Black', 'Red', 'Brown']\n",
    "data = np.array([[4, 1, 3], \n",
    "                 [1, 2, 3]])\n",
    "\n",
    "male_freq = data[0,:]\n",
    "\n",
    "width = 0.4\n",
    "x_pos = np.arange(n_colours)\n",
    "\n",
    "fig, ax = plt.subplots(figsize=(7, 5), dpi=300)\n",
    "ax.bar(x_pos, data[0, :], width, align='center', label='Male', color='r')\n",
    "ax.bar(x_pos, data[1, :], width, bottom=male_freq, align='center', label='Female', color='y')\n",
    "\n",
    "ax.set_xlabel(\"Hair Colour\")\n",
    "ax.set_ylabel(\"Frequency\")\n",
    "ax.set_title(\"Distribution of gender amongst hair colours\")\n",
    "\n",
    "ax.set_xticks(x_pos)\n",
    "ax.set_xticklabels(labels)\n",
    "\n",
    "ax.legend(loc=0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "draw a stacked bar plot grouped on the gender."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "\"\"\"\n",
    "        black  red  brown\n",
    "Male      4     1     3\n",
    "Female    1     2     2\n",
    "\n",
    "\"\"\"\n",
    "\n",
    "labels = ['Black', 'Red', 'Brown']\n",
    "data = np.array([[4, 1, 3], \n",
    "                 [1, 2, 3]])\n",
    "\n",
    "black = data[:,0]\n",
    "red = data[:,1]\n",
    "brown = data[:,2]\n",
    "\n",
    "\n",
    "x_pos = np.arange(2)\n",
    "width = 0.4\n",
    "fig, ax = plt.subplots(figsize=(7, 5), dpi=300)\n",
    "ax.bar(x_pos, data[:, 0], width=width, color='black', label='Black', align='center')\n",
    "ax.bar(x_pos, data[:, 1], width=width, bottom=black, color='red', label='Red', align='center')\n",
    "ax.bar(x_pos, data[:, 2], width=width, color='brown', bottom=black+red, label='Brown', align='center')\n",
    "\n",
    "ax.legend(loc=0)\n",
    "\n",
    "ax.set_xlabel(\"Gender\")\n",
    "ax.set_ylabel(\"Frequency\")\n",
    "ax.set_title(\"Distribution of hair colour amongst genders\")\n",
    "\n",
    "ax.set_xticks(x_pos)\n",
    "ax.set_xticklabels(['Male', 'Female'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "labels = ['Black', 'Red', 'Brown']\n",
    "male_freq = [4, 1, 3]\n",
    "female_freq = [1, 2, 2]\n",
    "\n",
    "x_pos = np.arange(3)\n",
    "\n",
    "fig, ax = plt.subplots(nrows=1, ncols=2, figsize=(14, 5), dpi=300)\n",
    "\n",
    "w1 = 0.2\n",
    "w2 = 0.4\n",
    "\n",
    "ax[0].bar(x_pos, male_freq, width=w1, align='center', label='Male', color='r')\n",
    "ax[0].bar(x_pos+width, female_freq, width=w1, align='center', label='Female', color='y')\n",
    "ax[1].bar(x_pos, male_freq, width=w2, align='center', label='Male', color='r')\n",
    "ax[1].bar(x_pos, female_freq, width=w2, bottom=male_freq, align='center', label='Female', color='y')\n",
    "\n",
    "\n",
    "ax[0].set_xlabel(\"Hair Colour\")\n",
    "ax[0].set_ylabel(\"Frequency\")\n",
    "ax[0].set_title(\"Distribution of gender amongst hair colours\")\n",
    "ax[1].set_xlabel(\"Hair Colour\")\n",
    "ax[1].set_ylabel(\"Frequency\")\n",
    "ax[1].set_title(\"Distribution of gender amongst hair colours\")\n",
    "\n",
    "ax[0].set_xticks(x_pos+width/2)\n",
    "ax[0].set_xticklabels(labels)\n",
    "ax[1].set_xticks(x_pos)\n",
    "ax[1].set_xticklabels(labels)\n",
    "\n",
    "ax[0].legend()\n",
    "ax[1].legend(loc=0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "What if we have too many groups? Draw a bar chart for the Excellence in Research Australia data. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "import numpy as np\n",
    "\n",
    "# Excellence in Research Australia\n",
    "labels = ['HEALTH', 'ENGINEERING', 'COMPUTER SCIENCES', 'HUMAN SOCIETY', \n",
    "          'TOURISM SERVICES', 'EDUCATION', 'CHEMISTRY', 'BIOLOGY', 'PSYCHOLOGY', \n",
    "          'CREATIVE ARTS', 'LINGUISTICS', 'BUILT ENVIRONMENT', 'HISTORY', \n",
    "          'ECONOMICS', 'PHILOSOPHY', 'AGRICULTURE', 'ENVIRONMENT', 'TECHNOLOGY', \n",
    "          'LAW', 'MATHS', 'EARTH SCIENCES', 'PHYSICS']\n",
    "\n",
    "\n",
    "# frequency count\n",
    "xx = [2625.179999, 1306.259999, 1187.039999, 1166.04, 980.8599997, 810.5999998,\n",
    "      725.6399996, 678.7899998, 436.5999997, 404.3299999, 348.01, 304.33, 294.19, \n",
    "      293.02, 282.31, 228.21, 197.3399999, 164.0599998, 157, 50.49999998, 49.60999999, 48.08000005]\n",
    "\n",
    "xx_pos = np.arange(len(xx))\n",
    "\n",
    "fig, ax = plt.subplots(figsize=(15, 5))\n",
    "ax.bar(xx_pos, xx, align='center')\n",
    "ax.set_xlabel(\"research subject\")\n",
    "ax.set_ylabel(\"score\")\n",
    "ax.set_xticks(xx_pos)\n",
    "ax.set_xticklabels(labels, rotation=90)\n",
    "ax.set_xlim(-1, len(xx))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id = \"wordcloud\"></a>\n",
    "\n",
    "### <span style=\"color:#0b486b\">3.3 Wordcloud</span>\n",
    "\n",
    "As you saw, pie-chart is not very helpful when we have too many sectors. It is hard to read and visually ugly. Instead we can use wordcloud representation. A useful tool is [wordle.net](http://wordle.net). Go to [wordle.net](http://wordle.net) and use it to create a wordcloud for the previous data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "for i in range(len(labels)):\n",
    "    print(\"{}:{}\".format(labels[i], xx[i]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!pip3 install wordcloud"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import wget\n",
    "\n",
    "link_to_data = 'https://github.com/benbrandt/cs50/blob/master/pset5/keys/constitution.txt'\n",
    "#link_to_data = 'https://github.com/benbrandt/cs50/raw/master/pset5/keys/constitution.txt'\n",
    "DataSet = wget.download(link_to_data)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from os import path\n",
    "from wordcloud import WordCloud\n",
    "\n",
    "# Read the whole text.\n",
    "text = open('constitution.txt',encoding='gb18030',errors='ignore').read()\n",
    "\n",
    "# Generate a word cloud image\n",
    "wordcloud = WordCloud().generate(text)\n",
    "\n",
    "# Display the generated image:\n",
    "# the matplotlib way:\n",
    "import matplotlib.pyplot as plt\n",
    "plt.imshow(wordcloud, interpolation='bilinear')\n",
    "plt.axis(\"off\")\n",
    "\n",
    "# lower max_font_size\n",
    "wordcloud = WordCloud(max_font_size=40).generate(text)\n",
    "plt.figure()\n",
    "plt.imshow(wordcloud, interpolation=\"bilinear\")\n",
    "plt.axis(\"off\")\n",
    "plt.show()\n",
    "\n",
    "# The pil way (if you don't have matplotlib)\n",
    "# image = wordcloud.to_image()\n",
    "# image.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id = \"stepplot\"></a>\n",
    "\n",
    "### <span style=\"color:#0b486b\">3.4 Step plot</span>\n",
    "\n",
    "Draw a step plot for the seatbelt data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "freq = np.array([0, 2, 1, 5, 7])\n",
    "labels = ['Never', 'Rarely', 'Sometimes', 'Most-times', 'Always']\n",
    "freq_cumsum = np.cumsum(freq)\n",
    "x_pos = np.arange(len(freq))\n",
    "\n",
    "fig, ax = plt.subplots()\n",
    "ax.step(x_pos, freq_cumsum, where='mid')\n",
    "ax.set_xlabel(\"Fastening seatbelt behaviour\")\n",
    "ax.set_ylabel(\"Cumulative frequency\")\n",
    "ax.set_xticks(x_pos)\n",
    "ax.set_xticklabels(labels)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id = \"histogram\"></a>\n",
    "\n",
    "### <span style=\"color:#0b486b\">3.5 Histogram</span>\n",
    "\n",
    "Google for this paper:\n",
    "\n",
    "``Johnson, Roger W. \"Fitting percentage of body fat to simple body measurements.\" Journal of Statistics Education 4.1 (1996): 265-266.``\n",
    "\n",
    "Download the dataset and read the dataset description. Draw a histogram of male weights and female weights."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import wget\n",
    "\n",
    "link_to_data = 'https://ww2.amstat.org/publications/jse/datasets/body.dat.txt'\n",
    "DataSet = wget.download(link_to_data)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "data = np.genfromtxt('body.dat.txt')\n",
    "m_w = data[data[:, -1] == 1][:, -3]\n",
    "f_w = data[data[:, -1] == 0][:, -3]\n",
    "\n",
    "fig, ax = plt.subplots(figsize=(7, 5), dpi=300)\n",
    "ax.hist(m_w, bins=15, alpha=0.6, label='male')\n",
    "ax.hist(f_w, bins=15, alpha=0.6, label='female')\n",
    "ax.set_xlabel(\"weight (kg)\")\n",
    "ax.set_title(\"weight distribution amongst gXenders\")\n",
    "ax.legend()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id = \"box\"></a>\n",
    "\n",
    "### <span style=\"color:#0b486b\">3.6 Boxplot</span>\n",
    "Draw a box plot for male and female weights of the previous dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "fig, ax = plt.subplots(figsize=(7, 5), dpi=300)\n",
    "ax.boxplot([m_w, f_w], labels=['male', 'female'])\n",
    "ax.set_title(\"weight distribution amongst genders\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id = \"scatter\"></a>\n",
    "\n",
    "### <span style=\"color:#0b486b\">3.7 Scatter plot</span>\n",
    "\n",
    "Draw a scatter plot of the car weights and their fuel consumption as displayed in the lecture."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import wget\n",
    "\n",
    "#link_to_data = 'https://github.com/tulip-lab/mds/blob/master/Jupyter/data/Auto.csv'这是正确的地址\n",
    "#link_to_data = 'https://github.com/tuliplab/mds/raw/master/Jupyter/data/Auto.csv' 地址变了\n",
    "DataSet = wget.download(link_to_data)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "datafile = './data/Data1/Auto.csv'\n",
    "data = np.genfromtxt(datafile, delimiter=',')\n",
    "data = []\n",
    "with open(datafile,'r') as fp:\n",
    "    reader = csv.reader(fp, delimiter=',')\n",
    "    for row in reader:\n",
    "        data.append(row)\n",
    "miles = [dd[1] for dd in data[1:]]\n",
    "weights = [dd[5] for dd in data[1:]]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "fig, ax = plt.subplots(figsize=(7, 5), dpi=300)\n",
    "ax.scatter(weights,miles,alpha=0.6, edgecolor='none', s=200)\n",
    "ax.set_xlabel('Car Weight (tons)')\n",
    "ax.set_ylabel('Miles Per Gallon')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Can I also show the number of cylinders on this graph? In other words use the scatter plot to show three variable?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "cylinder = 75 * np.array([int(dd[2]) for dd in data[1:]])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "fig, ax = plt.subplots(figsize=(7, 5), dpi=300)\n",
    "ax.scatter(weights,miles, alpha=0.6, edgecolor='none', s=cylinder)\n",
    "ax.set_xlabel('Car Weight (tons)')\n",
    "ax.set_ylabel('Miles Per Gallon')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "## 4. Mini Exercise\n",
    "\n",
    "\n",
    "In 1970, US Congress instituted a random selection process for the military draft. All 366 possible birth dates were placed in plastic capsules in a rotating drum and were selected one by one. The first date drawn from the drum received draft number one and eligible men born on that date were drafted first. The data is provided in a text file with a structure like:\n",
    "\n",
    "```\n",
    "Day    Month    MO.NUMBER    DAY_OF_YEAR    DRAFT_NO.\n",
    "1      JAN      1            1              305\n",
    "2      JAN      1            2              159\n",
    ".\n",
    "31     JAN      1            31             221\n",
    "1      FEB      2            32             86\n",
    ".\n",
    "31     Dec      12           366            100\n",
    "```\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Using what you have learnt by now, can you tell if it was a fair lottary or not?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Read the data file and save the values in a 2D array."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import wget\n",
    "\n",
    "link_to_data = 'https://github.com/tulip-lab/mds/blob/master/Jupyter/data/DraftLottery.txt'\n",
    "#link_to_data = 'https://github.com/tulip-lab/mds/blod/master/Jupyter/data/DraftLottery.txt'\n",
    "DataSet = wget.download(link_to_data)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "data = []\n",
    "with open('./data/Data1/DraftLottery.txt', 'r') as fp:\n",
    "    reader = csv.reader(fp, delimiter='\\t')\n",
    "    for row in reader:\n",
    "        data.append(row)\n",
    "        \n",
    "birthdays = np.array([int(row[3]) for row in data[1:]])\n",
    "draft_no = np.array([int(row[4]) for row in data[1:]])\n",
    "months = np.array([int(row[2]) for row in data[1:]])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Plot a `'scatter plot'` of the draft priority vs birthdays."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "fig, ax = plt.subplots(figsize=(10, 7), dpi=300)\n",
    "ax.scatter(birthdays, draft_no, alpha=0.7, s = 150, edgecolor='none')\n",
    "ax.set_xlabel(\"Birthday (day of the year)\", fontsize=12)\n",
    "ax.set_ylabel(\"Draft priority value\", fontsize=12)\n",
    "ax.set_title(\"USA Draft Lottery Data\", fontsize=14)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In a truly random lottery there should be no relationship between the date and the draft number. To investigate this further we draw boxplots by months and compare them together."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "fig, ax = plt.subplots(figsize=(10, 7), dpi=300)\n",
    "months_range = range(1, 13)\n",
    "\n",
    "# boxplot data\n",
    "boxplot_data = [draft_no[months == mm] for mm in months_range]\n",
    "ax.boxplot(boxplot_data)\n",
    "\n",
    "# medians\n",
    "medians = [np.median(dd) for dd in boxplot_data]\n",
    "ax.plot(months_range, medians, \"g--\", lw=2)\n",
    "\n",
    "# means\n",
    "means = [dd.mean() for dd in boxplot_data]\n",
    "ax.plot(months_range, means, \"k--\", lw=2)\n",
    "\n",
    "month_labels = [\"Jan\", \"Feb\", \"Mar\", \"Apr\", \"May\", \"Jun\",\n",
    "                \"Jul\", \"Aug\", \"Sep\", \"Oct\", \"Nov\", \"Dec\"]\n",
    "ax.set_xlabel(\"Month\", fontsize=12)\n",
    "ax.set_xticklabels(month_labels)\n",
    "ax.set_ylabel(\"Draft priority value\", fontsize=12)\n",
    "ax.set_title(\"USA Draft Lottery Data\", fontsize=14)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "While it is impossible to view this trend in a scatterplot of draft number vs. birth date, a series of side-by-side boxplots by month illustrate it clearly. A further investigation of the lottery revealed that the birthdates were placed in the drum by month and were not thoroughly mixed."
   ]
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}