01_python_tools_ds.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "# Python tools for data science\n",
    "(last updated 2020-05-18)\n",
    "\n",
    "## The PyData Stack\n",
    "\n",
    "The Python Data Stack comprises a set of packages that makes Python a powerful data science language. These include\n",
    "\n",
    "+ Numpy: provides arrays and matrix algebra\n",
    "+ Scipy: provides scientific computing capabilities\n",
    "+ matplotlib: provides graphing capabilities\n",
    "\n",
    "These were the original stack that was meant to replace Matlab. However, these were meant to tackle purely numerical data, and the kinds of heterogeneous data we regularly face needed more tools. These were added more recently.\n",
    "\n",
    "+ Pandas: provides data analytic structures like the data frame, as well as basic descriptive statistical capabilities\n",
    "+ statsmodels: provides a fairly comprehensive set of statistical functions\n",
    "+ scikit-learn: provides machine learning capabilities\n",
    "\n",
    "This is the basic stack of packages we will be using in this workshop. Additionally we will use a few packages that add some functionality to the data science process. These include\n",
    "\n",
    "+ seaborn: Better statistical graphs\n",
    "\n",
    "+ plotly: Interactive graphics\n",
    "\n",
    "+ biopython: Python for bioinformatics\n",
    "\n",
    "We may also introduce the package `rpy2` which allows one to run R from within Python. This can be useful since many bioinformatic pipelines are already implemented in R. \n",
    "\n",
    "> The [PyData stack](https://scipy.org) also includes `sympy`, a symbolic mathematics package emulating Maple\n",
    "\n",
    "## Numpy (numerical and scientific computing)\n",
    "\n",
    "We start by importing the Numpy package into Python using the alias `np`. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "name": "01-python-tools-ds-2"
   },
   "outputs": [],
   "source": [
    "import numpy as np"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Numpy provides both arrays (vectors, matrices, higher dimensional arrays) and vectorized functions which are very fast. Let's see how this works."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "name": "01-python-tools-ds-3"
   },
   "outputs": [],
   "source": [
    "z = [1,2,3,4,5,6,7,8,9.3,10.6] # This is a list\n",
    "z_array = np.array(z)\n",
    "z_array"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, we have already seen functions in Python earlier. In Numpy, there are functions that are optimized for arrays, that can be accessed directly from the array objects. This is an example of *object-oriented programming* in Python, where functions are provided for particular *classes* of objects, and which can be directly accessed from the objects. We will use several such functions over the course of this workshop, but we won't actually talk about how to do this program development here. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> Numpy functions are often very fast, and are *vectorized*, i.e., they are written to work on vectors of numbers rather than single numbers. This is an advantage in data science since we often want to do the same operation to all elements of a column of data, which is essentially a vector"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We apply the functions `sum`, `min` (minimum value) and `max` (maximum value) to `z_array`. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "name": "01-python-tools-ds-4"
   },
   "outputs": [],
   "source": [
    "z_array.sum()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "name": "01-python-tools-ds-5"
   },
   "outputs": [],
   "source": [
    "z_array.min()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "name": "01-python-tools-ds-6"
   },
   "outputs": [],
   "source": [
    "z_array.max()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The versions of these functions in Numpy are optimized for arrays and are quite a bit faster than the corresponding functions available in base Python. When doing data work, these are the preferred functions. \n",
    "\n",
    "These functions can also be used in the usual function manner:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "name": "01-python-tools-ds-7"
   },
   "outputs": [],
   "source": [
    "np.max(z_array)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Calling `np.max` ensures that we are using the `max` function from numpy, and not the one in base Python. \n",
    "\n",
    "### Numpy data types\n",
    "\n",
    "Numpy arrays are homogeneous in type."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "name": "01-python-tools-ds-8"
   },
   "outputs": [],
   "source": [
    "np.array(['a','b','c'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "name": "01-python-tools-ds-9"
   },
   "outputs": [],
   "source": [
    "np.array([1,2,3,6,8,29])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "But, what if we provide a heterogeneous list?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "name": "01-python-tools-ds-10"
   },
   "outputs": [],
   "source": [
    "y = [1,3,'a']\n",
    "np.array(y)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So what's going on here? Upon conversion from a heterogeneous list, numpy converted the numbers into strings. This is necessary since, by definition, numpy arrays can hold data of a single type. When one of the elements is a string, numpy casts all the other entities into strings as well. Think about what would happen if the opposite rule was used. The string 'a' doesn't have a corresponding number, while both numbers 1 and 3 have corresponding string representations, so going from string to numeric would create all sorts of problems. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> The advantage of numpy arrays is that the data is stored in a contiguous section of memory, and  you can be very efficient with homogeneous arrays in terms of manipulating them, applying functions, etc. However, `numpy` does provide a \"catch-all\" `dtype` called `object`, which can be any Python object. This `dtype` essentially is an array of pointers to actual data stored in different parts of the memory. You can get to the actual objects by extracting them. So one could do  <a name='object'></a>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "name": "01-python-tools-ds-11"
   },
   "outputs": [],
   "source": [
    "np.array([1,3,'a'], dtype='object')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> which would basically be a valid `numpy` array, but would go back to the actual objects when used, much like a list. We can see this later if we want to transform a heterogeneous `pandas` `DataFrame` into a `numpy` array. It's not particularly useful as is, but it prevents errors from popping up during transformations from `pandas` to `numpy`."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Generating data in numpy\n",
    "\n",
    "We had seen earlier how we could generate a sequence of numbers in a list using `range`. In numpy, you can generate a sequence of numbers in an array using `arange` (which actually creates the array rather than provide an iterator like `range`)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "name": "01-python-tools-ds-12"
   },
   "outputs": [],
   "source": [
    "np.arange(10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can also generate regularly spaced sequences of numbers between particular values"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "name": "01-python-tools-ds-13"
   },
   "outputs": [],
   "source": [
    "np.linspace(start=0, stop=1, num=11) # or np.linspace(0, 1, 11)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can also do this with real numbers rather than integers."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "name": "01-python-tools-ds-14"
   },
   "outputs": [],
   "source": [
    "np.linspace(start = 0, stop = 2*np.pi, num = 10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "More generally, you can transform lists into `numpy` arrays. We saw this above for vectors. For matrices, you can provide a list of lists. Note the double `[` in front and back. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "name": "01-python-tools-ds-15"
   },
   "outputs": [],
   "source": [
    "np.array([[1,3,5,6],[4,3,9,7]])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can generate an array of 0's"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "name": "01-python-tools-ds-16"
   },
   "outputs": [],
   "source": [
    "np.zeros(10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This can easily be extended to a two-dimensional array (a matrix), by specifying the dimension of the matrix as a tuple. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "name": "01-python-tools-ds-17"
   },
   "outputs": [],
   "source": [
    "np.zeros((10,10))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can also generate a matrix of 1s in a similar manner."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "name": "01-python-tools-ds-18"
   },
   "outputs": [],
   "source": [
    "np.ones((3,4))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In matrix algebra, the identity matrix is important. It is a square matrix with 1's on the diagonal and 0's everywhere else. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "name": "01-python-tools-ds-19"
   },
   "outputs": [],
   "source": [
    "np.eye(4)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can also create numpy vectors directly from lists, as long as lists are made up of atomic elements of the same type. This means a list of numbers or a list of strings. The elements can't be more composite structures, generally. One exception is a list of lists, where all the lists contain the same type of atomic data, which, as we will see, can be used to create a matrix or 2-dimensional array."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "name": "01-python-tools-ds-20"
   },
   "outputs": [],
   "source": [
    "a = [1,2,3,4,5,6,7,8]\n",
    "b = ['a','b','c','d','3']\n",
    "\n",
    "np.array(a)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "name": "01-python-tools-ds-21"
   },
   "outputs": [],
   "source": [
    "np.array(b)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Random numbers\n",
    "\n",
    "Generating random numbers is quite useful in many areas of data science. All computers don't produce truly random numbers but generate *pseudo-random* sequences. These are completely deterministic sequences defined algorithmically that emulate the properties of random numbers. Since these are deterministic, we can set a *seed* or starting value for the sequence, so that we can exactly reproduce this sequence to help debug our code. To actually see how things behave in simulations we will often run several sequences of random numbers starting at different seed values. \n",
    "\n",
    "The seed is set by the `RandomState` function within the `random` submodule of numpy. Note that all Python names are case-sensitive. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "name": "01-python-tools-ds-22"
   },
   "outputs": [],
   "source": [
    "rng = np.random.RandomState(35) # set seed\n",
    "rng.randint(0, 10, (3,4))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We have created a 3x4 matrix of random integers between 0 and 10 (in line with slicing rules, this includes 0 but not 10). \n",
    "\n",
    "We can also create a random sample of numbers between 0 and 1. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "name": "01-python-tools-ds-23"
   },
   "outputs": [],
   "source": [
    "rng.random_sample((5,2))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We'll see later how to generate random numbers from particular probability distributions. \n",
    "\n",
    "### Vectors and matrices\n",
    "\n",
    "Numpy generates arrays, which can be of arbitrary dimension. However the most useful are vectors (1-d arrays) and matrices (2-d arrays).\n",
    "\n",
    "In these examples, we will generate samples from the Normal (Gaussian) distribution, with mean 0 and variance 1. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "name": "01-python-tools-ds-24"
   },
   "outputs": [],
   "source": [
    "A = rng.normal(0,1,(4,5))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can compute some characteristics of this matrix's dimensions. The number of rows and columns are given by `shape`. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "name": "01-python-tools-ds-25"
   },
   "outputs": [],
   "source": [
    "A.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The total number of elements are given by `size`. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "name": "01-python-tools-ds-26"
   },
   "outputs": [],
   "source": [
    "A.size"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If we want to create a matrix of 0's with the same dimensions as `A`, we don't actually have to compute its dimensions. We can use the `zeros_like` function to figure it out."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "name": "01-python-tools-ds-27"
   },
   "outputs": [],
   "source": [
    "np.zeros_like(A)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can also create vectors by only providing the number of rows to the random sampling function. The number of columns will be assumed to be 1. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "name": "01-python-tools-ds-28"
   },
   "outputs": [],
   "source": [
    "B = rng.normal(0, 1, (4,))\n",
    "B"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Extracting elements from arrays\n",
    "\n",
    "The syntax for extracting elements from arrays is almost exactly the same as for lists, with the same rules for slices.\n",
    "\n",
    "**Exercise:** State what elements of B are extracted by each of the following statements\n",
    "\n",
    "```\n",
    "B[:3]\n",
    "B[:-1]\n",
    "B[[0,2,4]]\n",
    "B[[0,2,5]]\n",
    "```\n",
    "\n",
    "For matrices, we have two dimensions, so you can slice by rows, or columns or both. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "name": "01-python-tools-ds-29"
   },
   "outputs": [],
   "source": [
    "A"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can extract the first column by specifying `:` (meaning everything) for the rows, and the index for the column (reminder, Python starts counting at 0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "name": "01-python-tools-ds-30"
   },
   "outputs": [],
   "source": [
    "A[:,0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Similarly the 4th row can be extracted by putting the row index, and `:` for the column index. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "name": "01-python-tools-ds-31"
   },
   "outputs": [],
   "source": [
    "A[3,:]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "All slicing operations work for rows and columns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "name": "01-python-tools-ds-32"
   },
   "outputs": [],
   "source": [
    "A[:2,:2]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Array operations\n",
    "\n",
    "We can do a variety of vector and matrix operations in `numpy`. \n",
    "\n",
    "First, all usual arithmetic operations work on arrays, like adding or multiplying an array with a scalar."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "name": "01-python-tools-ds-33"
   },
   "outputs": [],
   "source": [
    "A = rng.randn(3,5)\n",
    "A"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "name": "01-python-tools-ds-34"
   },
   "outputs": [],
   "source": [
    "A + 10"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can also add and multiply arrays __element-wise__ as long as they are the same shape."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "name": "01-python-tools-ds-35"
   },
   "outputs": [],
   "source": [
    "B = rng.randint(0,10, (3,5))\n",
    "B"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "name": "01-python-tools-ds-36"
   },
   "outputs": [],
   "source": [
    "A + B"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "name": "01-python-tools-ds-37"
   },
   "outputs": [],
   "source": [
    "A * B"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can also do **matrix multiplication**. Recall what this is.\n",
    "\n",
    "If you have a matrix $A_{m x n}$ and another matrix $B_{n x p}$, as long as the number of columns of $A$ and rows of $B$ are the same, you can multiply them ($C_{m x p} = A_{m x n}B_{n x p}$), with the (i,j)-th element of C being\n",
    "\n",
    "$$ c_{ij} = \\sum_{k=1}^n a_{ik}b_{kj}, i= 1, \\dots, m; j = 1, \\dots, p$$\n",
    "\n",
    "In `numpy` the operant for matrix multiplication is `@`. \n",
    "\n",
    "In the above examples, `A` and `B` cannot be multiplied since they have incompatible dimensions. However, we can take the *transpose* of `B`, i.e. flip the rows and columns, to make it compatible with `A` for matrix multiplication."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "name": "01-python-tools-ds-38"
   },
   "outputs": [],
   "source": [
    "A @ np.transpose(B)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "name": "01-python-tools-ds-39"
   },
   "outputs": [],
   "source": [
    "np.transpose(A) @ B"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "More generally, you can *reshape* a `numpy` array into a new shape, provided it is compatible with the number of elements in the original array."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "name": "01-python-tools-ds-40"
   },
   "outputs": [],
   "source": [
    "D = rng.randint(0,5, (4,4))\n",
    "D"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "name": "01-python-tools-ds-41"
   },
   "outputs": [],
   "source": [
    "D.reshape(8,2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "name": "01-python-tools-ds-42"
   },
   "outputs": [],
   "source": [
    "D.reshape(1,16)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This can also be used to cast a vector into a matrix. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "name": "01-python-tools-ds-43"
   },
   "outputs": [],
   "source": [
    "e = np.arange(20)\n",
    "E = e.reshape(5,4)\n",
    "E"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> One thing to note in all the reshaping operations above is that the new array takes elements of the old array **by row**. See the examples above to convince yourself of that. \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Statistical operations on arrays\n",
    "\n",
    "You can sum all the elements of a matrix using `sum`. You can also sum along rows or along columns by adding an argument to the `sum` function. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "name": "01-python-tools-ds-44"
   },
   "outputs": [],
   "source": [
    "A = rng.normal(0, 1, (4,2))\n",
    "A"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "name": "01-python-tools-ds-45"
   },
   "outputs": [],
   "source": [
    "A.sum()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can sum along rows (i.e., down columns) with the option `axis = 0`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "name": "01-python-tools-ds-46"
   },
   "outputs": [],
   "source": [
    "A.sum(axis=0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can sum along columns (i.e., across rows) with `axis = 1`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "name": "01-python-tools-ds-47"
   },
   "outputs": [],
   "source": [
    "A.sum(axis=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> Of course, you can use the usual function calls: `np.sum(A, axis = 1)`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can also find the minimum and maximum values."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "name": "01-python-tools-ds-48"
   },
   "outputs": [],
   "source": [
    "A.min(axis = 0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "name": "01-python-tools-ds-49"
   },
   "outputs": [],
   "source": [
    "A.max(axis = 0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can also find the **position** where the minimum and maximum values occur."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "name": "01-python-tools-ds-50"
   },
   "outputs": [],
   "source": [
    "A.argmin(axis=0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "name": "01-python-tools-ds-51"
   },
   "outputs": [],
   "source": [
    "A.argmax(axis=0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can sort arrays and also find the indices which will result in the sorted array. I'll demonstrate this for a vector, where it is more relevant"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "name": "01-python-tools-ds-52"
   },
   "outputs": [],
   "source": [
    "a = rng.randint(0,10, 8)\n",
    "a"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "name": "01-python-tools-ds-53"
   },
   "outputs": [],
   "source": [
    "np.sort(a)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "name": "01-python-tools-ds-54"
   },
   "outputs": [],
   "source": [
    "np.argsort(a)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "name": "01-python-tools-ds-55"
   },
   "outputs": [],
   "source": [
    "a[np.argsort(a)]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`np.argsort` can also help you find the 2nd smallest or 3rd largest value in an array, too."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "name": "01-python-tools-ds-56"
   },
   "outputs": [],
   "source": [
    "ind_2nd_smallest = np.argsort(a)[1]\n",
    "a[ind_2nd_smallest]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "name": "01-python-tools-ds-57"
   },
   "outputs": [],
   "source": [
    "ind_3rd_largest = np.argsort(a)[-3]\n",
    "a[ind_3rd_largest]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can also sort strings in this way."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "name": "01-python-tools-ds-58"
   },
   "outputs": [],
   "source": [
    "m = np.array(['Aram','Raymond','Elizabeth','Donald','Harold'])\n",
    "np.sort(m)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you want to sort arrays **in place**, you can use the `sort` function in a different way."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "name": "01-python-tools-ds-59"
   },
   "outputs": [],
   "source": [
    "m.sort()\n",
    "m"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Putting arrays together\n",
    "\n",
    "We can put arrays together by row or column, provided the corresponding axes have compatible lengths. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "name": "01-python-tools-ds-60"
   },
   "outputs": [],
   "source": [
    "A = rng.randint(0,5, (3,5))\n",
    "B = rng.randint(0,5, (3,5))\n",
    "print('A = ', A)\n",
    "print('B = ', B)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "name": "01-python-tools-ds-61"
   },
   "outputs": [],
   "source": [
    "np.hstack((A,B))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "name": "01-python-tools-ds-62"
   },
   "outputs": [],
   "source": [
    "np.vstack((A,B))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note that both `hstack` and `vstack` take a **tuple** of arrays as input."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Logical/Boolean operations\n",
    "\n",
    "You can query a matrix to see which elements meet some criterion. In this example, we'll see which elements are negative."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "name": "01-python-tools-ds-63"
   },
   "outputs": [],
   "source": [
    "A < 0"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This is called **masking**, and is useful in many contexts. \n",
    "\n",
    "We can extract all the negative elements of A using"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "name": "01-python-tools-ds-64"
   },
   "outputs": [],
   "source": [
    "A[A<0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This forms a 1-d array. You can also count the number of elements that meet the criterion"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "name": "01-python-tools-ds-65"
   },
   "outputs": [],
   "source": [
    "np.sum(A<0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Since the entity `A<0` is a matrix as well, we can do row-wise and column-wise operations as well. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Beware of copies\n",
    "\n",
    "One has to be a bit careful with copying objects in Python. By default, if you just assign one object to a new name, it does a *shallow copy*, which means that both names point to the same memory. So if you change something in the original, it also changes in the new copy. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "name": "01-python-tools-ds-66"
   },
   "outputs": [],
   "source": [
    "A[0,:]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "name": "01-python-tools-ds-67"
   },
   "outputs": [],
   "source": [
    "A1 = A\n",
    "A1[0,0] = 4\n",
    "A[0,0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To actually create a copy that is not linked back to the original, you have to make a *deep copy*, which creates a new space in memory and a new pointer, and copies the original object to the new memory location"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "name": "01-python-tools-ds-68"
   },
   "outputs": [],
   "source": [
    "A1 = A.copy()\n",
    "A1[0,0] = 6\n",
    "A[0,0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can also replace sub-matrices of a matrix with new data, provided that the dimensions are compatible. (Make sure that the sub-matrix we are replacing below truly has 2 rows and 2 columns, which is what `np.eye(2)` will produce)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "name": "01-python-tools-ds-69"
   },
   "outputs": [],
   "source": [
    "A[:2,:2] = np.eye(2)\n",
    "A"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Reducing matrix dimensions\n",
    "\n",
    "Sometimes the output of some operation ends up being a matrix of one column or one row. We can reduce it to become a vector. There are two functions that can do that, `flatten` and `ravel`. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "name": "01-python-tools-ds-70"
   },
   "outputs": [],
   "source": [
    "A = rng.randint(0,5, (5,1))\n",
    "A"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "name": "01-python-tools-ds-71"
   },
   "outputs": [],
   "source": [
    "A.flatten()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "name": "01-python-tools-ds-72"
   },
   "outputs": [],
   "source": [
    "A.ravel()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So why two functions? I'm not sure, but they do different things behind the scenes. `flatten` creates a **copy**, i.e. a new array disconnected from `A`. `ravel` creates a **view**, so a representation of the original array. If you then changed a value after a `ravel` operation, you would also change it in the original array; if you did this after a `flatten` operation, you would not. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Broadcasting in Python"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Python deals with arrays in an interesting way, in terms of matching up dimensions of arrays for arithmetic operations. There are 3 rules:\n",
    "\n",
    "1. If two arrays differ in the number of dimensions, the shape of the smaller array is padded with 1s on its _left_ side\n",
    "2. If the shape doesn't match in any dimension, the array with shape = 1 in that dimension is stretched to match the others' shape\n",
    "3. If in any dimension the sizes disagree and none of the sizes are 1, then an error is generated"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "name": "01-python-tools-ds-73"
   },
   "outputs": [],
   "source": [
    "A = rng.normal(0,1,(4,5))\n",
    "B = rng.normal(0,1,5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "name": "01-python-tools-ds-74"
   },
   "outputs": [],
   "source": [
    "A.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "name": "01-python-tools-ds-75"
   },
   "outputs": [],
   "source": [
    "B.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "name": "01-python-tools-ds-76"
   },
   "outputs": [],
   "source": [
    "A - B"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "B is 1-d, A is 2-d, so B's shape is made into (1,5) (added to the left). Then it is repeated into 4 rows to make it's shape (4,5), then the operation is performed. This means that we subtract the first element of B from the first column of A, the second element of B from the second column of A, and so on."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can be explicit about adding dimensions for broadcasting by using `np.newaxis`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "name": "01-python-tools-ds-77"
   },
   "outputs": [],
   "source": [
    "B[np.newaxis,:].shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "name": "01-python-tools-ds-78"
   },
   "outputs": [],
   "source": [
    "B[:,np.newaxis].shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### An example (optional, intermediate/advanced))\n",
    "\n",
    "This can be very useful, since these operations are faster than for loops. For example:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "name": "01-python-tools-ds-79"
   },
   "outputs": [],
   "source": [
    "d = rng.random_sample((10,2))\n",
    "d"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We want to find the Euclidean distance (the sum of squared differences) between the points defined by the rows. This should result in a 10x10 distance matrix"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "name": "01-python-tools-ds-80"
   },
   "outputs": [],
   "source": [
    "d.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "name": "01-python-tools-ds-81"
   },
   "outputs": [],
   "source": [
    "d[np.newaxis,:,:]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "creates a 3-d array with the first dimension being of length 1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "name": "01-python-tools-ds-82"
   },
   "outputs": [],
   "source": [
    "d[np.newaxis,:,:].shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "name": "01-python-tools-ds-83"
   },
   "outputs": [],
   "source": [
    "d[:, np.newaxis,:]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "creates a 3-d array with the 2nd dimension being of length 1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "name": "01-python-tools-ds-84"
   },
   "outputs": [],
   "source": [
    "d[:,np.newaxis,:].shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now for the trick, using broadcasting of arrays. These two arrays are incompatible without broadcasting, but with broadcasting, the right things get repeated to make things compatible"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "name": "01-python-tools-ds-85"
   },
   "outputs": [],
   "source": [
    "dist_sq = np.sum((d[:,np.newaxis,:] - d[np.newaxis,:,:]) ** 2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "name": "01-python-tools-ds-86"
   },
   "outputs": [],
   "source": [
    "dist_sq.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "name": "01-python-tools-ds-87"
   },
   "outputs": [],
   "source": [
    "dist_sq"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Whoops! we wanted a 10x10 matrix, not a scalar. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "name": "01-python-tools-ds-88"
   },
   "outputs": [],
   "source": [
    "(d[:,np.newaxis,:] - d[np.newaxis,:,:]).shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "What we really want is the 10x10 distance matrix. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "name": "01-python-tools-ds-89"
   },
   "outputs": [],
   "source": [
    "dist_sq = np.sum((d[:,np.newaxis,:] - d[np.newaxis,:,:]) ** 2, axis=2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can verify what is happening by creating `D = d[:,np.newaxis,:]-d[np.newaxis,:,:]` and then looking at `D[:,:,0]` and `D[:,:,1]`. These are the difference between each combination in the first and second columns of d, respectively. Squaring and summing along the 3rd axis then gives the sum of squared differences. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "name": "01-python-tools-ds-90"
   },
   "outputs": [],
   "source": [
    "dist_sq"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "name": "01-python-tools-ds-91"
   },
   "outputs": [],
   "source": [
    "dist_sq.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "name": "01-python-tools-ds-92"
   },
   "outputs": [],
   "source": [
    "dist_sq.diagonal()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Conclusions moving forward\n",
    "\n",
    "It's important to understand numpy and arrays, since most data sets we encounter are rectangular. The notations and operations we saw in numpy will translate to data, except for the fact that data is typically heterogeneous, i.e., of different types. The problem with using numpy for modern data analysis is that if you have mixed data types, it will all be coerced to strings, and then you can't actually do any data analysis. \n",
    "\n",
    "The solution to this issue (which is also present in Matlab) came about with the `pandas` package, which is the main workhorse of data science in Python\n"
   ]
  }
 ],
 "metadata": {
  "jupytext": {
   "formats": "ipynb,Rmd"
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}