selections-pandas-dataframes.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Selections From Pandas Dataframes\n",
    "\n",
    "In the lesson introducing `pandas dataframes`, you learned that these data structures have an inherent tabular structure (i.e. rows and columns with header names) that support selecting data with indexing, such as selecting individual cells identified by their location at the intersection of rows and columns. \n",
    "\n",
    "You can also select data from `pandas dataframes` without knowing the location of that data within the `pandas dataframe`, using specific values such as a column name or data value.\n",
    "\n",
    "In this lesson, you will review how indexing works for `pandas dataframes` and you will learn how to select data from `pandas dataframes` using specific values as well as indexing.\n",
    "\n",
    "\n",
    "## Indexing For Pandas Dataframes\n",
    "\n",
    "There are two kinds of indexing in `pandas dataframes`: location-based and label-based. \n",
    "\n",
    "After working with indexing for `Python` lists and `numpy arrays`, you are familiar with location-based indexing. \n",
    "\n",
    "You already know that `Python` location-based indexing begins with `[0]`, and you have learned how to use location-based indexing to query data within `Python` lists or `numpy arrays`.\n",
    "\n",
    "You can also use location-based indexing to query `pandas dataframes` using `.iloc` and providing the row and column selection as ranges (i.e. start and stop locations along the rows and columns). \n",
    "\n",
    "The range provided is inclusive of the first value, but not the second value. This means that you need to use the range `[0:1]` to select the first index, so your selection begins at `[0]` but does not include `[1]`. \n",
    "\n",
    "For example, you can select the first row and the first column of a `pandas dataframes` providing the range `[0:1]` for the row selection and then providing the range `[0:1]` for the column selection.\n",
    "\n",
    "```python\n",
    "dataframe.iloc[0:1, 0:1]\n",
    "```\n",
    "In addition to location-based indexing, `pandas dataframes` can also be queried using label-based indexing. This feature of `pandas dataframes` is very useful because you can create an index for `pandas dataframes` based on a list of values that you want to use for organizing and querying your data.\n",
    "\n",
    "For example, you can create an index from a specific column of values, and then use `.loc` to select data from the `pandas dataframes` using a value that is found in that index. \n",
    "\n",
    "```python\n",
    "\n",
    "dataframe.setindex(\"column\")\n",
    "dataframe.loc[\"value\"]\n",
    "```\n",
    "In this lesson, you will use both location and label indexing to select data from `pandas dataframes`.\n",
    "\n",
    "## Begin Writing Your Code\n",
    "\n",
    "By now, you are familiar with importing the necessary `Python` packages to set your working directory and download the needed datasets using the `os` and `urllib` packages. You also need to import the `pandas` package with the alias `pd`.\n",
    "\n",
    "Begin by reviewing these tasks.\n",
    "\n",
    "### Import Packages"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# import necessary Python packages\n",
    "import os\n",
    "import urllib.request\n",
    "import pandas as pd\n",
    "\n",
    "# print message after packages imported successfully\n",
    "print(\"import of packages successful\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Set Working Directory\n",
    "\n",
    "Remember that you can check the current working directory using `os.getcwd()` and set the current working directory using `os.chdir()`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# set the working directory to the `earth-analytics-bootcamp` directory\n",
    "# replace `jpalomino` with your username here and all paths in this lesson\n",
    "os.chdir(\"/home/jpalomino/earth-analytics-bootcamp/\")\n",
    "\n",
    "#For Windows users, you may have to set the directory using the full path that Windows maintains for your user. For example:\n",
    "#os.chdir(\"C:/Users/Jenny Palomino/earth-analytics-bootcamp/\")\n",
    "\n",
    "#For Mac Users, you may have to set the directory using the full path that Mac maintains for your user. For example:\n",
    "#os.chdir(\"/Users/jpalomino/earth-analytics-bootcamp/\")\n",
    "\n",
    "# print the current working directory\n",
    "os.getcwd()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Download Data\n",
    "\n",
    "Recall that you can use the `urllib` package to download data from the Earth Lab `Figshare.com` repository.\n",
    "\n",
    "For this lesson, you will use the files from the prevous lesson: a .csv file containing the average monthly precipitation data for Boulder, CO, and another .csv file containing monthly precipitation for Boulder, CO in 2002 and 2013."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# use `urllib` download files from Earth Lab figshare repository\n",
    "\n",
    "# download .csv containing monthly average precipitation for Boulder, CO\n",
    "urllib.request.urlretrieve(url = \"https://ndownloader.figshare.com/files/12710618\", \n",
    "                           filename = \"data/avg-precip-months-seasons.csv\")\n",
    "\n",
    "# download .csv containing monthly precipitation for Boulder, CO in 2002 and 2013\n",
    "urllib.request.urlretrieve(url = \"https://ndownloader.figshare.com/files/12710621\", \n",
    "                           filename = \"data/precip-2002-2013-months-seasons.csv\")\n",
    "\n",
    "# print message that data downloads were successful\n",
    "print(\"datasets downloaded successfully\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Import Tabular Data Into Pandas Dataframes\n",
    "\n",
    "You also know how to import CSV files into `pandas dataframes`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# import the monthly average precipitation values as a pandas dataframe\n",
    "avg_precip = pd.read_csv(\"/home/jpalomino/earth-analytics-bootcamp/data/avg-precip-months-seasons.csv\")\n",
    "\n",
    "# import the monthly precipitation values in 2002 and 2013 as a pandas dataframe\n",
    "precip_2002_2013 = pd.read_csv(\"/home/jpalomino/earth-analytics-bootcamp/data/precip-2002-2013-months-seasons.csv\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Selections Using Location Index (.iloc)\n",
    "\n",
    "You can use `.iloc` to select individual rows and columns or a series of rows and columns by providing the range (i.e. start and stop locations along the rows and columns) that you want to select.\n",
    "\n",
    "Recall that in `Python` indexing begins with `[0]` and that the range you provide is inclusive of the first value, but not the second value.\n",
    "\n",
    "This means that you can use `dataframe.iloc[0:1, 0:1]` to select the cell value at the intersection of the first row and first column of the dataframe."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# select first row and first column\n",
    "avg_precip.iloc[0:1, 0:1]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can expand the range for either the row index or column index to select more data.\n",
    "\n",
    "For example, you can select the first two rows of the first column using `dataframe.iloc[0:2, 0:1]` or the first columns of the first row using `dataframe.iloc[0:1, 0:2]`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# select first two rows and first column\n",
    "avg_precip.iloc[0:2, 0:1]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# select first row and first two columns\n",
    "avg_precip.iloc[0:1, 0:2]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can also use `iloc` to select an entire row or an entire column by leaving the other range without values.\n",
    "\n",
    "For example, you can use `dataframe.iloc[0:1, :]` to select the first row of a dataframe and all of the columns, or `dataframe.iloc[ :, 0:1]` to select the first column of a dataframe and all of the rows. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# select first row with all columns\n",
    "avg_precip.iloc[0:1, :]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# select first column with all rows\n",
    "avg_precip.iloc[:, 0:1]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Selections Using Specific Criteria Values\n",
    "\n",
    "In addition to location-based indexing, you can also select data from `pandas dataframes` using specific values. \n",
    "\n",
    "For example, you can select all data from a specific column in `pandas dataframes` using `dataframe.[[\"columnname\"]]`. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "avg_precip[[\"months\"]]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Notice that your results are also a `pandas dataframe`.\n",
    "\n",
    "You can also select all data from multiple columns in `pandas dataframes` using `dataframe.[[\"columnname\", \"columnname\"]]`.  As the results of your selection are also a `pandas dataframe`, you can assign the results to a new `pandas dataframe`.\n",
    "\n",
    "For example, you can create a new `pandas dataframe` that only contains the `months` and `seasons` columns, effectively dropping the `precip` values. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "avg_precip_text = avg_precip[['months', 'seasons']]\n",
    "\n",
    "avg_precip_text"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you call your original `pandas dataframe`, you will see it is unchanged."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "avg_precip"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can also select data based on specific values within a column using `dataframe[dataframe.columnname == \"value\"]`. This will return all rows containing that value within the specified column. \n",
    "\n",
    "If you are selecting data using a text string column, you need to provide the value within parentheses (e.g. `\"textvalue\"`). \n",
    "\n",
    "For example, you can select all rows that have a `seasons` value of `Summer`. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "avg_precip[avg_precip.seasons == \"Summer\"]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can also select data based on numeric values; these selections do not require the use of parentheses. \n",
    "\n",
    "For example, you can select all rows that have a specific value in `precip` such as `1.62`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "avg_precip[avg_precip.precip == 1.62]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Selections Using Label Index (.loc)\n",
    "\n",
    "In addition to selecting data based on specific values, you can also create new index based on a list of values that you want to use for organizing and querying your data. \n",
    "\n",
    "For example, you can create an index from a specific column of values using `dataframe.setindex(\"column\")`.\n",
    "\n",
    "<i class=\"fa fa-star\" aria-hidden=\"true\"></i> **Data Tip:** Creating a new index will restructure the data by replacing the default location indexing (i.e. `[0]`) with the new index.  This also means the column used to create the index is no longer a functional column, but rather an index of the dataframe.   \n",
    "{: .notice--success}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# create index using the values in the column `months`\n",
    "avg_precip = avg_precip.set_index(\"months\")\n",
    "\n",
    "# print the data in precip_df to see the new structure\n",
    "avg_precip"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Test that `months` no longer functions as a column by atempting to select all data using that column name, as you did previously when `months` was still a column. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Uncomment and run the line below to see the error message that appears\n",
    "# avg_precip[[\"months\"]]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Notice that the error message indicates that the value `months` is not in the index. This is because `months` is actually now the index!\n",
    "\n",
    "After setting an index, you can use `.loc` to select data from the `pandas dataframes` using a value that is found in that index. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# select data for `Aug` using the new index `months` \n",
    "avg_precip.loc[[\"Aug\"]]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Congratulations! You have now learned how to select data from `pandas dataframes` using specific values as well as location-based and label-based indexing. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Optional Challenge \n",
    "\n",
    "Test your `Python` skills to:\n",
    "\n",
    "1. Use the `.describe()` method to summarize the precipitation values in `precip_2002_2013`. Note the maximum values in 2002 and 2013. \n",
    "\n",
    "2. Use indexing to create two new dataframes:\n",
    "    * one containing the month with the maximum value in 2002\n",
    "    * one containing the month with the maximum value in 2013\n",
    "\n",
    "3. Compare these new dataframes. \n",
    "    * Do they occur in the same season?\n",
    "    * What do you notice about the precipitation value for the maximum month in 2013 (`Sept`), as compared to that same month in 2002? \n"
   ]
  }
 ],
 "metadata": {
  "celltoolbar": "Tags",
  "kernelspec": {
   "display_name": "Python [default]",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}