sample_project_workflow.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "import requests\n",
    "from bs4 import BeautifulSoup\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Project workflow"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "An important component of this course is the project, which will integrate all of the topics we've learned over the past few weeks. Once we've built some basic Python skills, we can get started -- but there is still a lot on the practical side to learn.\n",
    "\n",
    "The goal of this notebook is to provide a brief overview of resources that might help you on your project. Each project will have specific needs, and you might need to read different tutorials to achieve your goals. Here, I try to provide pointers to some useful resources.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1. Make a project directory\n",
    "\n",
    "See notebooks:\n",
    "\n",
    "- [01-Introduction_to_Python/L2-Folders.ipynb](https://github.com/khof312/Summer2018_ProfHoffmannPham/blob/master/01-Introduction_to_Python/L2-Folders.ipynb)\n",
    "- [12-UNIX_Basics/A-Basic_Unix_Shell_Commands.ipynb](https://github.com/khof312/Summer2018_ProfHoffmannPham/blob/master/12-UNIX_Basics/A-Basic_Unix_Shell_Commands.ipynb)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's start by making a project directory. You can do this manually, or using command line tools. To call the command line from within Python, you prefix your command with a`!`. We'll see three simple examples:\n",
    "    \n",
    "    pwd                    <- Print working directory\n",
    "    mkdir [DIRECTORYNAME]  <- Make a new directory in the working directory\n",
    "    ls    [DIRECTORYNAME]  <- List the contents of a directory; lists current working directory if blank"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Let's print our working directory using \"pwd\"\n",
    "!pwd"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Let's make a new directory called \"project\" with the \"mkdir\" command\n",
    "!mkdir project"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Let's list what's in our current working directory\n",
    "# ...we should see the \"project\" folder\n",
    "!ls"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Using the \"ls\" command, let's list what is in the folder\n",
    "# ...nothing, for now\n",
    "!ls project"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2. Download some data\n",
    "See notebooks:\n",
    "\n",
    "- [12-UNIX_Basics/B-Fetching_Data_Using_CURL.ipynb](https://github.com/khof312/Summer2018_ProfHoffmannPham/blob/master/12-UNIX_Basics/B-Fetching_Data_Using_CURL.ipynb)\n",
    "\n",
    "### a. Download a file"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It's time to get some data! Let's download a CSV that I've posted on my website. We will use two more commands:\n",
    "\n",
    "    curl [URL] -o [DIRECTORYNAME/FILENAME]   -> Save a URL's contents to a chosen directory and file\n",
    "    head -5       [DIRECTORYNAME/FILENAME]   -> Print the first 5 lines of a file\n",
    "    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Use the \"curl\" command to download a URL\n",
    "# Specify the -o option to name the output file\n",
    "!curl \"http://people.stern.nyu.edu/khoffman/intro_programming_datasci/assets/csv/trains.csv\" -o project/trains.csv"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Using the \"ls\" command, let's list what is in the folder\n",
    "# ...we got the new file!\n",
    "!ls project"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Using the \"head\" command, let's inspect what's in the file\n",
    "!head -5 project/trains.csv"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3. Get data in and out of Python\n",
    "See notebooks:\n",
    "- [03-Pandas](https://github.com/khof312/Summer2018_ProfHoffmannPham/tree/master/03-Pandas)\n",
    "\n",
    "\n",
    "### a. Read in to Pandas"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `pandas` library is a very convenient tool for working with data. We will explore it in depth later. For now, let's just see an example of what it can do."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Read our csv file in\n",
    "trains = pd.read_csv(\"project/trains.csv\", index_col=['route_id'])\n",
    "trains"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Select a row\n",
    "trains.loc['A']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Select a column\n",
    "trains[['route_long_name']]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Add a new column\n",
    "trains['age'] = 2018 - trains['line_introduced']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Plot \n",
    "trains.plot(kind = 'scatter', \n",
    "            x ='line_introduced', \n",
    "            y ='age')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### b. Write to csv"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "trains.to_csv(\"project/trains_with_age_column.csv\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4. Query an API\n",
    "\n",
    "See notebooks:\n",
    "- [05-APIs](https://github.com/khof312/Summer2018_ProfHoffmannPham/tree/master/04-WebAPIs)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Query via URL"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Instead of downloading a file directly, we might want to query an API directly.\n",
    "\n",
    "Let's try to query OpenWeatherMap now, to get data about the weather. [Documentation](http://openweathermap.org/current#geo). Below you can find the URL that you can copy and paste in your browser, to get the weather for New York. You will notice that it contains parameters as part of the URL, including an `appid` which is a key that is used to limit the number of calls that can be issued by a single application. \n",
    "\n",
    "    http://api.openweathermap.org/data/2.5/weather?&appid=ffb7b9808e07c9135bdcc7d1e867253d&q=New%20York,NY,USA&mode=json \n",
    "    \n",
    "Try the URL in your browser. Also try to change the query parameter `q` and change it from `New%20York,NY` to something different. (Note: The `%20` is a transformation for the space (` `) character in URLs.)\n",
    "\n",
    "In Python, we often query such URLS using the `requests` library.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import requests\n",
    "\n",
    "openweathermap_url = \"http://api.openweathermap.org/data/2.5/weather\"\n",
    "parameters = {\n",
    "    'q'     : 'New York, NY, USA',\n",
    "    'units' : 'imperial',\n",
    "    'mode'  : 'json',\n",
    "    'appid' : 'ffb7b9808e07c9135bdcc7d1e867253d'\n",
    "}\n",
    "resp = requests.get(openweathermap_url, params=parameters)\n",
    "data = resp.json()\n",
    "data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here is a simple example that queries the API for five boroughs."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "for borough in ['Brooklyn', 'Queens', 'Bronx', 'Manhattan', 'Staten Island']:\n",
    "    # Put borough into the query\n",
    "    parameters = {'q'     : borough + ', NY, USA',\n",
    "            'units' : 'imperial',\n",
    "            'mode'  : 'json',\n",
    "            'appid' : 'ffb7b9808e07c9135bdcc7d1e867253d'\n",
    "            }\n",
    "    resp = requests.get(openweathermap_url, params = parameters)\n",
    "    data = resp.json()\n",
    "    \n",
    "    # Print the result\n",
    "    print(f\"The temperature in {borough} is {data['main']['temp']}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's do the same thing, but store it in a file:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Let's open the file and write a row of headers\n",
    "with open(\"project/weather.csv\", \"w\") as f:\n",
    "    f.write(\"borough, temperature\\n\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Now, let's loop through the tweets and write them to file\n",
    "with open(\"project/weather.csv\", \"a\") as f:\n",
    "    \n",
    "    for borough in ['Brooklyn', 'Queens', 'Bronx', 'Manhattan', 'Staten Island']:\n",
    "        parameters = {'q'     : borough + ', NY, USA',\n",
    "                'units' : 'imperial',\n",
    "                'mode'  : 'json',\n",
    "                'appid' : 'ffb7b9808e07c9135bdcc7d1e867253d'\n",
    "                }\n",
    "        resp = requests.get(openweathermap_url, params = parameters)\n",
    "        data = resp.json()\n",
    "\n",
    "        f.write(f\"{borough},{data['main']['temp']}\\n\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Did it work?\n",
    "!ls project"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!head -5 project/weather.csv"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Query via Python package"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here is a simple example using the Twitter API and the `tweepy` package.\n",
    "`tweepy` makes it easy to access tweets and their contents in Python.\n",
    "\n",
    "First, we need to authenticate ourselves:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "import tweepy"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# YOUR APPLICATION CREDENTIALS HERE\n",
    "consumer_key = 'YOUR KEY'\n",
    "consumer_secret = 'YOUR SECRET'\n",
    "\n",
    "access_token = 'YOUR TOKEN'\n",
    "access_secret = 'YOUR ACCESS SECRET'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Then you need to supply your credentials to tweepy\n",
    "auth = tweepy.OAuthHandler(consumer_key, consumer_secret)\n",
    "auth.set_access_token(access_token, access_secret)\n",
    "api = tweepy.API(auth)\n",
    "\n",
    "# If the authentication was successful, you should see the name of the account print out\n",
    "print (\"My name is\", api.me().name)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Let's get a tweet\n",
    "api.search(q='from:NYCTSubway', count=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# What a mess! Let's print the text, created date, and the author\n",
    "api.search(q='from:NYCTSubway', count=1)[0].text"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "api.search(q='from:NYCTSubway', count=1)[0].author.name"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "api.search(q='from:NYCTSubway', count=1)[0].created_at"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The specific syntax for API calls, and accessing results, will vary by API and by library. When starting out, it's often best to search for online examples of how others have used the library, and then modify or simplify their code to achieve your task.\n",
    "\n",
    "For now, let's just complete our example by pulling 5 tweets from the subway and storing them in a CSV."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Let's open the file and write a row of headers\n",
    "with open(\"project/mta_tweets.csv\", \"w\") as f:\n",
    "    f.write(\"author, created_at, text\\n\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Now, let's loop through the tweets and write them to file\n",
    "with open(\"project/mta_tweets.csv\", \"a\") as f:\n",
    "    for t in api.search(q='from:NYCTSubway', count=50):\n",
    "        if t.text[0]!=\"@\": # Let's make a small filter to avoid direct tweets\n",
    "            f.write(\",\".join([t.author.name, str(t.created_at), t.text]) + '\\n')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "# Did it work?\n",
    "!ls project\n",
    "!head -5 project/mta_tweets.csv"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 5. Web Scraping"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finally, we can scrape a web page. This is conceptually the most difficult, and almost guaranteed to be the most time consuming.\n",
    "\n",
    "[**_BeautifulSoup_**](http://www.crummy.com/software/BeautifulSoup/bs4/doc/) is an incredible Python tool (open library) for pulling out information from a webpage. You can use it to extract tables, lists, paragraph and you can also put filters to extract information from web pages.\n",
    "\n",
    "For more information on Web Scraping, see:\n",
    "- [06-Web Scraping](https://github.com/khof312/Summer2018_ProfHoffmannPham/tree/master/06-Web_Scraping)\n"
   ]
  }
 ],
 "metadata": {
  "celltoolbar": "Raw Cell Format",
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}