step2.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# ODI Queensland workshop - Web Scraping \n",
    "\n",
    "## QUT DMRC - 2015\n",
    "\n",
    "###  Extract all road sign names on a single page\n",
    "\n",
    "This notebook extends the previous step to get all of the sign names from a single page."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "import bs4\n",
    "import requests"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# this is the base_url\n",
    "base_url = \"http://www.qld.gov.au/transport/safety/signs/\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# select which page to scrape based on the type of road sign\n",
    "sign_type = \"regulatory\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# build the url\n",
    "thepage = base_url + sign_type + '/'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# call the url\n",
    "stuff = requests.get(thepage)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# transform to soup using lxml parser\n",
    "soup = bs4.BeautifulSoup(stuff.text, \"lxml\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# find the table with the signs - it is the first table on the page\n",
    "signs_table = soup.find('table')\n",
    "\n",
    "# extract all the rows from the table\n",
    "lotsofitems = signs_table.findAll('tr')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now process ```lotsofitems``` in a new way to get all the items instead of just one."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# now let's do the same thing as in the previous step but for all items in the page\n",
    "\n",
    "thelist = []\n",
    "\n",
    "for an_item in lotsofitems: \n",
    "    \n",
    "    if an_item.find('th'):        \n",
    "        print('skipping the header row')\n",
    "    else:\n",
    "        # extract the name of the sign inside the <strong></strong> tags\n",
    "        theitem = an_item.find(\"strong\").get_text()\n",
    "    \n",
    "        # add the item to a list\n",
    "        thelist += [theitem]\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "thelist"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Some of these have extra spaces in them - lets tidy that up."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "thelist = []\n",
    "\n",
    "for an_item in lotsofitems: \n",
    "    \n",
    "    if an_item.find('th'):        \n",
    "        print('skipping the header row')\n",
    "    else:\n",
    "        # extract and clean up the sign name\n",
    "        temptemp = an_item.find(\"strong\").get_text()\n",
    "        temptemp = temptemp.split()\n",
    "        theitem = \" \".join(temptemp)\n",
    "\n",
    "    # add the item to a list\n",
    "    thelist += [theitem]\n",
    "\n",
    "thelist"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# alternative way of skipping the header row\n",
    "thelist = []\n",
    "\n",
    "for an_item in lotsofitems[1:]:    \n",
    "    # extract and clean up the sign name\n",
    "    temptemp = an_item.find(\"strong\").get_text()\n",
    "    temptemp = temptemp.split()\n",
    "    theitem = \" \".join(temptemp)\n",
    "    # add the item to a list\n",
    "    thelist += [theitem]\n",
    "\n",
    "thelist"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "But if we check the page, the 'No pedestrians sign' is not the last on the page - there are actually 3 tables of signs on the page.\n",
    "\n",
    "So we need to find and process the extra tables.\n",
    "\n",
    "By finding all the tables on the page, we need to be more careful that we only process the ones that have signs in them - to do this we check that the table header is what we expect."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# find all the tables on the page\n",
    "tables = soup.findAll('table')\n",
    "\n",
    "# move outside the loop so we just make one list\n",
    "thelist = []\n",
    "\n",
    "for table in tables:\n",
    "    # find all the table rows\n",
    "    lotsofitems = table.findAll('tr')\n",
    "    \n",
    "    # check if the first row contains a 'th' elements (table header)\n",
    "    if lotsofitems[0].find('th'): \n",
    "\n",
    "        # get all header elements\n",
    "        temp = lotsofitems[0].findAll('th')\n",
    "        \n",
    "        # check that the table header has the text we expect for the table\n",
    "        if temp[0].get_text() == 'Sign' and temp[1].get_text() == 'Meaning':\n",
    "            \n",
    "            print('Traffic sign table found')\n",
    "            \n",
    "            # process the table of traffic signs\n",
    "            for an_item in lotsofitems[1:]:    \n",
    "                # extract and clean up the sign name\n",
    "                temptemp = an_item.find(\"strong\").get_text()\n",
    "                temptemp = temptemp.split()\n",
    "                theitem = \" \".join(temptemp)\n",
    "                thelist += [theitem]            \n",
    "        else:\n",
    "            print('Different table - with header row:', temp)\n",
    "    else:\n",
    "        print('Different table - no header row:', lotsofitems[0])\n",
    "    \n",
    "thelist"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we are ready to move onto the third notebook - [Extract all traffic sign data from a single page](step3.ipynb)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.4.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}