data_collection.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Speech Scraper\n",
    "Data scientists often need to find creative ways to obtain data relevant for an analysis. Webscraping is a common method data scientists use to get web data.\n",
    "\n",
    "Here, we are going to obtain the Secretary of Defense's public speeches from 2014 through the present. These speeches are available [online here](https://www.defense.gov/News/Speeches/Customspeechwho/16001/) but there are over 200 of them. So, we will build a quick scraper to collect them.\n",
    "\n",
    "First, let's import a few key packages:\n",
    "\n",
    "1. `requests`: this allows us to make requests to webpages\n",
    "2. `BeautifulSoup`: this is a handy tool for parsing websites\n",
    "3. `pandas`: this allows us to manipulate tabular data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import requests\n",
    "from bs4 import BeautifulSoup\n",
    "import pandas as pd"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we can define two functions for processing the web data. The first gets links to speeches from the main page"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_links(soup):\n",
    "    links = []\n",
    "    for div in soup.findAll('div', {'class': 'item'}):\n",
    "        for a in div.findAll('a'):\n",
    "            links.append(a['href'])\n",
    "    print(str(len(links)) + \" links were found\")\n",
    "    return links"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The second function, `process_speech`, parses the speech and transforms it into something we can use."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "def process_speech(link, soup):\n",
    "    url = link\n",
    "    title = soup.find('div', {'class': 'article-body'}).find('h1').text\n",
    "    date = soup.find('time').text\n",
    "    \n",
    "    body = soup.find('div', {'class': 'article-body'}).findAll('p', {'class': None})\n",
    "    speech = ''\n",
    "    \n",
    "    for p in body:\n",
    "        stripped = p.text.strip() + ' '\n",
    "        speech += stripped\n",
    "        \n",
    "    speech_object = {'url': url,\n",
    "                     'title': title,\n",
    "                     'date': date,\n",
    "                     'speech': speech}\n",
    "    return speech_object"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, we can obtain links to each of the respective speeches:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "speech_links = []\n",
    "base = 'https://www.defense.gov/News/Speeches/Customspeechwho/16001/'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "30 links were found\n",
      "30 links were found\n",
      "30 links were found\n",
      "30 links were found\n",
      "30 links were found\n",
      "30 links were found\n",
      "30 links were found\n",
      "15 links were found\n"
     ]
    }
   ],
   "source": [
    "for i in range(1,9):\n",
    "    if i == 1:\n",
    "        url = base\n",
    "    else:\n",
    "        url = '{0}?Page={1}'.format(base, i)\n",
    "    \n",
    "    response = requests.get(url)\n",
    "    soup = BeautifulSoup(response.text, \"html.parser\")\n",
    "    links = get_links(soup)\n",
    "    speech_links += links"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "speech_links = list(set(speech_links))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "In total, 225 speeches were found.\n"
     ]
    }
   ],
   "source": [
    "print('In total, {} speeches were found.'.format(len(speech_links)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that we have links to the speeches, we can go ahead and obtain the speeches themselves and save them to `.csv`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "speeches = []\n",
    "for link in list(set(speech_links)):\n",
    "    response = requests.get(link)\n",
    "    soup = BeautifulSoup(response.text, \"html.parser\")\n",
    "    scraped_speech = process_speech(link, soup)\n",
    "    speeches.append(scraped_speech)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "df = pd.DataFrame.from_records(speeches)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's make sure that the speeches are at least 1000 characters in length. Otherwise, it might be junk data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "df = df[df.speech.str.len() > 1000 ]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, we can see that we collected 204 speeches which meet the criteria."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(204, 4)"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finally, we can save these speeches as a `.csv` file for future use!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "df.to_csv('SecDef_Speeches.csv', index=False)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}