classify_the_disputed_federalist_papers.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true,
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Supervised Text Classification\n",
    "\n",
    "### with the disputed Federalist papers\n",
    "\n",
    "\n",
    "<img src=\"http://teachingamericanhistory.org/wp-content/themes/tah-main/images/imported/ratification/federalist10.jpg\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true,
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### What is supervised text classification?\n",
    "\n",
    "Email Subject | True Label\n",
    "--|--\n",
    "Your Mystery Tuesday Deal has arrived | Spam\n",
    "End-of-Season Clearance :: 3,500+ Items on Sale! | Spam\n",
    "Metis pre-class Info/Meetup sessions | **Not Spam**\n",
    "The Best Content Marketers in the World | ?\n",
    "\n",
    "\n",
    "A model is [ text $\\rightarrow$ label ]\n",
    "---- "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true,
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Process\n",
    "\n",
    "1. Find the data and parse into sets of [document, label]\n",
    "2. Train a model on the labeled documents, preferably with cross validation. Select a model that performs best\n",
    "3. Run the model on the documents that don't have labels yet"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true,
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Motivating Problem\n",
    "\n",
    "<img src=\"http://tenthamendmentcenter.com/wp-content/uploads/2012/08/federalist-papers.jpg\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true,
    "slideshow": {
     "slide_type": "notes"
    }
   },
   "source": [
    "85 Papers written in Alexander Hamilton, James Madison, and John Jay in 1787 \n",
    "\n",
    "Published by prominent New York Newspapers under a pseudonym\n",
    "\n",
    "12 Papers known as \"the disupted federalist papers\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true,
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Solved by Wallace and Mosteller in 1959-1963, using the IBM 7090:\n",
    "\n",
    "\n",
    "<img src=\"https://pix-media.priceonomics-media.com/blog/1252/IBM_7090_computer.jpg\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true,
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Process\n",
    "\n",
    "1. **Find the data and parse into sets of [document, label]**\n",
    "2. Train a model on the labeled documents, preferably with cross validation. Select a model that performs best\n",
    "3. Run the model on the documents that don't have labels yet"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true,
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "A quick google search reveals that we can find the federalist papers from the gutenberg project\n",
    "\n",
    "We note that the URL is http://www.gutenberg.org/cache/epub/18/pg18.txt\n",
    "\n",
    "Everything is plain text, so we need to clean this to be sets of (text, author) \n",
    "\n",
    "e.g.\n",
    "\n",
    "\"Among the numerous advantages promised by a well constructed Union..\" $\\rightarrow$ \"Hamilton\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true,
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Stored 'fed_text_raw_string' (str)\n"
     ]
    }
   ],
   "source": [
    "# packages for grabbing web data\n",
    "import re\n",
    "import urllib\n",
    "import sys\n",
    "\n",
    "# Read in raw data\n",
    "location_of_federalist_papers = urllib.urlopen(\"http://www.gutenberg.org/cache/epub/18/pg18.txt\")\n",
    "fed_text_raw_string = location_of_federalist_papers.read()\n",
    "location_of_federalist_papers.close()\n",
    "\n",
    "# Save for future use\n",
    "%store fed_text_raw_string"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true,
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'\\xef\\xbb\\xbfThe Project Gutenberg EBook of The Federalist Papers, by \\r\\nAlexander Hamilton and John Jay and James Madison\\r\\n\\r\\nThis eBook is for the use of anyone anywhere at no cost and with\\r\\nalmost no restrictions whatsoever.  You may copy it, give it away or\\r\\nre-use it under the terms of the Project Gutenberg License included\\r\\nwith this eBook or online at www.gutenberg.net\\r\\n\\r\\n\\r\\nTitle: The Federalist Papers\\r\\n\\r\\nAuthor: Alexander Hamilton\\r\\n        John Jay\\r\\n        James Madison\\r\\n\\r\\nPosting Date: December 12'"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Let's see what it looks like:\n",
    "fed_text_raw_string[:500]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true,
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "# Trial and error regex with https://regex101.com/r/fKSaBx/1\n",
    "regex_string = '(?xs)(FEDERALIST[. ]+No\\.\\s[0-9].*?)(?=(?:FEDERALIST)|(?:End\\ of\\ the\\ Project\\ Gutenberg))'\n",
    "pattern = re.compile(regex_string)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true,
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "from nltk import tokenize \n",
    "# tokenize takes in the regex to break apart\n",
    "list_o_papers = tokenize.regexp_tokenize(fed_text_raw_string, pattern)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true,
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "86"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# How many papers?\n",
    "len(list_o_papers)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true,
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'FEDERALIST No. 5\\r\\n\\r\\n\\r\\n\\r\\nThe Same Subject Continued\\r\\n\\r\\n(Concerning Dangers From Foreign Force and Influence)\\r\\n\\r\\nFor the Independent Journal.\\r\\n\\r\\n\\r\\n\\r\\nJAY\\r\\n\\r\\n\\r\\n\\r\\nTo the People of the State of New York:\\r\\n\\r\\nQUEEN ANNE, in her letter of the 1st July, 1706, to the Scotch\\r\\nParliament, makes some observations on the importance of the UNION\\r\\nthen forming between England and Scotland, which merit our attention.\\r\\nI shall present the public with one or two extracts from it: \"An\\r\\nentire and perfect union will '"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "list_o_papers[4][:500]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true,
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "list_o_papers_no_returns = [re.sub(\"\\r|\\n\", \" \", essay) for essay in list_o_papers]\n",
    "# More like turn all whitespace into a single space\n",
    "list_o_papers_no_spaces = [re.sub(\"\\s+\", \" \", essay) for essay in list_o_papers]\n",
    "list_o_papers_cleaned = list_o_papers_no_spaces"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true,
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "# make list of authors\n",
    "author_search = \"(HAMILTON|JAY|MADISON)(\\s(AND|OR)\\s(MADISON))?\"\n",
    "fed_author_list = [re.search(author_search,essay).group() for essay in list_o_papers_cleaned]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true,
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['HAMILTON',\n",
       " 'JAY',\n",
       " 'JAY',\n",
       " 'JAY',\n",
       " 'JAY',\n",
       " 'HAMILTON',\n",
       " 'HAMILTON',\n",
       " 'HAMILTON',\n",
       " 'HAMILTON',\n",
       " 'MADISON']"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Print out the first 10 to make sure it worked\n",
    "fed_author_list[:10]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true,
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'HAMILTON', 'HAMILTON AND MADISON', 'HAMILTON OR MADISON', 'JAY', 'MADISON'}"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#unique authors: \n",
    "set(fed_author_list)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true,
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "# keep just the text of the essays\n",
    "pattern = re.compile(r'''\n",
    "             (To\\ the\\ People\\ of\\ the\\ State\\ of\\ New\\ York)\n",
    "             .*?                              # anything non-greedy\n",
    "          $                                # end of string\n",
    "             ''', re.VERBOSE)\n",
    "for i in range(len(list_o_papers_cleaned)):\n",
    "    text_search = re.search(pattern, list_o_papers_cleaned[i])\n",
    "    list_o_papers_cleaned[i] = text_search.group()\n",
    "\n",
    "\n",
    "# lowercase everything\n",
    "list_o_papers_cleaned = [essay.lower() for essay in list_o_papers_cleaned]\n",
    "\n",
    "# remove most punctuation \n",
    "list_o_papers_cleanedNoPunct = [re.sub(\"[.?!:;,()`'*]|(--)|\\[|\\]\", \"\", essay) for essay in list_o_papers_cleaned]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true,
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'to the people of the state of new york the three last numbers of this paper have been dedicated to an enumeration of the dangers to which we should be exposed in a state of disunion from the arms and arts of foreign nations i shall now proceed to delineate dangers of a different and perhaps still more alarming kindthose which will in all probability flow from dissensions between the states themselves and from domestic factions and convulsions these have been already in some instances slightly an'"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "list_o_papers_cleanedNoPunct[5][:500]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true,
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Seperate into Training and Test set"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true,
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "train_documents = [list_o_papers_cleanedNoPunct[i]  for i in xrange(len(fed_author_list)) if fed_author_list[i] in ['HAMILTON', 'MADISON']]\n",
    "\n",
    "train_labels = [author for author in fed_author_list if author in ['HAMILTON', 'MADISON']]\n",
    "\n",
    "test_documents = [list_o_papers_cleanedNoPunct[i]  for i in xrange(len(fed_author_list))  if fed_author_list[i] == 'HAMILTON OR MADISON']\n",
    "\n",
    "test_labels = [author for author in fed_author_list if author == 'HAMILTON OR MADISON']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "67"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(train_documents)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "13652.731343283582"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import numpy as np\n",
    "np.mean([len(i) for i in train_documents])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true,
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Process\n",
    "\n",
    "1. Find the data and parse into sets of [document, label]\n",
    "2. **Train a model on the labeled documents, preferably with cross validation. Select a model that performs best**\n",
    "3. Run the model on the documents that don't have labels yet"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true,
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "from sklearn.pipeline import Pipeline\n",
    "from sklearn.feature_extraction.text import CountVectorizer, TfidfTransformer\n",
    "from sklearn.linear_model import SGDClassifier\n",
    "\n",
    "text_clf = Pipeline([('vect', CountVectorizer()),\n",
    "                     ('tfidf', TfidfTransformer()),\n",
    "                     ('clf', SGDClassifier(loss='hinge', penalty='l2',\n",
    "                                           alpha=1e-3, n_iter=5, random_state=42)),\n",
    "])\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true,
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Pipeline(steps=[('vect', CountVectorizer(analyzer=u'word', binary=False, decode_error=u'strict',\n",
       "        dtype=<type 'numpy.int64'>, encoding=u'utf-8', input=u'content',\n",
       "        lowercase=True, max_df=1.0, max_features=None, min_df=1,\n",
       "        ngram_range=(1, 1), preprocessor=None, stop_words=None,\n",
       "        st...     penalty='l2', power_t=0.5, random_state=42, shuffle=True, verbose=0,\n",
       "       warm_start=False))])"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "text_clf.fit(train_documents, train_labels)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true,
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "predicted = text_clf.predict(train_documents)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true,
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array(['HAMILTON', 'HAMILTON', 'HAMILTON', 'HAMILTON', 'HAMILTON',\n",
       "       'MADISON', 'HAMILTON', 'HAMILTON', 'HAMILTON', 'MADISON'], \n",
       "      dtype='|S8')"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "predicted[:10]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true,
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.86567164179104472"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import numpy\n",
    "numpy.mean(predicted == train_labels) "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true,
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "from sklearn.model_selection import GridSearchCV\n",
    "parameters = {'vect__ngram_range': [(1, 1), (1, 2)],\n",
    "              'tfidf__use_idf': (True, False),\n",
    "              'clf__alpha': (1e-2, 1e-3),\n",
    "}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true,
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "gs_clf = GridSearchCV(text_clf, parameters, n_jobs=-1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true,
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "gs_clf = gs_clf.fit(train_documents, train_labels)\n",
    "predictions = gs_clf.predict(train_documents)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true,
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.9850746268656716"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "numpy.mean(predictions == train_labels)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true,
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Process\n",
    "\n",
    "1. Find the data and parse into sets of [document, label]\n",
    "2. Train a model on the labeled documents, preferably with cross validation. Select a model that performs best\n",
    "3. **Run the model on the documents that don't have labels yet**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true,
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array(['MADISON', 'MADISON', 'MADISON', 'MADISON', 'MADISON', 'MADISON',\n",
       "       'MADISON', 'MADISON', 'MADISON', 'MADISON', 'MADISON'], \n",
       "      dtype='|S8')"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "gs_clf.predict(test_documents)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true,
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## From the 1963 Paper:\n",
    "\n",
    "> In summary, the following points are clear:\n",
    "\n",
    "> 1) Madison is the principal author. The data make it possible to say far more than ever before that the odds are enormously high that Madison wrote the 12 disputed papers..."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true,
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Appendix\n",
    "\n",
    "References:\n",
    "\n",
    "Original Paper: Mosteller and Wallace, 1965 https://www.stat.cmu.edu/Exams/mosteller.pdf \n",
    "\n",
    "The story of the discovery: https://priceonomics.com/how-statistics-solved-a-175-year-old-mystery-about/\n",
    "\n",
    "(uses Naive Bayes)\n",
    "\n",
    "Recent approach (SVM): http://pages.cs.wisc.edu/~gfung/federalist.pdf\n",
    "\n",
    "Overview of Naive Bayes classifier: https://web.stanford.edu/~jurafsky/slp3/6.pdf\n",
    "\n",
    "Overview of text processing: https://web.stanford.edu/class/cs124/lec/naivebayes.pdf\n",
    "\n",
    "#### Scikit Learn Docs for Text Learning:\n",
    "\n",
    "http://scikit-learn.org/stable/tutorial/text_analytics/working_with_text_data.html\n",
    "\n",
    "http://scikit-learn.org/stable/auto_examples/text/document_classification_20newsgroups.html\n"
   ]
  }
 ],
 "metadata": {
  "celltoolbar": "Slideshow",
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}