2_Policy_Gradients-problems.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Assignment 2: Policy Gradient Methods\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# Automatically reload changes to external code\n",
    "%load_ext autoreload\n",
    "%autoreload 2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "In this assignment, you will implement the \"vanilla\" policy gradient method, i.e., a method that repeatedly computes estimates $\\hat{g}$ of $\\nabla_{\\theta} E[\\sum_t R_t]$ and takes gradient ascent steps $\\theta \\rightarrow \\theta + \\epsilon \\hat{g}$.\n",
    "To keep our code generic, so that we can write the same code to solve multiple MDPs with multiple policy parameterizations, we'll use classes for Policy and MDP. You can view the base classes in the file `rl.py`.\n",
    "\n",
    "We will use a grid-world MDP called FrozenLake.\n",
    "The following code constructs an instance of the MDP, and then prints its documentation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "from rl import rollout, Policy\n",
    "from frozen_lake import FrozenLake\n",
    "import numpy as np, numpy.random as nr\n",
    "\n",
    "np.random.seed(1)\n",
    "\n",
    "map4x4 = [\n",
    "    \"SFFF\",\n",
    "    \"FHFH\",\n",
    "    \"FFFH\",\n",
    "    \"HFFG\"]\n",
    "mdp = FrozenLake(map4x4)\n",
    "\n",
    "# FrozenLake is a MDP with finite state and action that involves navigating across a frozen lake.\n",
    "# (It's conventionally called a \"grid-world\" MDP, as the state space involves points in a 2D grid)\n",
    "# Let's look at the docstring for details\n",
    "print FrozenLake.__doc__\n",
    "print \"-----------------\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, we will define a random policy and use it to perform a rollout. The rollout data will be used for gradient checking."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "class RandomDiscreteActionChooser(Policy):\n",
    "    def __init__(self, n_actions):\n",
    "        self.n_actions = n_actions\n",
    "    def step(self, observation):\n",
    "        return {\"action\":np.array([nr.randint(0, self.n_actions)])}\n",
    "\n",
    "policy = RandomDiscreteActionChooser(mdp.n_actions)\n",
    "    \n",
    "rdata = rollout(mdp, policy, 100)\n",
    "print rdata"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "s_n = rdata['observations'] # Vector of states (same as observations since MDP is fully-observed)\n",
    "a_n = rdata['actions'] # Vector of actions (each is an int in {0,1,2,3})\n",
    "n = a_n.shape[0] # Length of trajectory\n",
    "q_n = np.random.randn(n) # Returns (random for the sake of gradient checking)\n",
    "f_sa = np.random.randn(mdp.n_states, mdp.n_actions) # Policy parameter vector. explained shortly."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, a little explanation about the preceding definitions.\n",
    "First, we are using the convention that the letters after the underscore indicate what is indexing the array.\n",
    "For example, `s_n` has a single subscript `n`, meaning that it is a 1D vector, and we will use `n` to index over timesteps.\n",
    "`f_sa` is a matrix, whose first dimension `s` indexes over states, and whose second dimension `a` indexes over actions.\n",
    "This convention helps us avoid errors where axes are mismatched.\n",
    "\n",
    "The policy for our discrete MDP will be encoded by the matrix `f_sa`. The action probabilities are defined by exponentiating this matrix (elementwise) and then normalizing across the `a` dimension so the probabilities add up to 1. I.e., the matrix of probabilities is defined as follows"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "def softmax_prob(f_na):    \n",
    "    \"\"\"\n",
    "    Exponentiate f_na and normalize rows to have sum 1\n",
    "    so each row gives a probability distribution over discrete\n",
    "    action set\n",
    "    \"\"\"\n",
    "    prob_nk = np.exp(f_na - f_na.max(axis=1,keepdims=True))\n",
    "    prob_nk /= prob_nk.sum(axis=1,keepdims=True)\n",
    "    return prob_nk"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Problem 1: implement policy gradient computation\n",
    "\n",
    "Next, you're going to implement a function for computing the policy gradient.\n",
    "We will define a function called `softmax_policy_checkfunc` -- your function should compute the gradient of it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def softmax_policy_checkfunc(f_sa, s_n, a_n, q_n):\n",
    "    r\"\"\"\n",
    "    An auxilliary function that's useful for checking the policy gradient\n",
    "    The inputs are \n",
    "    s_n : states (vector of int)\n",
    "    a_n : actions (vector of int)\n",
    "    q_n : returns (vectof of float)\n",
    "\n",
    "    This function returns\n",
    "\n",
    "    \\sum_n \\log \\pi(a_n | s_n) q_n\n",
    "\n",
    "    whose gradient is the policy gradient estimator\n",
    "\n",
    "    \\sum_n \\grad \\log \\pi(a_n | s_n) q_n\n",
    "\n",
    "    \"\"\"\n",
    "    f_na = f_sa[s_n]\n",
    "    p_na = softmax_prob(f_na)\n",
    "    n = s_n.shape[0]\n",
    "    return np.log(p_na[np.arange(n), a_n]).dot(q_n)/n\n",
    "\n",
    "def softmax_policy_gradient(f_sa, s_n, a_n, adv_n):\n",
    "    \"\"\"\n",
    "    Compute policy gradient of policy for discrete MDP, where probabilities\n",
    "    are obtained by exponentiating f_sa and normalizing\n",
    "    \"\"\"\n",
    "    # YOUR CODE HERE >>>>>>\n",
    "    # <<<<<<<< \n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We'll verify your function by checking the gradient numerically. You'll need the `numdifftools` module, which computes highly-precise numerical gradient estimates. (To install this module, open up a shell and run `pip install numdifftools`)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "import numdifftools as ndt\n",
    "\n",
    "stepdir = np.random.randn(*f_sa.shape)\n",
    "auxfunc = lambda x: softmax_policy_checkfunc(f_sa+stepdir*x, s_n, a_n, q_n)\n",
    "\n",
    "numgrad = ndt.Derivative(auxfunc)(0)\n",
    "g = softmax_policy_gradient(f_sa, s_n, a_n, q_n)\n",
    "anagrad = (stepdir*g).sum()\n",
    "\n",
    "assert abs(numgrad - anagrad) < 1e-10"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now you have a function that computes the policy gradient. Now you're ready to write a function that uses it to optimize the policy.\n",
    "First we'll define a Policy class that'll be used by your policy gradient optimization algorithm."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from hw_utils import Message, discount, fmt_row\n",
    "from rl import rollout, pathlength\n",
    "import numpy as np\n",
    "from collections import defaultdict\n",
    "from categorical import cat_sample, cat_entropy, cat_kl\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "class FrozenLakeTabularPolicy(Policy):\n",
    "    def __init__(self, n_states):\n",
    "        self.n_states = n_states\n",
    "        self.n_actions = n_actions = 4        \n",
    "        self.f_sa = np.zeros((n_states, n_actions))\n",
    "\n",
    "    def step(self, s_n):\n",
    "        f_na = self.f_sa[s_n]\n",
    "        prob_nk = softmax_prob(f_na)\n",
    "        acts_n = cat_sample(prob_nk)\n",
    "        return {\"action\":acts_n,\n",
    "                \"pdist\" : f_na}\n",
    "    \n",
    "    \n",
    "    def compute_pdists(self, s_n):\n",
    "        return self.f_sa[s_n]\n",
    "\n",
    "    def compute_entropy(self, f_na):\n",
    "        prob_nk = softmax_prob(f_na)\n",
    "        return cat_entropy(prob_nk)\n",
    "\n",
    "    def compute_kl(self, f0_na, f1_na):\n",
    "        p0_na = softmax_prob(f0_na)\n",
    "        p1_na = softmax_prob(f1_na)\n",
    "        return cat_kl(p0_na, p1_na)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Problem 2: Implement policy gradient optimization algorithm\n",
    "\n",
    "Complete the function below, to obtain an algorithm that optimizes the policy by computing policy gradient estimates.\n",
    "Output of a successful implementation is shown below this cell. Your result doesn't need to be exactly the same numerically, but the performance should be on far."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "def policy_gradient_optimize(mdp, policy,\n",
    "        gamma,\n",
    "        max_pathlength,\n",
    "        timesteps_per_batch,\n",
    "        n_iter,\n",
    "        stepsize):\n",
    "    stat2timeseries = defaultdict(list)\n",
    "    widths = (17,10,10,10,10)\n",
    "    print fmt_row(widths, [\"EpRewMean\",\"EpLenMean\",\"Perplexity\",\"KLOldNew\"])\n",
    "    for i in xrange(n_iter):\n",
    "        total_ts = 0\n",
    "        paths = [] \n",
    "        while True:\n",
    "            path = rollout(mdp, policy, max_pathlength)                \n",
    "            paths.append(path)\n",
    "            total_ts += pathlength(path)\n",
    "            if total_ts > timesteps_per_batch: \n",
    "                break\n",
    "\n",
    "        # get observations:\n",
    "        obs_no = np.concatenate([path[\"observations\"] for path in paths])\n",
    "\n",
    "        # Your code should compute the policy gradient and update the policy parameters\n",
    "        # you'll need to compute the discounted returns, compute the policy gradient, and update the parameters\n",
    "        # YOUR CODE HERE >>>>>>\n",
    "        # <<<<<<<< \n",
    "\n",
    "        pdists = np.concatenate([path[\"pdists\"] for path in paths])\n",
    "        kl = policy.compute_kl(pdists, policy.compute_pdists(obs_no)).mean()\n",
    "        perplexity = np.exp(policy.compute_entropy(pdists).mean())\n",
    "\n",
    "        stats = {  \"EpRewMean\" : np.mean([path[\"rewards\"].sum() for path in paths]),\n",
    "                   \"EpRewSEM\" : np.std([path[\"rewards\"].sum() for path in paths])/np.sqrt(len(paths)),\n",
    "                   \"EpLenMean\" : np.mean([pathlength(path) for path in paths]),\n",
    "                   \"Perplexity\" : perplexity,\n",
    "                   \"KLOldNew\" : kl }\n",
    "        print fmt_row(widths, ['%.3f+-%.3f'%(stats[\"EpRewMean\"], stats['EpRewSEM']), stats['EpLenMean'], stats['Perplexity'], stats['KLOldNew']])\n",
    "        \n",
    "        \n",
    "        for (name,val) in stats.items():\n",
    "            stat2timeseries[name].append(val)\n",
    "    return stat2timeseries\n",
    "            \n",
    "policy = FrozenLakeTabularPolicy(mdp.n_states)\n",
    "\n",
    "np.random.seed(0)            \n",
    "stat2ts = policy_gradient_optimize(mdp, policy,\n",
    "                gamma=.98,\n",
    "                max_pathlength=100,\n",
    "                timesteps_per_batch=2000,\n",
    "                n_iter=100,\n",
    "                stepsize=100)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Output of successful implementation for problem 2...\n",
    "\n",
    "#      0.232+-0.042 |    20.2525 |     1.7738 | 0.00553472\n",
    "#      0.180+-0.041 |    22.5281 |    1.64021 | 0.000818475\n",
    "#      0.286+-0.046 |    20.4592 |    1.65031 | 0.00210285\n",
    "#      0.310+-0.050 |    23.1379 |    1.58822 | 0.000949984\n",
    "#      0.229+-0.043 |    21.1667 |    1.51748 | 0.00126626\n",
    "#      0.333+-0.053 |    25.7821 |    1.45515 | 0.00517389\n",
    "#      0.279+-0.054 |    29.7794 |    1.45849 | 0.00126206\n",
    "#      0.279+-0.054 |    29.8382 |    1.39328 | 0.000978533\n",
    "#      0.405+-0.057 |     27.473 |    1.36296 | 0.00081737\n",
    "#      0.395+-0.056 |    26.5921 |    1.33036 | 0.000578766\n",
    "#      0.439+-0.061 |    30.9394 |    1.29982 | 0.000903682\n",
    "#      0.403+-0.062 |    32.6935 |    1.28342 |  0.0003479\n",
    "#      0.461+-0.057 |    26.3421 |    1.30297 | 0.000425174\n",
    "#      0.368+-0.058 |    30.6912 |    1.27384 | 0.000224248\n",
    "#      0.545+-0.061 |    30.5758 |     1.2691 | 0.00110955\n",
    "#      0.508+-0.062 |    31.2308 |    1.26778 | 0.000117507\n",
    "#      0.391+-0.061 |    31.5156 |      1.242 | 0.000249331\n",
    "#      0.365+-0.061 |    31.9048 |     1.2347 | 0.000131674\n",
    "#      0.492+-0.064 |    32.8852 |    1.25998 | 0.000360727\n",
    "#      0.394+-0.058 |    28.4648 |    1.21109 | 0.000364956\n",
    "#      0.419+-0.063 |    32.4032 |    1.22287 | 0.000336991\n",
    "#      0.582+-0.067 |    37.6909 |    1.21275 | 0.00017788\n",
    "#      0.485+-0.062 |    30.4545 |    1.24474 | 0.000368472\n",
    "#      0.509+-0.067 |    36.4909 |    1.19584 | 0.000191174\n",
    "#      0.576+-0.064 |    33.9831 |     1.1816 | 8.99578e-05\n",
    "#      0.690+-0.061 |    35.1379 |    1.17238 | 0.000160158\n",
    "#      0.627+-0.063 |    34.1356 |    1.16137 | 8.35832e-05\n",
    "#      0.500+-0.071 |      40.34 |    1.16211 | 9.52723e-05\n",
    "#      0.464+-0.067 |    36.8036 |    1.17209 | 8.54947e-05\n",
    "#      0.509+-0.066 |    35.6491 |    1.16041 | 7.82523e-05\n",
    "#      0.632+-0.064 |    35.1579 |    1.15874 | 7.54936e-05\n",
    "#      0.582+-0.067 |    36.5273 |    1.15924 | 8.17684e-05\n",
    "#      0.630+-0.066 |    37.3704 |     1.1467 | 0.000170618\n",
    "#      0.583+-0.064 |    33.5167 |    1.15033 | 4.80961e-05\n",
    "#      0.662+-0.057 |    30.6324 |    1.13978 | 7.26294e-05\n",
    "#      0.618+-0.066 |    36.8545 |    1.13128 | 3.96185e-05\n",
    "#      0.529+-0.070 |    39.8431 |     1.1494 | 9.74017e-05\n",
    "#      0.589+-0.066 |    36.4107 |    1.13971 | 0.000125295\n",
    "#      0.644+-0.062 |    34.1695 |    1.12859 | 4.46991e-05\n",
    "#      0.625+-0.065 |    36.6071 |    1.11733 | 3.96966e-05\n",
    "#      0.526+-0.066 |    36.1053 |    1.13024 | 7.80788e-05\n",
    "#      0.667+-0.061 |    33.8667 |    1.10236 | 0.000207277\n",
    "#      0.617+-0.063 |      33.85 |    1.10662 | 6.32027e-05\n",
    "#      0.529+-0.070 |    39.8627 |    1.11525 | 2.46512e-05\n",
    "#      0.577+-0.069 |    38.7885 |    1.11533 | 1.71757e-05\n",
    "#      0.656+-0.061 |     34.082 |    1.14483 | 0.000279053\n",
    "#      0.709+-0.061 |    37.0182 |     1.1183 | 5.70132e-05\n",
    "#      0.589+-0.066 |    36.0536 |    1.13235 | 3.61136e-05\n",
    "#      0.714+-0.060 |    36.3929 |    1.12589 | 7.89316e-05\n",
    "#      0.617+-0.063 |    33.6333 |    1.11578 | 0.000110617\n",
    "#      0.661+-0.062 |    34.9661 |    1.11546 | 0.000102611\n",
    "#      0.661+-0.062 |    34.1864 |    1.12734 | 0.000132361\n",
    "#      0.660+-0.069 |    42.7872 |    1.10657 | 0.000389893\n",
    "#      0.732+-0.059 |    35.8036 |    1.12287 | 0.000284315\n",
    "#      0.569+-0.065 |    34.9483 |    1.11962 | 0.000197253\n",
    "\n",
    "# Several plots can be generated from the results:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "%matplotlib inline\n",
    "plt.figure()\n",
    "plt.title(\"Episode Reward\")\n",
    "EpRewMean = np.array(stat2ts[\"EpRewMean\"])\n",
    "EpRewStd = np.array(stat2ts[\"EpRewSEM\"])\n",
    "plt.errorbar(np.arange(len(EpRewMean)), EpRewMean, yerr=EpRewStd, errorevery=5, linewidth=1)\n",
    "plt.figure()\n",
    "plt.title(\"Mean Episode Length\")\n",
    "plt.plot(stat2ts[\"EpLenMean\"])\n",
    "plt.figure()\n",
    "plt.title(\"Perplexity\")\n",
    "plt.plot(stat2ts[\"Perplexity\"])\n",
    "plt.figure()\n",
    "plt.title(\"Mean KL Divergence Between Old & New Policies\")\n",
    "plt.plot(stat2ts[\"KLOldNew\"]);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note that the perplexity goes down to 1, corresponding to a deterministic policy."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's \"plot\" the learned policy:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "from rl import animate_rollout\n",
    "animate_rollout(mdp,policy,delay=.001,horizon=100)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Problem 3: Implement some enhancement or variation on policy gradient optimization algorithm\n",
    "\n",
    "Some possibilities:\n",
    "- add a state-dependent baseline function\n",
    "- use RMSProp, momentum gradient descent, or ADAM as the underlying stochastic optimization algorithm instead of SGD\n",
    "- implement some variant of the natural gradient algorithm or trust region policy optimization\n",
    "- something else that you invented\n",
    "\n",
    "Run your algorithm on the mdp above (with a 4x4 grid), and also run it on the following 8x8 grid.\n",
    "Plot the learning curve you obtain. You will probably have to tune stepsize, and it might help to adjust the discount factor gamma.\n",
    "\n",
    "    map8x8 = [\n",
    "        \"SFFFFFFF\",\n",
    "        \"FFFFFFFF\",\n",
    "        \"FFFHFFFF\",\n",
    "        \"FFFFFHFF\",\n",
    "        \"FFFHFFFF\",\n",
    "        \"FHHFFFHF\",\n",
    "        \"FHFFHFHF\",\n",
    "        \"FFFHFFFG\"\n",
    "    ]\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Problem 4: Experiment with Atari domain\n",
    "\n",
    "For this problem, you will experiment with the Atari domain. You will learn a policy to play Atari using the RAM state of the Atari emulator as input. It's significantly easier and faster to train that using images; however, it's still a highly non-trivial problem. In the instructors' experience, methods that work well on RAM also work well on images.\n",
    "\n",
    "First, build the required libraries by navigating into the code directory and running\n",
    "\n",
    "    ./setup_ale.sh\n",
    "\n",
    "You have two choices\n",
    "1. Use the instructor's implementation of *Proximal Policy Optimization* (A variant of [Trust Region Policy Optimization](http://arxiv.org/abs/1502.05477) which is simpler because it solves an unconstrained optimization problem). The code is provided in `ppo.py`. It requires [CGT](http://rll.berkeley.edu/cgt). Implement some variation on this algorithm. Some possibilities:\n",
    "   - Use a neural network value function instead of a linear value function\n",
    "   - Adjust the KL divergence penalty coefficient in an adaptive way so that the KL divergence is roughly fixed across iterations.\n",
    "   - Use SGD instead of LBFGS on each batch\n",
    "   - Use a different kind of discounting scheme, e.g. hyperbolic discounts\n",
    "   - Vary the neural network architecture of the policy\n",
    "   - Something else that you invented\n",
    "2. Implement a policy gradient method yourself, and run that. For example, you could use the policy gradient method you implemented in problems 1-3. In my experience, \"vanilla\" policy gradient methods don't work very well on this problem in a reasonable amount of time, but you may be able to get it working with some tinkering.\n",
    "\n",
    "Note: it's quite helpful to plot what the current policy is doing, as a diagnostic. To do this, you can use the `animate_rollout` function in `rl.py`.\n",
    "Also, it's helpful to run simulations in parallel, because simulation is usually the bottleneck on these problems with relatively low-dimensional inputs. See `ppo.py` for an example of how to do this with the multiprocessing module.\n",
    "\n",
    "<div class=\"alert alert-warning\">\n",
    "**Final Deliverable**: implement a policy gradient algorithm for Atari, following guidelines of (1) and (2) above, and perform a comparison of at least two different experimental conditions. E.g. [reference implementation, your variant], or [your idea 1, your idea 2, your idea 3]. Plot learning curves (i.e. the mean episode reward) for all experimental conditions.\n",
    "\n",
    "You may want to write the code outside of this notebook (if so, please include it in a zip file with your submission), but please put the plots **here**.\n",
    "</div>\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Problem 5: Experiment with MuJoCo / Locomotion\n",
    "\n",
    "For this problem, you will experiment with the MuJoCo domain, and learn a controller that enables a simulated robot character to hop.\n",
    "\n",
    "First, build the required libraries by navigating into the code directory and running\n",
    "\n",
    "    ./setup_mujoco.py\n",
    "    \n",
    "Again, you have the choice of modifying the instructor's implementation or using your own implementation.\n",
    "Follow the exact same instructions as in Problem 4 and plot learning curves showing the performance of your implementation."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}