Python-with-statement.ipynb

{
 "metadata": {
  "name": "Python-with-statement"
 },
 "nbformat": 3,
 "nbformat_minor": 0,
 "worksheets": [
  {
   "cells": [
    {
     "cell_type": "heading",
     "level": 1,
     "metadata": {},
     "source": [
      "The Python 'with' statement"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "In this lesson, you'll learn how to use Python's \"with\" statement to improve the reliability of your code. The \"with\" statement ensures that cleanup and shutdown code will always be executed, even in the presence of exceptions and nonstandard control flows. A resource is cleaned up as soon as it is no longer needed."
     ]
    },
    {
     "cell_type": "heading",
     "level": 2,
     "metadata": {},
     "source": [
      "The problem: Guaranteeing that an open file is closed when no longer needed"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "Most people learn that the sequence for opening and using a file in Python is open/read/close. For example, suppose we wanted to do something to every line in a file. We would write code like this:"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "f = open('myfile.txt')\n",
      "for line in f:\n",
      "    # Do something with this line\n",
      "    print line.strip()\n",
      "f.close()"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "This my file.\n",
        "And a very nice file it is.\n",
        "All my family tell me how nice it is.\n"
       ]
      }
     ],
     "prompt_number": 1
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "(Yes, there are simpler ways of doing the above. The `with` statement will still be useful for those cases, but describing how it's useful would get us ahead of ourselves.)"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "You probably already know that every file should be closed when you no longer will be using it. The above code looks like it guarantees the file will be closed as soon as the loop completes. But what if the loop body is more complicated? Suppose we know every line begins with an integer and want to get the value of that integer:"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "!cat my-numeric-file.txt"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "1 first line\r\n",
        "20 second line\r\n",
        "-3 third line\r\n",
        "44 fourth line\r\n"
       ]
      }
     ],
     "prompt_number": 2
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "f = open('my-numeric-file.txt')\n",
      "for line in f:\n",
      "    num = int(line.split()[0])\n",
      "    # Do something with num\n",
      "f.close()"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 3
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "Works great!  Unless ... one of the lines doesn't start with an integer:"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "!cat my-nearly-numeric-file.txt"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "1 first line\r\n",
        "20 second line\r\n",
        "30 third line\r\n",
        "fourth line\r\n",
        "5 fifth line\r\n"
       ]
      }
     ],
     "prompt_number": 4
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "f = open('my-nearly-numeric-file.txt')\n",
      "for line in f:\n",
      "    num = int(line.split()[0])\n",
      "    # Do something with num\n",
      "f.close()"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "ename": "ValueError",
       "evalue": "invalid literal for int() with base 10: 'fourth'",
       "output_type": "pyerr",
       "traceback": [
        "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m\n\u001b[0;31mValueError\u001b[0m                                Traceback (most recent call last)",
        "\u001b[0;32m<ipython-input-5-cd61065ad6aa>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m()\u001b[0m\n\u001b[1;32m      1\u001b[0m \u001b[0mf\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mopen\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m'my-nearly-numeric-file.txt'\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m      2\u001b[0m \u001b[0;32mfor\u001b[0m \u001b[0mline\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mf\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 3\u001b[0;31m     \u001b[0mnum\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mint\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mline\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0msplit\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;36m0\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m      4\u001b[0m     \u001b[0;31m# Do something with num\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m      5\u001b[0m \u001b[0mf\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mclose\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
        "\u001b[0;31mValueError\u001b[0m: invalid literal for int() with base 10: 'fourth'"
       ]
      }
     ],
     "prompt_number": 5
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "Because we had an exception processing the fourth line of the file, the file was never closed (`f.closed` is `False`) and we can read the fifth line:"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "print f.closed\n",
      "f.next()"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "False\n"
       ]
      },
      {
       "output_type": "pyout",
       "prompt_number": 6,
       "text": [
        "'5 fifth line\\n'"
       ]
      }
     ],
     "prompt_number": 6
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "Oops!\n"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "f.close() # Clean up manually\n",
      "f.closed"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "pyout",
       "prompt_number": 7,
       "text": [
        "True"
       ]
      }
     ],
     "prompt_number": 7
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "There are other situations where the `close()` function  might be missed. For example, suppose we have a function with multiple `return` statements:"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "def multReturns():\n",
      "    global f # Make it visible outside our function (for this example)\n",
      "    f = open('my-numeric-file.txt')\n",
      "    for line in f:\n",
      "        num = int(line.split()[0])\n",
      "        if num < 0:\n",
      "            return\n",
      "    f.close()\n",
      "    return\n",
      "multReturns()"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 8
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "The third line had a negative integer, so we returned from the function early. The `f.close()` was never executed and the file remains open:"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "print f.closed\n",
      "f.next()"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "False\n"
       ]
      },
      {
       "output_type": "pyout",
       "prompt_number": 9,
       "text": [
        "'44 fourth line\\n'"
       ]
      }
     ],
     "prompt_number": 9
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "f.close() # Clean up manually\n",
      "f.closed"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "pyout",
       "prompt_number": 10,
       "text": [
        "True"
       ]
      }
     ],
     "prompt_number": 10
    },
    {
     "cell_type": "heading",
     "level": 2,
     "metadata": {},
     "source": [
      "The solution: Python's 'with' statement"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "There are ways of surmounting the above problms individually, such as reading the entire file at once using `f.readlines()`, using the Python `try ... except` statement, or the Python `try ... finally` statement. Those statements are useful and well worth understanding and using but they in fact only *reduce* the risk of not closing your file, they do not entirely *eliminate* it.\n",
      "\n",
      "We need a method of indicating in Python the exact **lifetime** of the file access, the exact period during the program's run when the file needs to be open. Then Python can close the file automatically whenever that lifetime ends.\n",
      "\n",
      "But how will we specify the lifetime of the file's access? The examples above were tricky because the file access could end so many different ways. We'll link the *semantics* of the lifetime, which play out as the program runs, with the *syntax* of how the program is represented statically on the page. The `with` statement does just this: It limits the range of the file's to a region of the static program text. Let's look at how the function `multReturns` would appear, rewritten using `with`:"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "def multReturnsWith(fname):\n",
      "    global f # Make it visible outside our function (for this example)\n",
      "    with open(fname) as f: # NEW\n",
      "        for line in f:\n",
      "            num = int(line.split()[0])\n",
      "            if num < 0:\n",
      "                return\n",
      "    return\n",
      "\n",
      "multReturnsWith('my-numeric-file.txt')\n",
      "f.closed"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "pyout",
       "prompt_number": 11,
       "text": [
        "True"
       ]
      }
     ],
     "prompt_number": 11
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "There's a call to `open()` here but no call to `close()`. Yet the value of `f.closed` shows the file has been closed, even though the function did a `return` partway through the file, on the third line. If we try to read the next line, we get an error:"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "f.next()"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "ename": "ValueError",
       "evalue": "I/O operation on closed file",
       "output_type": "pyerr",
       "traceback": [
        "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m\n\u001b[0;31mValueError\u001b[0m                                Traceback (most recent call last)",
        "\u001b[0;32m<ipython-input-12-c3e65e5362fb>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m()\u001b[0m\n\u001b[0;32m----> 1\u001b[0;31m \u001b[0mf\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mnext\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m",
        "\u001b[0;31mValueError\u001b[0m: I/O operation on closed file"
       ]
      }
     ],
     "prompt_number": 12
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "In fact, no matter how the function returns, the file will be closed.  The file will be closed if the function has an exception:"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "multReturnsWith('my-nearly-numeric-file.txt')"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "ename": "ValueError",
       "evalue": "invalid literal for int() with base 10: 'fourth'",
       "output_type": "pyerr",
       "traceback": [
        "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m\n\u001b[0;31mValueError\u001b[0m                                Traceback (most recent call last)",
        "\u001b[0;32m<ipython-input-13-5fe6ebd98525>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m()\u001b[0m\n\u001b[0;32m----> 1\u001b[0;31m \u001b[0mmultReturnsWith\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m'my-nearly-numeric-file.txt'\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m",
        "\u001b[0;32m<ipython-input-11-e256a21316b5>\u001b[0m in \u001b[0;36mmultReturnsWith\u001b[0;34m(fname)\u001b[0m\n\u001b[1;32m      3\u001b[0m     \u001b[0;32mwith\u001b[0m \u001b[0mopen\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mfname\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;32mas\u001b[0m \u001b[0mf\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0;31m# NEW\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m      4\u001b[0m         \u001b[0;32mfor\u001b[0m \u001b[0mline\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mf\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 5\u001b[0;31m             \u001b[0mnum\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mint\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mline\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0msplit\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;36m0\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m      6\u001b[0m             \u001b[0;32mif\u001b[0m \u001b[0mnum\u001b[0m \u001b[0;34m<\u001b[0m \u001b[0;36m0\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m      7\u001b[0m                 \u001b[0;32mreturn\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
        "\u001b[0;31mValueError\u001b[0m: invalid literal for int() with base 10: 'fourth'"
       ]
      }
     ],
     "prompt_number": 13
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "f.closed"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "pyout",
       "prompt_number": 14,
       "text": [
        "True"
       ]
      }
     ],
     "prompt_number": 14
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "And the file will be closed if it only begins with positive numbers and the function runs to completion:"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "!cat my-positive-numeric-file.txt"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "1 first line\r\n",
        "20 second line\r\n",
        "30 third line\r\n",
        "44 fourth line\r\n"
       ]
      }
     ],
     "prompt_number": 15
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "multReturnsWith('my-positive-numeric-file.txt')\n",
      "f.closed"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "pyout",
       "prompt_number": 16,
       "text": [
        "True"
       ]
      }
     ],
     "prompt_number": 16
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "The `with` statement is a Python [compound statement](http://docs.python.org/3/reference/compound_stmts.html#the-with-statement). The statement itself initializes a resource and associates a variable with that resource.  In our example, the resource is a file and we associate it with the variable `f`. Then the block of statements the `with` delimits---the block following the colon and indented from the `with`---is the **scope** of the variable.  As soon as control leaves the block inside the `with`, whether due to an exception, a `return` from a function, or simply completing the last statement in the block, the resource is closed."
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "That is why there are no `f.close()` calls in our example. It is closed automatically whenever the variable `f`'s lifetime ends. Furthermore, the semantics of the `with` statement that the lifetime of variable `f` ends whenever we leave the `with` statement's block for any reason. "
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "This is exactly what we want: Guaranteed closing of the file as soon as we no longer want to access it. The `with` statement gives us a way to syntactically express exactly when the file will be used and when we will no longer need it."
     ]
    },
    {
     "cell_type": "heading",
     "level": 2,
     "metadata": {},
     "source": [
      "The new idiom: Always open your files in a 'with' statement"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "You can avert all the above problems by always opening a file using the `with` idiom, replacing the filename and the variable `f` as necessary:"
     ]
    },
    {
     "cell_type": "raw",
     "metadata": {},
     "source": [
      "with open('myfile.txt') as f:\n",
      "    # all the code that will access the file, using variable 'f'\n",
      "# The file will be closed and cannot be accessed here"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "You might have to reorder your code so that all accesses fit within the body of the `with`.  If the code using the file is extensive, you may want to move it all into a function. You will have to pass the file variable as an argument so the function can access the file:"
     ]
    },
    {
     "cell_type": "raw",
     "metadata": {},
     "source": [
      "with open('world-census.csv') as census:\n",
      "    crossTabCensusData(census)\n",
      "# The file is closed automatically"
     ]
    },
    {
     "cell_type": "heading",
     "level": 2,
     "metadata": {},
     "source": [
      "The 'with' statement can be used with other resources, such as URLs"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "The `with` statement can be used in many other contexts as well.  Any time you have a resource that requires cleanup, such as a database link that needs to be closed, a large array that needs to be deleted to free up storage, or a temporary decimal precision context, you should use a `with` statement to guarantee that the resource is cleaned up *all the time, every time*. These more advanced uses often require tools from the `contextlib` library. By using the `with` statement, you increase the reliability, security, and readability of your code. How many other tools give you all that?"
     ]
    }
   ],
   "metadata": {}
  }
 ]
}