F1D-01-theStewBS4.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# FLIP(01):  Advanced Data Science\n",
    "**(Module D: Web Crawler)**\n",
    "\n",
    "---\n",
    "- Materials in this module include resources collected from various open-source online repositories.\n",
    "- You are free to use, but NOT allowed to change or distribute this package.\n",
    "- If you found any issue/bug for this document, please submit an issue at [tulip lab/flip01](https://github.com/tulip-lab/flip01/issues)\n",
    "\n",
    "\n",
    "Prepared by and for \n",
    "**Student Members** |\n",
    "2006-2022 [TULIP Lab](http://www.tulip.org.au)\n",
    "\n",
    "---\n",
    "\n",
    "\n",
    "# Session 01 - The Stew: Beautiful Soup 4"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "After you have your ingredients, now what? Now you make them into a stew… a beautiful stew.\n",
    "\n",
    "Beautiful Soup (BS4) is a parsing library that can use different parsers. A parser is simply a program that can extract data from HTML and XML documents.\n",
    "\n",
    "Beautiful Soup’s default parser comes from Python’s standard library. It’s flexible and forgiving, but a little slow. The good news is that you can swap out its parser with a faster one if you need the speed.\n",
    "\n",
    "One advantage of BS4 is its ability to automatically detect encodings. This allows it to gracefully handle HTML documents with special characters.\n",
    "\n",
    "In addition, BS4 can help you navigate a parsed document and find what you need. This makes it quick and painless to build common applications. For example, if you wanted to find all the links in the web page we pulled down earlier, it’s only a few lines:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import requests\n",
    "page = requests.get('http://examplesite.com')\n",
    "contents = page.content"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from bs4 import BeautifulSoup\n",
    "soup = BeautifulSoup(contents, 'html.parser')\n",
    "soup.find_all('a')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from bs4 import BeautifulSoup"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "html_doc = \"\"\"\n",
    "<html><head><title>The Dormouse's story</title></head>\n",
    "<body>\n",
    "<p class=\"title\"><b>The Dormouse's story</b></p>\n",
    "\n",
    "<p class=\"story\">Once upon a time there were three little sisters; and their names were\n",
    "<a href=\"http://example.com/elsie\" class=\"sister\" id=\"link1\">Elsie</a>,\n",
    "<a href=\"http://example.com/lacie\" class=\"sister\" id=\"link2\">Lacie</a> and\n",
    "<a href=\"http://example.com/tillie\" class=\"sister\" id=\"link3\">Tillie</a>;\n",
    "and they lived at the bottom of a well.</p>\n",
    "\n",
    "<p class=\"story\">...</p>\n",
    "\"\"\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "soup = BeautifulSoup(html_doc, 'html.parser')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(soup.prettify())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here are some simple ways to navigate that data structure:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "soup.title"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "soup.title.name"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "soup.title.string"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "soup.title.parent.name"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "soup.p"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "soup.p['class']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "soup.a"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "soup.find_all('a')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "soup.find(id='link3')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "One common task is extracting all the URLs found within a page’s <a> tags:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "for link in soup.find_all('a'):\n",
    "    print(link.get('href'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Another common task is extracting all the text from a page:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(soup.get_text())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Making the soup"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To parse a document, pass it into the BeautifulSoup constructor. You can pass in a string or an open filehandle:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from bs4 import BeautifulSoup\n",
    "\n",
    "with open(\"index.html\") as fp:\n",
    "    soup = BeautifulSoup(fp)\n",
    "\n",
    "soup = BeautifulSoup(\"<html>data</html>\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First, the document is converted to Unicode, and HTML entities are converted to Unicode characters:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "BeautifulSoup(\"Sacr&eacute; bleu!\")\n",
    "<html><head></head><body>Sacré bleu!</body></html>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Kinds of objects"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Beautiful Soup transforms a complex HTML document into a complex tree of Python objects. But you’ll only ever have to deal with about four kinds of objects: Tag, NavigableString, BeautifulSoup, and Comment."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "** Tag **\n",
    "\n",
    "A Tag object corresponds to an XML or HTML tag in the original document:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "soup = BeautifulSoup('<b class=\"boldest\">Extremely bold</b>')\n",
    "tag = soup.b\n",
    "type(tag)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Tags have a lot of attributes and methods, and I’ll cover most of them in Navigating the tree and Searching the tree. For now, the most important features of a tag are its name and attributes."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "** Name ** \n",
    "\n",
    "Every tag has a name, accessible as .name:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "tag.name"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you change a tag’s name, the change will be reflected in any HTML markup generated by Beautiful Soup:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "tag.name = 'blockquote'\n",
    "tag"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "** Attributes **\n",
    "\n",
    "A tag may have any number of attributes. The tag $<id=\"boldest\">$ has an attribute “id” whose value is “boldest”. You can access a tag’s attributes by treating the tag like a dictionary:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "tag['id'] = 'boldest'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "tag['id']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can access that dictionary directly as  *.attrs*:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "tag.attrs"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can add, remove, and modify a tag’s attributes. Again, this is done by treating the tag as a dictionary:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "tag['id'] = 'verybold'\n",
    "tag['another-attribute'] = 1\n",
    "tag"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "del tag['id']\n",
    "del tag['another-attribute']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "tag"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, if you print *tag[' id ']*, it will apper an KeyError."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "tag['id']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(tag.get('id'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Multi-valued attributes**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "HTML 4 defines a few attributes that can have multiple values. HTML 5 removes a couple of them, but defines a few more. The most common multi-valued attribute is *class* (that is, a tag can have more than one CSS class). Others include *rel, rev, accept-charset, headers*, and *accesskey*. Beautiful Soup presents the value(s) of a multi-valued attribute as a list:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "css_soup = BeautifulSoup('<p class=\"body\"></p>')\n",
    "css_soup.p['class']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "css_soup = BeautifulSoup('<p class=\"body strikeout\"></p>')\n",
    "css_soup.p['class']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If an attribute *looks* like it has more than one value, but it’s not a multi-valued attribute as defined by any version of the HTML standard, Beautiful Soup will leave the attribute alone:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "id_soup = BeautifulSoup('<p id=\"my id\"></p>')\n",
    "id_soup.p['id']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "When you turn a tag back into a string, multiple attribute values are consolidated:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "rel_soup = BeautifulSoup('<p>Back to the <a rel=\"index\">homepage</a></p>')\n",
    "rel_soup.a['rel']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "rel_soup.a['rel'] = ['index', 'contents']\n",
    "print(rel_soup.p)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can use `get_attribute_list` to get a value that’s always a list, string, whether or not it’s a multi-valued atribute\n",
    "\n",
    "    id_soup.p.get_attribute_list(‘id’) # [“my id”]\n",
    "    \n",
    "If you parse a document as XML, there are no multi-valued attributes:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "xml_soup = BeautifulSoup('<p class=\"body strikeout\"></p>', 'xml')\n",
    "xml_soup.p['class']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Comments and other special strings"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`Tag, NavigableString,` and `BeautifulSoup` cover almost everything you’ll see in an HTML or XML file, but there are a few leftover bits. The only one you’ll probably ever need to worry about is the comment:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "markup = \"<b><!--Hey, buddy. Want to buy a used parser?--></b>\"\n",
    "soup = BeautifulSoup(markup)\n",
    "comment = soup.b.string\n",
    "type(comment)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `Comment` object is just a special type of NavigableString:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "comment"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "when it appears as part of an HTML document, a `Comment` is displayed with special formatting:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(soup.b.prettify())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Beautiful Soup defines classes for anything else that might show up in an XML document: `CData, ProcessingInstruction, Declaration,` and `Doctype`. Just like `Comment`, these classes are subclasses of `NavigableString` that add something extra to the string. Here’s an example that replaces the comment with a CDATA block:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from bs4 import CData\n",
    "cdata = CData(\"A CDATA block\")\n",
    "comment.replace_with(cdata)\n",
    "\n",
    "print(soup.b.prettify())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Navigating the tree"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here’s the “Three sisters” HTML document again:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "html_doc = \"\"\"\n",
    "<html><head><title>The Dormouse's story</title></head>\n",
    "<body>\n",
    "<p class=\"title\"><b>The Dormouse's story</b></p>\n",
    "\n",
    "<p class=\"story\">Once upon a time there were three little sisters; and their names were\n",
    "<a href=\"http://example.com/elsie\" class=\"sister\" id=\"link1\">Elsie</a>,\n",
    "<a href=\"http://example.com/lacie\" class=\"sister\" id=\"link2\">Lacie</a> and\n",
    "<a href=\"http://example.com/tillie\" class=\"sister\" id=\"link3\">Tillie</a>;\n",
    "and they lived at the bottom of a well.</p>\n",
    "\n",
    "<p class=\"story\">...</p>\n",
    "\"\"\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from bs4 import BeautifulSoup\n",
    "soup = BeautifulSoup(html_doc, 'html.parser')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "I’ll use this as an example to show you how to move from one part of a document to another."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Going down"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Tags may contain strings and other tags. These elements are the tag’s children. Beautiful Soup provides a lot of different attributes for navigating and iterating over a tag’s children.\n",
    "\n",
    "Note that Beautiful Soup strings don’t support any of these attributes, because a string can’t have children."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Navigating using tag names"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The simplest way to navigate the parse tree is to say the name of the tag you want. If you want the <head> tag, just say `soup.head:`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "soup.head"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "soup.title"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can do use this trick again and again to zoom in on a certain part of the parse tree. This code gets the first `<b>` tag beneath the `<body>` tag:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "soup.body.b"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Using a tag name as an attribute will give you only the *first* tag by that name:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "soup.a"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you need to get all the `<a>` tags, or anything more complicated than the first tag with a certain name, you’ll need to use one of the methods described in Searching the tree, such as *find_all()*:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "soup.find_all('a')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### .contents and .children"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A tag’s children are available in a list called `.contents:`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "head_tag = soup.head\n",
    "head_tag"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "head_tag.contents"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "title_tag = head_tag.contents[0]\n",
    "title_tag"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "title_tag.contents"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `BeautifulSoup` object itself has children. In this case, the `<html>` tag is the child of the `BeautifulSoup` object.:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "len(soup.contents)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "soup.contents[0].name"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A string does not have .contents, because it can’t contain anything:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "text = title_tag.contents[0]\n",
    "text.contents"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Instead of getting them as a list, you can iterate over a tag’s children using the `.children` generator:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "for child in title_tag.children:\n",
    "    print(child)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `.contents` and `.children` attributes only consider a tag’s direct children. For instance, the `<head>` tag has a single direct child–the `<title>` tag:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "head_tag.contents"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "But the `<title>` tag itself has a child: the string “The Dormouse’s story”. There’s a sense in which that string is also a child of the `<head>` tag. The `.descendants` attribute lets you iterate over all of a tag’s children, recursively: its direct children, the children of its direct children, and so on:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "for child in head_tag.descendants:\n",
    "    print(child)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `<head>` tag has only one child, but it has two descendants: the `<title>` tag and the `<title>` tag’s child. The `BeautifulSoup` object only has one direct child (the `<html>` tag), but it has a whole lot of descendants:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "len(list(soup.children))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "len(list(soup.descendants))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If a tag has only one child, and that child is a `NavigableString`, the child is made available as `.string:`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "title_tag.string"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If a tag’s only child is another tag, and that tag has a `.string`, then the parent tag is considered to have the same `.string` as its child:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "head_tag.contents"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "head_tag.string"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If a tag contains more than one thing, then it’s not clear what `.string` should refer to, so `.string` is defined to be `None:`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(soup.html.string)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If there’s more than one thing inside a tag, you can still look at just the strings. Use the `.strings` generator:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "for string in soup.strings:\n",
    "    print(repr(string))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "These strings tend to have a lot of extra whitespace, which you can remove by using the `.stripped_strings` generator instead:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "for string in soup.stripped_strings:\n",
    "    print(repr(string))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can access an element’s parent with the `.parent` attribute. In the example “three sisters” document, the `<head>` tag is the parent of the `<title>` tag:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "title_tag = soup.title\n",
    "title_tag"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "title_tag.parent"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The title string itself has a parent: the `<title>` tag that contains it:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "title_tag.string.parent"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The parent of a top-level tag like `<html>` is the `BeautifulSoup` object itself:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "html_tag = soup.html\n",
    "type(html_tag.parent)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And the `.parent` of a `BeautifulSoup` object is defined as None:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(soup.parent)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can iterate over all of an element’s parents with `.parents`. This example uses `.parents` to travel from an `<a>`  tag buried deep within the document, to the very top of the document:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "link = soup.a\n",
    "link\n",
    "# <a class=\"sister\" href=\"http://example.com/elsie\" id=\"link1\">Elsie</a>\n",
    "for parent in link.parents:\n",
    "    if parent is None:\n",
    "        print(parent)\n",
    "    else:\n",
    "        print(parent.name)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Going sideways"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Consider a simple document like this:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sibling_soup = BeautifulSoup(\"<a><b>text1</b><c>text2</c></b></a>\")\n",
    "print(sibling_soup.prettify())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `<b>` tag and the `<c>` tag are at the same level: they’re both direct children of the same tag. We call them siblings. When a document is pretty-printed, siblings show up at the same indentation level. You can also use this relationship in the code you write."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can use `.next_sibling` and `.previous_sibling` to navigate between page elements that are on the same level of the parse tree:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sibling_soup.b.next_sibling"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sibling_soup.c.previous_sibling"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `<b>` tag has a `.next_sibling`, but no `.previous_sibling`, because there’s nothing before the `<b>` tag on the same level of the tree. For the same reason, the `<c>` tag has a `.previous_sibling` but no `.next_sibling:`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(sibling_soup.b.previous_sibling)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(sibling_soup.c.next_sibling)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The strings “text1” and “text2” are not siblings, because they don’t have the same parent:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sibling_soup.b.string"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(sibling_soup.b.string.next_sibling)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In real documents, the `.next_sibling` or `.previous_sibling` of a tag will usually be a string containing whitespace. Going back to the “three sisters” document:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a href=\"http://example.com/elsie\" class=\"sister\" id=\"link1\">Elsie</a>\n",
    "\n",
    "<a href=\"http://example.com/lacie\" class=\"sister\" id=\"link2\">Lacie</a>\n",
    "\n",
    "<a href=\"http://example.com/tillie\" class=\"sister\" id=\"link3\">Tillie</a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You might think that the .next_sibling of the first `<a>` tag would be the second `<a>` tag. But actually, it’s a string: the comma and newline that separate the first `<a>` tag from the second:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "link = soup.a\n",
    "link"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "link.next_sibling"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The second `<a>` tag is actually the `.next_sibling` of the comma:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "link.next_sibling.next_sibling"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can iterate over a tag’s siblings with `.next_siblings` or `.previous_siblings:`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "for sibling in soup.a.next_siblings:\n",
    "    print(repr(sibling))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "for sibling in soup.find(id=\"link3\").previous_siblings:\n",
    "    print(repr(sibling))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here’s the final `<a>` tag in the “three sisters” document. Its `.next_sibling` is a string: the conclusion of the sentence that was interrupted by the start of the `<a>`  tag.:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "last_a_tag = soup.find(\"a\", id=\"link3\")\n",
    "last_a_tag"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "last_a_tag.next_sibling"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "But the `.next_element` of that `<a>` tag, the thing that was parsed immediately after the `<a>` tag, is not the rest of that sentence: it’s the word “Tillie”:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "last_a_tag.next_element"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "That’s because in the original markup, the word “Tillie” appeared before that semicolon. The parser encountered an `<a>` tag, then the word “Tillie”, then the closing `</a>` tag, then the semicolon and rest of the sentence. The semicolon is on the same level as the `<a>` tag, but the word “Tillie” was encountered first.\n",
    "\n",
    "The `.previous_element` attribute is the exact opposite of `.next_element`. It points to whatever element was parsed immediately before this one:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "last_a_tag.previous_element"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "last_a_tag.previous_element.next_element"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You should get the idea by now. You can use these iterators to move forward or backward in the document as it was parsed:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "for element in last_a_tag.next_elements:\n",
    "    print(repr(element))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Searching the tree"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Beautiful Soup defines a lot of methods for searching the parse tree, but they’re all very similar. I’m going to spend a lot of time explaining the two most popular methods: `find()` and `find_all()`. The other methods take almost exactly the same arguments, so I’ll just cover them briefly.\n",
    "\n",
    "Once again, I’ll be using the “three sisters” document as an example:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "html_doc = \"\"\"\n",
    "<html><head><title>The Dormouse's story</title></head>\n",
    "<body>\n",
    "<p class=\"title\"><b>The Dormouse's story</b></p>\n",
    "\n",
    "<p class=\"story\">Once upon a time there were three little sisters; and their names were\n",
    "<a href=\"http://example.com/elsie\" class=\"sister\" id=\"link1\">Elsie</a>,\n",
    "<a href=\"http://example.com/lacie\" class=\"sister\" id=\"link2\">Lacie</a> and\n",
    "<a href=\"http://example.com/tillie\" class=\"sister\" id=\"link3\">Tillie</a>;\n",
    "and they lived at the bottom of a well.</p>\n",
    "\n",
    "<p class=\"story\">...</p>\n",
    "\"\"\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from bs4 import BeautifulSoup\n",
    "soup = BeautifulSoup(html_doc, 'html.parser')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "By passing in a filter to an argument like `find_all()`, you can zoom in on the parts of the document you’re interested in."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Kinds of filters"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Before talking in detail about find_all() and similar methods, I want to show examples of different filters you can pass into these methods. These filters show up again and again, throughout the search API. You can use them to filter based on a tag’s name, on its attributes, on the text of a string, or on some combination of these."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### A string"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The simplest filter is a string. Pass a string to a search method and Beautiful Soup will perform a match against that exact string. This code finds all the `<b>` tags in the document:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "soup.find_all('b')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you pass in a byte string, Beautiful Soup will assume the string is encoded as UTF-8. You can avoid this by passing in a Unicode string instead."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### A regular expression"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you pass in a regular expression object, Beautiful Soup will filter against that regular expression using its `search()` method. This code finds all the tags whose names start with the letter “b”; in this case, the `<body>` tag and the `<b>` tag:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import re\n",
    "for tag in soup.find_all(re.compile(\"^b\")):\n",
    "    print(tag.name)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### A regular expression"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you pass in a regular expression object, Beautiful Soup will filter against that regular expression using its `search()` method. This code finds all the tags whose names start with the letter “b”; in this case, the `<body>` tag and the `<b>` tag:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import re\n",
    "for tag in soup.find_all(re.compile(\"^b\")):\n",
    "    print(tag.name)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This code finds all the tags whose names contain the letter ‘t’:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "for tag in soup.find_all(re.compile(\"t\")):\n",
    "    print(tag.name)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Alist"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you pass in a list, Beautiful Soup will allow a string match against any item in that list. This code finds all the `<a>` tags and all the `<b>` tags:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "soup.find_all([\"a\", \"b\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The value `True` matches everything it can. This code finds all the tags in the document, but none of the text strings:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "for tag in soup.find_all(True):\n",
    "    print(tag.name)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### A function"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If none of the other matches work for you, define a function that takes an element as its only argument. The function should return `True` if the argument matches, and `False` otherwise.\n",
    "\n",
    "Here’s a function that returns `True` if a tag defines the “class” attribute but doesn’t define the “id” attribute:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def has_class_but_no_id(tag):\n",
    "    return tag.has_attr('class') and not tag.has_attr('id')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Pass this function into `find_all()` and you’ll pick up all the `<p>` tags:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "soup.find_all(has_class_but_no_id)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This function only picks up the `<p>` tags. It doesn’t pick up the `<a>` tags, because those tags define both “class” and “id”. It doesn’t pick up tags like `<html>` and `<title>`, because those tags don’t define “class”.\n",
    "\n",
    "If you pass in a function to filter on a specific attribute like `href`, the argument passed into the function will be the attribute value, not the whole tag. Here’s a function that finds all `a` tags whose `href` attribute does not match a regular expression:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def not_lacie(href):\n",
    "    return href and not re.compile(\"lacie\").search(href)\n",
    "soup.find_all(href=not_lacie)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The function can be as complicated as you need it to be. Here’s a function that returns `True` if a tag is surrounded by string objects:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from bs4 import NavigableString\n",
    "def surrounded_by_strings(tag):\n",
    "    return (isinstance(tag.next_element, NavigableString)\n",
    "            and isinstance(tag.previous_element, NavigableString))\n",
    "\n",
    "for tag in soup.find_all(surrounded_by_strings):\n",
    "    print (tag.name)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### The name argument"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Pass in a value for `name` and you’ll tell Beautiful Soup to only consider tags with certain names. Text strings will be ignored, as will tags whose names that don’t match.\n",
    "\n",
    "This is the simplest usage:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "soup.find_all(\"title\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Recall from Kinds of filters that the value to `name` can be a string, a regular expression, a list, a function, or the value True."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### The keyword arguments"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Any argument that’s not recognized will be turned into a filter on one of a tag’s attributes. If you pass in a value for an argument called `id,` Beautiful Soup will filter against each tag’s ‘id’ attribute:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "soup.find_all(id='link2')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you pass in a value for `href`, Beautiful Soup will filter against each tag’s ‘href’ attribute:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "soup.find_all(href=re.compile(\"elsie\"))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can filter an attribute based on a string, a regular expression, a list, a function, or the value True.\n",
    "This code finds all tags whose `id` attribute has a value, regardless of what the value is:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "soup.find_all(id=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can filter multiple attributes at once by passing in more than one keyword argument:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "soup.find_all(href=re.compile(\"elsie\"), id='link1')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Some attributes, like the  $data-*$ attributes in HTML 5, have names that <font color = 'red'>can’t</font> be used as the names of keyword arguments:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "data_soup = BeautifulSoup('<div data-foo=\"value\">foo!</div>')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "data_soup.find_all(data-foo=\"value\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can use these attributes in searches by putting them into a dictionary and passing the dictionary into `find_all()` as the `attrs` argument:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "data_soup.find_all(attrs={\"data-foo\": \"value\"})"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can’t use a keyword argument to search for HTML’s ‘name’ element, because Beautiful Soup uses the `name` argument to contain the name of the tag itself. Instead, you can give a value to ‘name’ in the `attrs` argument."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Searching by CSS class"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It’s very useful to search for a tag that has a certain CSS class, but the name of the CSS attribute, “class”, is a reserved word in Python. Using `class` as a keyword argument will give you a syntax error. As of Beautiful Soup 4.1.2, you can search by CSS class using the keyword argument `class_:`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "soup.find_all(\"a\", class_=\"sister\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As with any keyword argument, you can pass `class_` a string, a regular expression, a function, or `True`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "soup.find_all(class_=re.compile(\"itl\"))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def has_six_characters(css_class):\n",
    "    return css_class is not None and len(css_class) == 6"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "soup.find_all(class_=has_six_characters)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Remember that a single tag can have multiple values for its “class” attribute. When you search for a tag that matches a certain CSS class, you’re matching against any of its CSS classes:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "css_soup = BeautifulSoup('<p class=\"body strikeout\"></p>')\n",
    "css_soup.find_all(\"p\", class_=\"strikeout\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "css_soup.find_all(\"p\", class_=\"body\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can also search for the exact string value of the `class` attribute:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "css_soup.find_all(\"p\", class_=\"body strikeout\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "But searching for variants of the string value won’t work:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "css_soup.find_all(\"p\", class_=\"strikeout body\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you want to search for tags that match two or more CSS classes, you should use a CSS selector:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "css_soup.select(\"p.strikeout.body\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In older versions of Beautiful Soup, which don’t have the `class_` shortcut, you can use the `attrs` trick mentioned above. Create a dictionary whose value for “class” is the string (or regular expression, or whatever) you want to search for:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "soup.find_all(\"a\", attrs={\"class\": \"sister\"})"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### The `string` argument"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "With `string` you can search for strings instead of tags. As with `name` and the keyword arguments, you can pass in a string, a regular expression, a list, a function, or the value True. Here are some examples:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "soup.find_all(string='Elsie')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "soup.find_all(string=[\"Tillie\", \"Elsie\", \"Lacie\"])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "soup.find_all(string=re.compile(\"Dormouse\"))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def is_the_only_string_within_a_tag(s):\n",
    "    \"\"\"Return True if this string is the only child of its parent tag.\"\"\"\n",
    "    return (s == s.parent.string)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "soup.find_all(string=is_the_only_string_within_a_tag)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Although `string` is for finding strings, you can combine it with arguments that find tags: Beautiful Soup will find all tags whose `.string` matches your value for `string.` This code finds the `<a>` tags whose `.string` is “Elsie”:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "soup.find_all(\"a\", string=\"Elsie\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `string` argument is new in Beautiful Soup 4.4.0. In earlier versions it was called `text:`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "soup.find_all(\"a\", text=\"Elsie\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### The `limit` argument"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`find_all()` returns all the tags and strings that match your filters. This can take a while if the document is large. If you don’t need all the results, you can pass in a number for `limit.` This works just like the LIMIT keyword in SQL. It tells Beautiful Soup to stop gathering results after it’s found a certain number.\n",
    "\n",
    "There are three links in the “three sisters” document, but this code only finds the first two:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "soup.find_all(\"a\", limit=2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### The `recursive` argument"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you call `mytag.find_all()`, Beautiful Soup will examine all the descendants of `mytag:` its children, its children’s children, and so on. If you only want Beautiful Soup to consider direct children, you can pass in `recursive=False.` See the difference here:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "soup.html.find_all(\"title\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "soup.html.find_all(\"title\", recursive=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here’s that part of the document:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<html>\n",
    " <head>\n",
    "  <title>\n",
    "   The Dormouse's story\n",
    "  </title>\n",
    " </head>\n",
    "..."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `<title>` tag is beneath the `<html>` tag, but it’s not directly beneath the `<html>` tag: the `<head>` tag is in the way. Beautiful Soup finds the `<title>` tag when it’s allowed to look at all descendants of the `<html>` tag, but when `recursive=False` restricts it to the `<html>` tag’s immediate children, it finds nothing.\n",
    "\n",
    "Beautiful Soup offers a lot of tree-searching methods (covered below), and they mostly take the same arguments as `find_all(): name, attrs, string, limit,`  and the keyword arguments. But the `recursive` argument is different: `find_all()` and `find()` are the only methods that support it. Passing `recursive=False` into a method like `find_parents()` wouldn’t be very useful."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Calling a tag is like calling `find_all()`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Because `find_all()` is the most popular method in the Beautiful Soup search API, you can use a shortcut for it. If you treat the `BeautifulSoup` object or a `Tag` object as though it were a function, then it’s the same as calling `find_all()` on that object. These two lines of code are equivalent:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "soup.find_all(\"a\")\n",
    "soup(\"a\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "These two lines are also equivalent:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "soup.title.find_all(string=True)\n",
    "soup.title(string=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`find()`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Signature: find(name, attrs, recursive, string, **kwargs)\n",
    "\n",
    "The `find_all()` method scans the entire document looking for results, but sometimes you only want to find one result. If you know a document only has one `<body>` tag, it’s a waste of time to scan the entire document looking for more. Rather than passing in `limit=1` every time you call `find_all`, you can use the `find()` method. These two lines of code are nearly equivalent:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "soup.find_all('title', limit=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "soup.find('title')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The only difference is that `find_all()` returns a list containing the single result, and `find()` just returns the result.\n",
    "\n",
    "If `find_all()` can’t find anything, it returns an empty list. If `find()` can’t find anything, it returns `None:`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(soup.find(\"nosuchtag\"))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Remember the `soup.head.title` trick from Navigating using tag names? That trick works by repeatedly calling `find():`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "soup.head.title"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "soup.find(\"head\").find(\"title\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### `find_parents()` and `find_parent()`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Signature: find_parents(name, attrs, string, limit, **kwargs)\n",
    "\n",
    "Signature: find_parent(name, attrs, string, **kwargs)\n",
    "\n",
    "I spent a lot of time above covering `find_all()` and `find()`. The Beautiful Soup API defines ten other methods for searching the tree, but don’t be afraid. Five of these methods are basically the same as `find_all()`, and the other five are basically the same as `find()`. The only differences are in what parts of the tree they search.\n",
    "\n",
    "First let’s consider `find_parents()` and `find_parent()`. Remember that `find_all()` and `find()` work their way down the tree, looking at tag’s descendants. These methods do the opposite: they work their way up the tree, looking at a tag’s (or a string’s) parents. Let’s try them out, starting from a string buried deep in the “three daughters” document:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "a_string = soup.find(string=\"Lacie\")\n",
    "a_string"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "a_string.find_parents(\"a\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "a_string.find_parent(\"p\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "One of the three `<a>` tags is the direct parent of the string in question, so our search finds it. One of the three `<p>` tags is an indirect parent of the string, and our search finds that as well. There’s a `<p>` tag with the CSS class “title” somewhere in the document, but it’s not one of this string’s parents, so we can’t find it with `find_parents()`.\n",
    "\n",
    "You may have made the connection between `find_parent()` and `find_parents()`, and the `.parent` and `.parents` attributes mentioned earlier. The connection is very strong. These search methods actually use `.parents` to iterate over all the parents, and check each one against the provided filter to see if it matches."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### `find_next_siblings()` and `find_next_sibling()`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Signature: find_next_siblings(name, attrs, string, limit, **kwargs)\n",
    "\n",
    "Signature: find_next_sibling(name, attrs, string, **kwargs)\n",
    "\n",
    "These methods use .next_siblings to iterate over the rest of an element’s siblings in the tree. The `find_next_siblings()` method returns all the siblings that match, and `find_next_sibling()` only returns the first one:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "first_link = soup.a\n",
    "first_link"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "first_link.find_next_siblings(\"a\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "first_story_paragraph = soup.find(\"p\", \"story\")\n",
    "first_story_paragraph.find_next_sibling(\"p\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### `find_previous_siblings()` and `find_previous_sibling()`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Signature: find_previous_siblings(name, attrs, string, limit, **kwargs)\n",
    "\n",
    "Signature: find_previous_sibling(name, attrs, string, **kwargs)\n",
    "\n",
    "These methods use `.previous_siblings` to iterate over an element’s siblings that precede it in the tree. The `find_previous_siblings()` method returns all the siblings that match, and `find_previous_sibling()` only returns the first one:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "last_link = soup.find(\"a\", id=\"link3\")\n",
    "last_link"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "last_link.find_previous_siblings(\"a\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "first_story_paragraph = soup.find(\"p\", \"story\")\n",
    "first_story_paragraph.find_previous_sibling(\"p\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### `find_all_next()` and `find_next()`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Signature: find_all_next(name, attrs, string, limit, **kwargs)\n",
    "\n",
    "Signature: find_next(name, attrs, string, **kwargs)\n",
    "\n",
    "These methods use `.next_elements` to iterate over whatever tags and strings that come after it in the document. The `find_all_next()` method returns all matches, and `find_next()` only returns the first match:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "first_link = soup.a\n",
    "first_link"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "first_link.find_all_next(string=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "first_link.find_next(\"p\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### `find_all_previous()` and `find_previous()`\n",
    "\n",
    "Signature: find_all_previous(name, attrs, string, limit, **kwargs)\n",
    "\n",
    "Signature: find_previous(name, attrs, string, **kwargs)\n",
    "\n",
    "These methods use `.previous_elements` to iterate over the tags and strings that came before it in the document. The `find_all_previous()`  method returns all matches, and `find_previous()` only returns the first match:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "first_link = soup.a\n",
    "first_link"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "first_link.find_all_previous(\"p\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "first_link.find_previous(\"title\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### CSS selectors"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Beautiful Soup supports the most commonly-used CSS selectors. Just pass a string into the `.select()` method of a `Tag` object or the `BeautifulSoup` object itself.\n",
    "\n",
    "You can find tags:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "soup.select(\"title\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "soup.select(\"p:nth-of-type(3)\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Find tags beneath other tags:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "soup.select(\"body a\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "soup.select(\"html head title\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Find tags directly beneath other tags:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "soup.select(\"head > title\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "soup.select(\"p > a\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "soup.select(\"p > a:nth-of-type(2)\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "soup.select(\"p > #link1\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "soup.select(\"body > a\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Find the siblings of tags:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "soup.select(\"#link1 ~ .sister\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "soup.select(\"#link1 + .sister\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Find tags by CSS class:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "soup.select(\".sister\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "soup.select(\"[class~=sister]\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Find tags by ID:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "soup.select(\"#link1\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "soup.select(\"a#link2\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Test for the existence of an attribute:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "soup.select('a[href]')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Find tags by attribute value:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "soup.select('a[href=\"http://example.com/elsie\"]')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "soup.select('a[href^=\"http://example.com/\"]')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "soup.select('a[href$=\"tillie\"]')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "soup.select('a[href*=\".com/el\"]')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Match language codes:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "multilingual_markup = \"\"\"\n",
    " <p lang=\"en\">Hello</p>\n",
    " <p lang=\"en-us\">Howdy, y'all</p>\n",
    " <p lang=\"en-gb\">Pip-pip, old fruit</p>\n",
    " <p lang=\"fr\">Bonjour mes amis</p>\n",
    "\"\"\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "multilingual_soup = BeautifulSoup(multilingual_markup)\n",
    "multilingual_soup.select('p[lang|=en]')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Find only the first tag that matches a selector:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "soup.select_one(\".sister\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This is all a convenience for users who know the CSS selector syntax. You can do all this stuff with the Beautiful Soup API. And if CSS selectors are all you need, you might as well use lxml directly: it’s a lot faster, and it supports more CSS selectors. But this lets you combine simple CSS selectors with the Beautiful Soup API."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Modifying the tree"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Beautiful Soup’s main strength is in searching the parse tree, but you can also modify the tree and write your changes as a new HTML or XML document."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Changing tag names and attributes"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "I covered this earlier, in `Attributes`, but it bears repeating. You can rename a tag, change the values of its attributes, add new attributes, and delete attributes:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "soup = BeautifulSoup('<b class=\"boldest\">Extremely bold</b>')\n",
    "tag = soup.b\n",
    "\n",
    "tag.name = \"blockquote\"\n",
    "tag['class'] = 'verybold'\n",
    "tag['id'] = 1\n",
    "tag"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "del tag['class']\n",
    "del tag['id']\n",
    "tag"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Modifying `.string`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you set a tag’s `.string` attribute, the tag’s contents are replaced with the string you give:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "markup = '<a href=\"http://example.com/\">I linked to <i>example.com</i></a>'\n",
    "soup = BeautifulSoup(markup)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "tag = soup.a\n",
    "tag.string = \"New link text.\"\n",
    "tag"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<font color = 'red'>Be careful:</font> if the tag contained other tags, they and all their contents will be destroyed."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### `append()`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can add to a tag’s contents with `Tag.append()`. It works just like calling `.append()` on a Python list:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "soup = BeautifulSoup(\"<a>Foo</a>\")\n",
    "soup.a.append(\"Bar\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "soup"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "soup.a.contents"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### `NavigableString()` and `.new_tag()`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you need to add a string to a document, no problem–you can pass a Python string in to `append()`, or you can call the `NavigableString` constructor:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "soup = BeautifulSoup(\"<b></b>\")\n",
    "tag = soup.b\n",
    "tag.append(\"Hello\")\n",
    "new_string = NavigableString(\" there\")\n",
    "tag.append(new_string)\n",
    "tag"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "tag.contents"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you want to create a comment or some other subclass of `NavigableString`, just call the constructor:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from bs4 import Comment\n",
    "new_comment = Comment(\"Nice to see you.\")\n",
    "tag.append(new_comment)\n",
    "tag"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "tag.contents"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### `insert()`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`Tag.insert()` is just like `Tag.append()`, except the new element doesn’t necessarily go at the end of its parent’s `.contents`. It’ll be inserted at whatever numeric position you say. It works just like `.insert()` on a Python list:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "markup = '<a href=\"http://example.com/\">I linked to <i>example.com</i></a>'\n",
    "soup = BeautifulSoup(markup)\n",
    "tag = soup.a"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "tag.insert(1, \"but did not endorse \")\n",
    "tag"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### `insert_before()` and `insert_after()`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `insert_before()` method inserts a tag or string immediately before something else in the parse tree:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "soup = BeautifulSoup(\"<b>stop</b>\")\n",
    "tag = soup.new_tag(\"i\")\n",
    "tag.string = \"Don't\"\n",
    "soup.b.string.insert_before(tag)\n",
    "soup.b"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `insert_after()` method moves a tag or string so that it immediately follows something else in the parse tree:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "soup.b.i.insert_after(soup.new_string(\" ever \"))\n",
    "soup.b"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "soup.b.contents"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### `clear()`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`Tag.clear()` removes the contents of a tag:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "markup = '<a href=\"http://example.com/\">I linked to <i>example.com</i></a>'\n",
    "soup = BeautifulSoup(markup)\n",
    "tag = soup.a\n",
    "\n",
    "tag.clear()\n",
    "tag"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### `extract()`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`PageElement.extract()` removes a tag or string from the tree. It returns the tag or string that was extracted:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "markup = '<a href=\"http://example.com/\">I linked to <i>example.com</i></a>'\n",
    "soup = BeautifulSoup(markup)\n",
    "a_tag = soup.a\n",
    "\n",
    "i_tag = soup.i.extract()\n",
    "\n",
    "a_tag"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "i_tag"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "a_tag"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(i_tag.parent)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "At this point you effectively have two parse trees: one rooted at the `BeautifulSoup` object you used to parse the document, and one rooted at the tag that was extracted. You can go on to call `extract` on a child of the element you extracted:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "my_string = i_tag.string.extract()\n",
    "my_string"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(my_string.parent)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "i_tag"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### `decompose()`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`Tag.decompose()` removes a tag from the tree, then completely destroys it and its contents:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "markup = '<a href=\"http://example.com/\">I linked to <i>example.com</i></a>'\n",
    "soup = BeautifulSoup(markup)\n",
    "a_tag = soup.a\n",
    "\n",
    "soup.i.decompose()\n",
    "\n",
    "a_tag"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### `replace_with()`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`PageElement.replace_with()` removes a tag or string from the tree, and replaces it with the tag or string of your choice:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "markup = '<a href=\"http://example.com/\">I linked to <i>example.com</i></a>'\n",
    "soup = BeautifulSoup(markup)\n",
    "a_tag = soup.a\n",
    "\n",
    "new_tag = soup.new_tag(\"b\")\n",
    "new_tag.string = \"example.net\"\n",
    "a_tag.i.replace_with(new_tag)\n",
    "\n",
    "a_tag"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`replace_with()` returns the tag or string that was replaced, so that you can examine it or add it back to another part of the tree."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### `wrap()`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`PageElement.wrap()` wraps an element in the tag you specify. It returns the new wrapper:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "soup = BeautifulSoup(\"<p>I wish I was bold.</p>\")\n",
    "soup.p.string.wrap(soup.new_tag(\"b\"))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "soup.p.wrap(soup.new_tag(\"div\"))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### `unwrap()`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`Tag.unwrap()` is the opposite of `wrap()`. It replaces a tag with whatever’s inside that tag. It’s good for stripping out markup:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "markup = '<a href=\"http://example.com/\">I linked to <i>example.com</i></a>'\n",
    "soup = BeautifulSoup(markup)\n",
    "a_tag = soup.a\n",
    "\n",
    "a_tag.i.unwrap()\n",
    "a_tag"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Output"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Pretty-printing"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `prettify()` method will turn a Beautiful Soup parse tree into a nicely formatted Unicode string, with each HTML/XML tag on its own line:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "markup = '<a href=\"http://example.com/\">I linked to <i>example.com</i></a>'\n",
    "soup = BeautifulSoup(markup)\n",
    "soup.prettify()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(soup.prettify())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can call `prettify()` on the top-level `BeautifulSoup` object, or on any of its `Tag` objects:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(soup.a.prettify())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Non-pretty printing"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you just want a string, with no fancy formatting, you can call `unicode()` or `str()` on a `BeautifulSoup` object, or a `Tag` within it:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "str(soup)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Output formatters\n",
    "\n",
    "If you give Beautiful Soup a document that contains HTML entities like “&lquot;”, they’ll be converted to Unicode characters:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "soup = BeautifulSoup(\"&ldquo;Dammit!&rdquo; he said.\")\n",
    "str(soup)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "By default, the only characters that are escaped upon output are bare ampersands and angle brackets. These get turned into $“&amp;”$, $“&lt;”$, and $“&gt;”$, so that Beautiful Soup doesn’t inadvertently generate invalid HTML or XML:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "soup = BeautifulSoup(\"<p>The law firm of Dewey, Cheatem, & Howe</p>\")\n",
    "soup.p"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "soup = BeautifulSoup('<a href=\"http://example.com/?foo=val1&bar=val2\">A link</a>')\n",
    "soup.a"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can change this behavior by providing a value for the `formatter` argument to `prettify(), encode(),` or `decode()`. Beautiful Soup recognizes four possible values for `formatter`.\n",
    "\n",
    "The default is `formatter=\"minimal\"`. Strings will only be processed enough to ensure that Beautiful Soup generates valid HTML/XML:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "french = \"<p>Il a dit &lt;&lt;Sacr&eacute; bleu!&gt;&gt;</p>\"\n",
    "soup = BeautifulSoup(french)\n",
    "print(soup.prettify(formatter=\"minimal\"))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you pass in `formatter=\"html\"`, Beautiful Soup will convert Unicode characters to HTML entities whenever possible:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(soup.prettify(formatter=\"html\"))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you pass in `formatter=None`, Beautiful Soup will not modify strings at all on output. This is the fastest option, but it may lead to Beautiful Soup generating invalid HTML/XML, as in these examples:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(soup.prettify(formatter=None))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "link_soup = BeautifulSoup('<a href=\"http://example.com/?foo=val1&bar=val2\">A link</a>')\n",
    "print(link_soup.a.encode(formatter=None))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finally, if you pass in a function for `formatter`, Beautiful Soup will call that function once for every string and attribute value in the document. You can do whatever you want in this function. Here’s a formatter that converts strings to uppercase and does absolutely nothing else:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def uppercase(str):\n",
    "    return str.upper()\n",
    "\n",
    "print(soup.prettify(formatter=uppercase))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(link_soup.a.prettify(formatter=uppercase))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here’s an example that replaces Unicode characters with HTML entities whenever possible, but also converts all strings to uppercase:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from bs4.dammit import EntitySubstitution\n",
    "def uppercase_and_substitute_html_entities(str):\n",
    "    return EntitySubstitution.substitute_html(str.upper())\n",
    "\n",
    "print(soup.prettify(formatter=uppercase_and_substitute_html_entities))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "One last caveat: if you create a `CData` object, the text inside that object is always presented exactly as it appears, with no formatting. Beautiful Soup will call the formatter method, just in case you’ve written a custom method that counts all the strings in the document or something, but it will ignore the return value:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from bs4.element import CData\n",
    "soup = BeautifulSoup(\"<a></a>\")\n",
    "soup.a.string = CData(\"one < three\")\n",
    "print(soup.a.prettify(formatter=\"xml\"))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### get_text()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you only want the text part of a document or tag, you can use the `get_text()` method. It returns all the text in a document or beneath a tag, as a single Unicode string:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "markup = '<a href=\"http://example.com/\">\\nI linked to <i>example.com</i>\\n</a>'\n",
    "soup = BeautifulSoup(markup)\n",
    "\n",
    "soup.get_text()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "soup.i.get_text()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can specify a string to be used to join the bits of text together:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "soup.get_text(\"|\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can tell Beautiful Soup to strip whitespace from the beginning and end of each bit of text:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "soup.get_text(\"|\", strip = True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "But at that point you might want to use the .stripped_strings generator instead, and process the text yourself:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "[text for text in soup.stripped_strings]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Differences between parsers"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Beautiful Soup presents the same interface to a number of different parsers, but each parser is different. Different parsers will create different parse trees from the same document. The biggest differences are between the HTML parsers and the XML parsers. Here’s a short document, parsed as HTML:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "BeautifulSoup(\"<a><b /></a>\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Since an empty `<b />` tag is not valid HTML, the parser turns it into a `<b></b>` tag pair.\n",
    "\n",
    "Here’s the same document parsed as XML (running this requires that you have lxml installed). Note that the empty `<b />` tag is left alone, and that the document is given an XML declaration instead of being put into an `<html>` tag.:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "BeautifulSoup(\"<a><b /></a>\", \"xml\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There are also differences between HTML parsers. If you give Beautiful Soup a perfectly-formed HTML document, these differences won’t matter. One parser will be faster than another, but they’ll all give you a data structure that looks exactly like the original HTML document.\n",
    "\n",
    "But if the document is not perfectly-formed, different parsers will give different results. Here’s a short, invalid document parsed using lxml’s HTML parser. Note that the dangling </p> tag is simply ignored:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "BeautifulSoup(\"<a></p>\", \"lxml\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here’s the same document parsed using html5lib:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "BeautifulSoup(\"<a></p>\", \"html5lib\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Instead of ignoring the dangling `</p>` tag, html5lib pairs it with an opening `<p>` tag. This parser also adds an empty `<head>` tag to the document.\n",
    "\n",
    "Here’s the same document parsed with Python’s built-in HTML parser:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "BeautifulSoup(\"<a></p>\", \"html.parser\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Encodings"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Any HTML or XML document is written in a specific encoding like ASCII or UTF-8. But when you load that document into Beautiful Soup, you’ll discover it’s been converted to Unicode:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "markup = \"<h1>Sacr\\xc3\\xa9 bleu!</h1>\"\n",
    "soup = BeautifulSoup(markup)\n",
    "soup.h1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "soup.h1.string"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It’s not magic. (That sure would be nice.) Beautiful Soup uses a sub-library called Unicode, Dammit to detect a document’s encoding and convert it to Unicode. The autodetected encoding is available as the `.original_encoding` attribute of the `BeautifulSoup` object:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "soup.original_encoding"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Unicode, Dammit guesses correctly most of the time, but sometimes it makes mistakes. Sometimes it guesses correctly, but only after a byte-by-byte search of the document that takes a very long time. If you happen to know a document’s encoding ahead of time, you can avoid mistakes and delays by passing it to the `BeautifulSoup` constructor as `from_encoding.`\n",
    "\n",
    "Here’s a document written in ISO-8859-8. The document is so short that Unicode, Dammit can’t get a good lock on it, and misidentifies it as ISO-8859-7:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "markup = b\"<h1>\\xed\\xe5\\xec\\xf9</h1>\"\n",
    "soup = BeautifulSoup(markup)\n",
    "soup.h1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "soup.original_encoding"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can fix this by passing in the correct `from_encoding:`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "soup = BeautifulSoup(markup, from_encoding=\"iso-8859-8\")\n",
    "soup.h1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "soup.original_encoding"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you don’t know what the correct encoding is, but you know that Unicode, Dammit is guessing wrong, you can pass the wrong guesses in as `exclude_encodings:`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "soup = BeautifulSoup(markup, exclude_encodings=[\"ISO-8859-7\"])\n",
    "soup.h1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "soup.original_encoding"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Output encoding"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "When you write out a document from Beautiful Soup, you get a UTF-8 document, even if the document wasn’t in UTF-8 to begin with. Here’s a document written in the Latin-1 encoding:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "markup = b'''\n",
    " <html>\n",
    "  <head>\n",
    "   <meta content=\"text/html; charset=ISO-Latin-1\" http-equiv=\"Content-type\" />\n",
    "  </head>\n",
    "  <body>\n",
    "   <p>Sacr\\xe9 bleu!</p>\n",
    "  </body>\n",
    " </html>\n",
    "'''"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "soup = BeautifulSoup(markup)\n",
    "print(soup.prettify())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note that the `<meta>` tag has been rewritten to reflect the fact that the document is now in UTF-8.\n",
    "\n",
    "If you don’t want UTF-8, you can pass an encoding into `prettify():`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(soup.prettify(\"latin-1\"))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can also call encode() on the `BeautifulSoup` object, or any element in the soup, just as if it were a Python string:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "soup.p.encode(\"latin-1\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "soup.p.encode(\"utf-8\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Any characters that can’t be represented in your chosen encoding will be converted into numeric XML entity references. Here’s a document that includes the Unicode character SNOWMAN:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "markup = u\"<b>\\N{SNOWMAN}</b>\"\n",
    "snowman_soup = BeautifulSoup(markup)\n",
    "tag = snowman_soup.b"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The SNOWMAN character can be part of a UTF-8 document (it looks like ☃), but there’s no representation for that character in ISO-Latin-1 or ASCII, so it’s converted into “&#9731” for those encodings:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(tag.encode(\"utf-8\"))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(tag.encode(\"latin-1\"))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(tag.encode(\"ascii\"))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Unicode, Dammit"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can use Unicode, Dammit without using Beautiful Soup. It’s useful whenever you have data in an unknown encoding and you just want it to become Unicode:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from bs4 import UnicodeDammit\n",
    "dammit = UnicodeDammit(\"Sacr\\xc3\\xa9 bleu!\")\n",
    "print(dammit.unicode_markup)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "dammit.original_encoding"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Unicode, Dammit’s guesses will get a lot more accurate if you install the `chardet` or `cchardet` Python libraries. The more data you give Unicode, Dammit, the more accurately it will guess. If you have your own suspicions as to what the encoding might be, you can pass them in as a list:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "dammit = UnicodeDammit(\"Sacr\\xe9 bleu!\", [\"latin-1\", \"iso-8859-1\"])\n",
    "print(dammit.unicode_markup)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "dammit.original_encoding"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Smart quotes"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can use Unicode, Dammit to convert Microsoft smart quotes to HTML or XML entities:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "markup = b\"<p>I just \\x93love\\x94 Microsoft Word\\x92s smart quotes</p>\"\n",
    "\n",
    "UnicodeDammit(markup, [\"windows-1252\"], smart_quotes_to=\"html\").unicode_markup"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "UnicodeDammit(markup, [\"windows-1252\"], smart_quotes_to=\"xml\").unicode_markup"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can also convert Microsoft smart quotes to ASCII quotes:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "UnicodeDammit(markup, [\"windows-1252\"], smart_quotes_to=\"ascii\").unicode_markup"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Hopefully you’ll find this feature useful, but Beautiful Soup doesn’t use it. Beautiful Soup prefers the default behavior, which is to convert Microsoft smart quotes to Unicode characters along with everything else:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "UnicodeDammit(markup, [\"windows-1252\"]).unicode_markup"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Inconsistent encodings"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Sometimes a document is mostly in UTF-8, but contains Windows-1252 characters such as (again) Microsoft smart quotes. This can happen when a website includes data from multiple sources. You can use `UnicodeDammit.detwingle()` to turn such a document into pure UTF-8. Here’s a simple example:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "snowmen = (u\"\\N{SNOWMAN}\" * 3)\n",
    "quote = (u\"\\N{LEFT DOUBLE QUOTATION MARK}I like snowmen!\\N{RIGHT DOUBLE QUOTATION MARK}\")\n",
    "doc = snowmen.encode(\"utf8\") + quote.encode(\"windows_1252\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This document is a mess. The snowmen are in UTF-8 and the quotes are in Windows-1252. You can display the snowmen or the quotes, but not both:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(doc)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(doc.decode(\"windows-1252\"))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "UnicodeDammit.detwingle() only knows how to handle Windows-1252 embedded in UTF-8 (or vice versa, I suppose), but this is the most common case.\n",
    "\n",
    "\n",
    "Note that you must know to call `UnicodeDammit.detwingle()` on your data before passing it into `BeautifulSoup` or the `UnicodeDammit` constructor. Beautiful Soup assumes that a document has a single encoding, whatever it might be. If you pass it a document that contains both UTF-8 and Windows-1252, it’s likely to think the whole document is Windows-1252, and the document will come out looking like `â˜ƒâ˜ƒâ˜ƒ“I like snowmen!”`.\n",
    "\n",
    "`UnicodeDammit.detwingle()` is new in Beautiful Soup 4.1.0."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Comparing objects for equality"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Beautiful Soup says that two `NavigableString` or `Tag` objects are equal when they represent the same HTML or XML markup. In this example, the two `<b>` tags are treated as equal, even though they live in different parts of the object tree, because they both look like `“<b>pizza</b>”`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "markup = \"<p>I want <b>pizza</b> and more <b>pizza</b>!</p>\"\n",
    "soup = BeautifulSoup(markup, 'html.parser')\n",
    "first_b, second_b = soup.find_all('b')\n",
    "print (first_b == second_b)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print (first_b.previous_element == second_b.previous_element)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you want to see whether two variables refer to exactly the same object, use is:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(first_b is second_b)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Copying Beautiful Soup objects"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can use `copy.copy()` to create a copy of any `Tag` or `NavigableString:`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import copy\n",
    "p_copy = copy.copy(soup.p)\n",
    "print(p_copy)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The copy is considered equal to the original, since it represents the same markup as the original, but it’s not the same object:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print (soup.p == p_copy)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print (soup.p is p_copy)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The only real difference is that the copy is completely detached from the original Beautiful Soup object tree, just as if `extract()` had been called on it:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print (p_copy.parent)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This is because two different `Tag` objects can’t occupy the same space at the same time."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Parsing only part of a document"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let’s say you want to use Beautiful Soup look at a document’s `<a>` tags. It’s a waste of time and memory to parse the entire document and then go over it again looking for `<a>` tags. It would be much faster to ignore everything that wasn’t an `<a>` tag in the first place. The `SoupStrainer` class allows you to choose which parts of an incoming document are parsed. You just create a `SoupStrainer` and pass it in to the `BeautifulSoup` constructor as the `parse_only` argument.\n",
    "\n",
    "(Note that this feature won’t work if you’re using the html5lib parser. If you use html5lib, the whole document will be parsed, no matter what. This is because html5lib constantly rearranges the parse tree as it works, and if some part of the document didn’t actually make it into the parse tree, it’ll crash. To avoid confusion, in the examples below I’ll be forcing Beautiful Soup to use Python’s built-in parser.)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### `SoupStrainer`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `SoupStrainer` class takes the same arguments as a typical method from Searching the tree: `name, attrs, string,` and `**kwargs`. Here are three `SoupStrainer` objects:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from bs4 import SoupStrainer\n",
    "\n",
    "only_a_tags = SoupStrainer(\"a\")\n",
    "\n",
    "only_tags_with_id_link2 = SoupStrainer(id=\"link2\")\n",
    "\n",
    "def is_short_string(string):\n",
    "    return len(string) < 10\n",
    "\n",
    "only_short_strings = SoupStrainer(string=is_short_string)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "I’m going to bring back the “three sisters” document one more time, and we’ll see what the document looks like when it’s parsed with these three `SoupStrainer` objects:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "html_doc = \"\"\"\n",
    "<html><head><title>The Dormouse's story</title></head>\n",
    "<body>\n",
    "<p class=\"title\"><b>The Dormouse's story</b></p>\n",
    "\n",
    "<p class=\"story\">Once upon a time there were three little sisters; and their names were\n",
    "<a href=\"http://example.com/elsie\" class=\"sister\" id=\"link1\">Elsie</a>,\n",
    "<a href=\"http://example.com/lacie\" class=\"sister\" id=\"link2\">Lacie</a> and\n",
    "<a href=\"http://example.com/tillie\" class=\"sister\" id=\"link3\">Tillie</a>;\n",
    "and they lived at the bottom of a well.</p>\n",
    "\n",
    "<p class=\"story\">...</p>\n",
    "\"\"\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(BeautifulSoup(html_doc, \"html.parser\", parse_only=only_a_tags).prettify())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(BeautifulSoup(html_doc, \"html.parser\", parse_only=only_tags_with_id_link2))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Troubleshooting"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`diagnose()`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you’re having trouble understanding what Beautiful Soup does to a document, pass the document into the `diagnose()` function. (New in Beautiful Soup 4.2.0.) Beautiful Soup will print out a report showing you how different parsers handle the document, and tell you if you’re missing a parser that Beautiful Soup could be using:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from bs4.diagnose import diagnose\n",
    "with open(\"bad.html\") as fp:\n",
    "    data = fp.read()\n",
    "diagnose(data)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Version mismatch problems\n",
    "\n",
    "\n",
    "    * SyntaxError: Invalid syntax (on the line ROOT_TAG_NAME = u'[document]'): Caused by running the Python 2 version of Beautiful Soup under Python 3, without converting the code.\n",
    "    * ImportError: No module named HTMLParser - Caused by running the Python 2 version of Beautiful Soup under Python 3.\n",
    "    * ImportError: No module named html.parser - Caused by running the Python 3 version of Beautiful Soup under Python 2.\n",
    "    * ImportError: No module named BeautifulSoup - Caused by running Beautiful Soup 3 code on a system that doesn’t have BS3 installed. Or, by writing Beautiful Soup 4 code without knowing that the package name has changed to bs4.\n",
    "    * ImportError: No module named bs4 - Caused by running Beautiful Soup 4 code on a system that doesn’t have BS4 installed."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}