01_document_loading.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "d4c3ab93",
   "metadata": {},
   "source": [
    "# Document Loading"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f2b5f165-b646-41fb-abc8-eb6ab31f5edc",
   "metadata": {},
   "source": [
    "## Note to students.\n",
    "During periods of high load you may find the notebook unresponsive. It may appear to execute a cell, update the completion number in brackets [#] at the left of the cell but you may find the cell has not executed. This is particularly obvious on print statements when there is no output. If this happens, restart the kernel using the command under the Kernel tab."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "af8b8ee8",
   "metadata": {},
   "source": [
    "## Retrieval augmented generation\n",
    " \n",
    "In retrieval augmented generation (RAG), an LLM retrieves contextual documents from an external dataset as part of its execution. \n",
    "\n",
    "This is useful if we want to ask question about specific documents (e.g., our PDFs, a set of videos, etc). "
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "3334357c",
   "metadata": {},
   "source": [
    "![overview.jpeg](https://github.com/kishorerv93/langchain-qna/blob/main/data/step1.jpeg)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f015f708",
   "metadata": {
    "height": 30,
    "tags": []
   },
   "outputs": [],
   "source": [
    "#! pip install langchain"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f26c2a2a-f7a7-4580-ab73-178bab918dd7",
   "metadata": {
    "height": 166,
    "tags": []
   },
   "outputs": [],
   "source": [
    "import os\n",
    "import openai\n",
    "import sys\n",
    "sys.path.append('../..')\n",
    "\n",
    "from dotenv import load_dotenv, find_dotenv\n",
    "_ = load_dotenv(find_dotenv()) # read local .env file\n",
    "\n",
    "openai.api_key  = os.environ['OPENAI_API_KEY']"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "38baf6d3",
   "metadata": {},
   "source": [
    "## PDFs\n",
    "\n",
    "Let's load a PDF [transcript](https://see.stanford.edu/materials/aimlcs229/transcripts/MachineLearning-Lecture01.pdf) from Andrew Ng's famous CS229 course! These documents are the result of automated transcription so words and sentences are sometimes split unexpectedly."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ab4583ef",
   "metadata": {
    "height": 64,
    "tags": []
   },
   "outputs": [],
   "source": [
    "# The course will show the pip installs you would need to install packages on your own machine.\n",
    "# These packages are already installed on this platform and should not be run again.\n",
    "#! pip install pypdf "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "38ef5d48",
   "metadata": {
    "height": 64,
    "tags": []
   },
   "outputs": [],
   "source": [
    "from langchain.document_loaders import PyPDFLoader\n",
    "loader = PyPDFLoader(\"docs/cs229_lectures/MachineLearning-Lecture01.pdf\")\n",
    "pages = loader.load()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a284cc8a",
   "metadata": {},
   "source": [
    "Each page is a `Document`.\n",
    "\n",
    "A `Document` contains text (`page_content`) and `metadata`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "fd28c723-3625-4219-b0f8-8d5b761ae79e",
   "metadata": {
    "height": 30,
    "tags": []
   },
   "outputs": [],
   "source": [
    "len(pages)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "26ff4112",
   "metadata": {
    "height": 30,
    "tags": []
   },
   "outputs": [],
   "source": [
    "page = pages[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5c94e3b5",
   "metadata": {
    "height": 30,
    "tags": []
   },
   "outputs": [],
   "source": [
    "print(page.page_content[0:500])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "605d0932",
   "metadata": {
    "height": 30,
    "tags": []
   },
   "outputs": [],
   "source": [
    "page.metadata"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ead28868",
   "metadata": {},
   "source": [
    "## YouTube"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4c5f360f",
   "metadata": {
    "height": 64,
    "tags": []
   },
   "outputs": [],
   "source": [
    "from langchain.document_loaders.generic import GenericLoader\n",
    "from langchain.document_loaders.parsers import OpenAIWhisperParser\n",
    "from langchain.document_loaders.blob_loaders.youtube_audio import YoutubeAudioLoader"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4835edd9",
   "metadata": {
    "height": 47,
    "tags": []
   },
   "outputs": [],
   "source": [
    "# ! pip install yt_dlp\n",
    "# ! pip install pydub"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "adaa8f7a-bd04-4bbd-96a9-8c2088426885",
   "metadata": {},
   "source": [
    "**Note**: This can take several minutes to complete."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "197f0936",
   "metadata": {
    "height": 132,
    "tags": []
   },
   "outputs": [],
   "source": [
    "url=\"https://www.youtube.com/watch?v=jGwO_UgTS7I\"\n",
    "save_dir=\"docs/youtube/\"\n",
    "loader = GenericLoader(\n",
    "    YoutubeAudioLoader([url],save_dir),\n",
    "    OpenAIWhisperParser()\n",
    ")\n",
    "docs = loader.load()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c2bf39c3",
   "metadata": {
    "height": 30,
    "tags": []
   },
   "outputs": [],
   "source": [
    "docs[0].page_content[0:500]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6b54e6f9",
   "metadata": {},
   "source": [
    "## URLs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ede7f5d4",
   "metadata": {
    "height": 64,
    "tags": []
   },
   "outputs": [],
   "source": [
    "from langchain.document_loaders import WebBaseLoader\n",
    "\n",
    "loader = WebBaseLoader(\"https://github.com/basecamp/handbook/blob/master/37signals-is-you.md\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "596c8f4e-6fd5-4230-9dfc-84e100e90d72",
   "metadata": {
    "height": 30,
    "tags": []
   },
   "outputs": [],
   "source": [
    "docs = loader.load()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3039f8ed-ebc1-44e7-829a-9499dc5d1f03",
   "metadata": {
    "height": 30,
    "tags": []
   },
   "outputs": [],
   "source": [
    "print(docs[0].page_content[:500])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e6c62299",
   "metadata": {},
   "source": [
    "## Notion"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "adc4b630",
   "metadata": {},
   "source": [
    "Follow steps [here](https://python.langchain.com/docs/modules/data_connection/document_loaders/integrations/notion) for an example Notion site such as [this one](https://yolospace.notion.site/Blendle-s-Employee-Handbook-e31bff7da17346ee99f531087d8b133f):\n",
    "\n",
    "* Duplicate the page into your own Notion space and export as `Markdown / CSV`.\n",
    "* Unzip it and save it as a folder that contains the markdown file for the Notion page.\n",
    " "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "be17ebad",
   "metadata": {},
   "source": [
    "![image.png](./img/image.png)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "292e2a8f",
   "metadata": {
    "height": 64,
    "tags": []
   },
   "outputs": [],
   "source": [
    "from langchain.document_loaders import NotionDirectoryLoader\n",
    "loader = NotionDirectoryLoader(\"docs/Notion_DB\")\n",
    "docs = loader.load()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8ce8fde2",
   "metadata": {
    "height": 30,
    "tags": []
   },
   "outputs": [],
   "source": [
    "print(docs[0].page_content[0:200])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f14fa1d2",
   "metadata": {
    "height": 30,
    "tags": []
   },
   "outputs": [],
   "source": [
    "docs[0].metadata"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "06b20ba2-8849-44c3-af8b-6ff0913895b0",
   "metadata": {
    "height": 30
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.17"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}