ocr-pdf-analysis.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "fccdc384-f963-4407-8d64-22d20aab1d56",
   "metadata": {
    "id": "XIyP_0r6zuVc"
   },
   "source": [
    "<!-- Banner Image -->\n",
    "<img src=\"https://uohmivykqgnnbiouffke.supabase.co/storage/v1/object/public/landingpage/ocr2.png?t=2023-11-09T00%3A26%3A25.198Z\" width=\"100%\">\n",
    "\n",
    "<!-- Links -->\n",
    "<center>\n",
    "  <a href=\"https://console.brev.dev\" style=\"color: #06b6d4;\">Console</a> •\n",
    "  <a href=\"https://brev.dev\" style=\"color: #06b6d4;\">Docs</a> •\n",
    "  <a href=\"/\" style=\"color: #06b6d4;\">Templates</a> •\n",
    "  <a href=\"https://discord.gg/NVDyv7TUgJ\" style=\"color: #06b6d4;\">Discord</a>\n",
    "</center>\n",
    "\n",
    "# OCR + Amazon's MistralLite for a PDF Analysis Chatbot 🤙\n",
    "\n",
    "Welcome!\n",
    "\n",
    "In this notebook and tutorial, we'll allow for long-context PDF analysis using [OCR (Optical Character Recognition)](https://en.wikipedia.org/wiki/Optical_character_recognition) + Amazon's adapted [Mistral 7B](https://github.com/mistralai/mistral-src) model, [MistralLite](https://huggingface.co/amazon/MistralLite?library=true), which allows for contexts of up to 32K, which is [roughly 24000 words, or 48 pages of text](https://twitter.com/SullyOmarr/status/1654576775970828293). From the Hugging Face page:\n",
    "\n",
    "> MistralLite is a fine-tuned Mistral-7B-v0.1 language model, with **enhanced capabilities of processing long context (up to 32K tokens)**. By utilizing an adapted Rotary Embedding and sliding window during fine-tuning, **MistralLite is able to perform significantly better on several long context retrieve and answering tasks**, while keeping the simple model structure of the original model. MistralLite is useful for applications such as long context line and topic retrieval, summarization, question-answering, and etc. \n",
    "\n",
    "Disclaimer: Note that [LLMs have had trouble effectively using information from long contexts](https://twitter.com/LouisKnightWebb/status/1683874116410155009), so you may find you'll still want to use [RAG](https://www.promptingguide.ai/techniques/rag), but it's worth a shot to first try without. Based on MistralLite's description - that it is made for long context retrieve and answering tasks - it just may work. It did for me (but I only used about 5400 tokens).\n",
    "\n",
    "**The text in the PDF I upload in this tutorial is too long to fit into ChatGPT's GPT-4, so this step-by-step guide shows you how to get around that limitation.**\n",
    "\n",
    "We will load the large model in 4-bit quantization using `bitsandbytes` so that we can load it on a smaller GPU (you can optionally skip this if you have the compute for the full model!).\n",
    "\n",
    "Note that if you ever have trouble importing something from Hugging Face, you may need to run `huggingface-cli login` in a shell. To open a shell in Jupyter Lab, click on 'Launcher' (or the '+' if it's not there) next to the notebook tab at the top of the screen. Under \"Other\", click \"Terminal\" and then run the command.\n",
    "\n",
    "### Help us make this tutorial better! Please provide feedback on the [Discord channel](https://discord.gg/pnCpkwU3G5) or on [X](https://x.com/harperscarroll)."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e20c34f9-2b9d-47bb-93f8-22b47be20420",
   "metadata": {
    "id": "SSBw-KpkyRga"
   },
   "source": [
    "#### Before we begin: A note on OOM errors\n",
    "\n",
    "If you get an error like this: `OutOfMemoryError: CUDA out of memory`, tweak your parameters to make the model less computationally intensive. I will help guide you through that in this guide, and if you have any additional questions you can reach out on the [Discord channel](https://discord.gg/pnCpkwU3G5) or on [X](https://x.com/harperscarroll).\n",
    "\n",
    "To re-try after you tweak your parameters, open a Terminal ('Launcher' or '+' in the nav bar above -> Other -> Terminal) and run the command `nvidia-smi`. Then find the process ID `PID` under `Processes` and run the command `kill [PID]`. You will need to re-start your notebook from the beginning. (There may be a better way to do this... if so please do let me know!)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6478f6cf-cb8c-4a2b-bad4-ebb655e21173",
   "metadata": {
    "id": "hWI-uRLEyRgb"
   },
   "source": [
    "## Let's begin!\n",
    "\n",
    "I used a GPU and dev environment from [brev.dev](https://brev.dev). Click the badge below to get your preconfigured instance:\n",
    "\n",
    "[![](https://uohmivykqgnnbiouffke.supabase.co/storage/v1/object/public/landingpage/brevdeploynavy.svg)](https://console.brev.dev/environment/new?instance=A10G:g5.xlarge&diskStorage=256&name=ocr-pdf-analysis&file=https://github.com/brevdev/notebooks/raw/main/ocr-pdf-analysis.ipynb&python=3.10&cuda=12.1.1)\n",
    "\n",
    "Once you've checked out your machine and landed in your instance page, select the specs you'd like (I used **Python 3.10 and CUDA 12.1.1**; these should be preconfigured for you if you use the badge above) and click the \"Build\" button to build your verb container. Give this a few minutes.\n",
    "\n",
    "A few minutes after your model has started Running, click the 'Notebook' button on the top right of your screen once it illuminates (you may need to refresh the screen). You will be taken to a Jupyter Lab environment, where you can upload this Notebook.\n",
    "\n",
    "Note: You can connect your cloud credits (AWS or GCP) by clicking \"Org: \" on the top right, and in the panel that slides over, click \"Connect AWS\" or \"Connect GCP\" under \"Connect your cloud\" and follow the instructions linked to attach your credentials.\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "9871a84f-b2da-4559-b8ef-c67a8b0ac42c",
   "metadata": {
    "id": "FuXIFTFapAMI"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "\u001b[1m[\u001b[0m\u001b[34;49mnotice\u001b[0m\u001b[1;39;49m]\u001b[0m\u001b[39;49m A new release of pip is available: \u001b[0m\u001b[31;49m23.0.1\u001b[0m\u001b[39;49m -> \u001b[0m\u001b[32;49m23.3.1\u001b[0m\n",
      "\u001b[1m[\u001b[0m\u001b[34;49mnotice\u001b[0m\u001b[1;39;49m]\u001b[0m\u001b[39;49m To update, run: \u001b[0m\u001b[32;49mpip install --upgrade pip\u001b[0m\n",
      "\n",
      "\u001b[1m[\u001b[0m\u001b[34;49mnotice\u001b[0m\u001b[1;39;49m]\u001b[0m\u001b[39;49m A new release of pip is available: \u001b[0m\u001b[31;49m23.0.1\u001b[0m\u001b[39;49m -> \u001b[0m\u001b[32;49m23.3.1\u001b[0m\n",
      "\u001b[1m[\u001b[0m\u001b[34;49mnotice\u001b[0m\u001b[1;39;49m]\u001b[0m\u001b[39;49m To update, run: \u001b[0m\u001b[32;49mpip install --upgrade pip\u001b[0m\n",
      "\n",
      "\u001b[1m[\u001b[0m\u001b[34;49mnotice\u001b[0m\u001b[1;39;49m]\u001b[0m\u001b[39;49m A new release of pip is available: \u001b[0m\u001b[31;49m23.0.1\u001b[0m\u001b[39;49m -> \u001b[0m\u001b[32;49m23.3.1\u001b[0m\n",
      "\u001b[1m[\u001b[0m\u001b[34;49mnotice\u001b[0m\u001b[1;39;49m]\u001b[0m\u001b[39;49m To update, run: \u001b[0m\u001b[32;49mpip install --upgrade pip\u001b[0m\n"
     ]
    }
   ],
   "source": [
    "# You only need to run this once per machine\n",
    "!pip install -q -U bitsandbytes reportlab scipy\n",
    "!pip install -q -U git+https://github.com/huggingface/transformers.git\n",
    "!pip install -q -U git+https://github.com/huggingface/accelerate.git"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "17b6dfb3-1028-4949-8128-4ce28479fa52",
   "metadata": {},
   "source": [
    "## 1. OCR: PDF → Text\n",
    "In this section, we'll use [OCR (Optical Character Recognition)](https://en.wikipedia.org/wiki/Optical_character_recognition) to extract text from our PDF. We will use the open-source tool [pd3f](https://pd3f.com/)."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dacfa158-b817-4e1f-a228-1cc7d7733f71",
   "metadata": {},
   "source": [
    "### Install docker-compose and pd3f"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "174e5f55-0246-45e5-86bb-dc0dba959537",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Hit:1 https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2204/x86_64  InRelease\n",
      "Hit:2 http://archive.ubuntu.com/ubuntu jammy InRelease\n",
      "Hit:3 http://security.ubuntu.com/ubuntu jammy-security InRelease\n",
      "Hit:4 http://archive.ubuntu.com/ubuntu jammy-updates InRelease\n",
      "Hit:5 http://archive.ubuntu.com/ubuntu jammy-backports InRelease\n",
      "Reading package lists...\n",
      "Building dependency tree...\n",
      "Reading state information...\n",
      "48 packages can be upgraded. Run 'apt list --upgradable' to see them.\n",
      "Reading package lists...\n",
      "Building dependency tree...\n",
      "Reading state information...\n",
      "ufw is already the newest version (0.36.1-4ubuntu0.1).\n",
      "0 upgraded, 0 newly installed, 0 to remove and 48 not upgraded.\n",
      "Reading package lists...\n",
      "Building dependency tree...\n",
      "Reading state information...\n",
      "docker-compose is already the newest version (1.29.2-1).\n",
      "0 upgraded, 0 newly installed, 0 to remove and 48 not upgraded.\n",
      "Cloning into 'pd3f'...\n",
      "remote: Enumerating objects: 492, done.\u001b[K\n",
      "remote: Counting objects: 100% (28/28), done.\u001b[K\n",
      "remote: Compressing objects: 100% (24/24), done.\u001b[K\n",
      "remote: Total 492 (delta 8), reused 18 (delta 4), pack-reused 464\u001b[K\n",
      "Receiving objects: 100% (492/492), 948.16 KiB | 27.89 MiB/s, done.\n",
      "Resolving deltas: 100% (257/257), done.\n"
     ]
    }
   ],
   "source": [
    "import time, requests, os\n",
    "\n",
    "os.chdir('/home/ubuntu')\n",
    "\n",
    "!sudo apt update -y -q\n",
    "!sudo apt-get install ufw -y -q\n",
    "!sudo apt install docker-compose -y -q\n",
    "!git clone https://github.com/pd3f/pd3f\n",
    "!sudo systemctl start docker.service"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3ca6f019-9aee-4601-b2ca-6402788144b5",
   "metadata": {},
   "source": [
    "### Run pd3f\n",
    "\n",
    "Now open a Terminal ('+' or 'Launcher' at the top tab section -> 'Terminal') and run this command:\n",
    "\n",
    "```\n",
    "cd /home/ubuntu/pd3f && sudo ./dev.sh\n",
    "```\n",
    "\n",
    "Wait until you see something like this, repeating:\n",
    "```\n",
    "ocr_worker_1  | ++ find /to-ocr -name '*.pdf' -type f\n",
    "ocr_worker_1  | + sleep 1\n",
    "```\n",
    "Leave it running."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "74423d99-d875-4048-bb62-5967cb484d81",
   "metadata": {},
   "source": [
    "### Gather PDFs\n",
    "\n",
    "I put my PDF in a directory called \"ocr-example\". You can put multiple PDFs in here, just keep your model's maximum context length in mind. In this case, we use a model that allows for 32K tokens, which is [roughly 24000 words, or 48 pages of text](https://twitter.com/SullyOmarr/status/1654576775970828293). We will want that context to contain:\n",
    "\n",
    "1. The PDF(s) text you'd like to ask questions about\n",
    "2. The instructions (\"Answer the following questions by referring to the data below...\")\n",
    "3. Optionally (recommended), chat history. What questions have been asked and the responses provided. Good for questions that require context, e.g. \"Can you explain that further?\". \n",
    "\n",
    "Keep this in mind as you gather your dataset and decide how you'd like your model to behave.\n",
    "\n",
    "My PDF is an old paper I wrote for a required writing class my sophomore year of college. It was a class on celebrity, and I wrote about how and why I thought Kylie Jenner would be the most successful Kardashian. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "2801a49b-01be-4ede-b676-bde2d33e2f75",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Cloning into 'ocr-example'...\n",
      "remote: Enumerating objects: 8, done.\u001b[K\n",
      "remote: Counting objects: 100% (8/8), done.\u001b[K\n",
      "remote: Compressing objects: 100% (5/5), done.\u001b[K\n",
      "remote: Total 8 (delta 0), reused 0 (delta 0), pack-reused 0\u001b[K\n",
      "Receiving objects: 100% (8/8), 836.07 KiB | 28.83 MiB/s, done.\n"
     ]
    }
   ],
   "source": [
    "# If you'd like to use my example, you can pull it here:\n",
    "!git clone https://github.com/harper-carroll/ocr-example.git\n",
    "\n",
    "directory = \"ocr-example\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "175e1001-4348-407b-a264-23c6de7bc3ea",
   "metadata": {},
   "outputs": [],
   "source": [
    "os.chdir('/home/ubuntu/') # The directory that contains the pdf directory (change if necessary)\n",
    "files = []\n",
    "\n",
    "for filename in os.listdir(directory):\n",
    "    if filename.endswith('.pdf'):\n",
    "        value = (filename, open(directory + '/' + filename, 'rb'))\n",
    "        files.append({'pdf': value})"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c2fdc0a2-0549-4786-9c78-c99fe3423e03",
   "metadata": {},
   "source": [
    "### Use the pd3f OCR API! \n",
    "\n",
    "Post params to put in `data` map (more info [here](https://pd3f.com/docs/pd3f/usage/)):\n",
    "\n",
    "- `lang`: set the language (options: ‘de’, ’en’, ’es’, ‘fr’)\n",
    "- `fast`: whether to check for tables (default: False) (Harper’s note: This seems weird to me, but it’s what the documentation says 🤷‍♀️)\n",
    "- `tables`: whether to check for tables (default: False)\n",
    "- `experimental`: whether to extract text in experimental mode (footnotes to endnotes, depuplicate page header / footer) (default: False)\n",
    "- `check_ocr`: whether to check first if all pages were OCRd (default: True, cannot be modified in GUI)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "139491a7-530b-4e42-8618-1a3ac30118ac",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The text extracted by pd3f is: {'rba.pdf': 'The Rise of Kylie Jenner\\n\\nHarper Carroll\\nPWR 2: Cultures of Personality\\nDr. Maxe Crandall\\nWinter 2016\\n\\n2\\n\\nKeeping Up With the Kardashians broke into the world of reality television on October 14 th , 2007 (Wikipedia). Since the airing of its first episode, it has had eleven seasons and is renewed for its twelfth season, set to air Spring 2016 (E! Online). Eighteen-year-old Kylie Jenner, the youngest of the Kardashian-Jenner clan, is worth far above $10 million owns a $2.7 million home, and retains 74% of the subscribers of all of her sisters\\' apps combined (Celebrity Net Worth, Hollywood Life 2015). In late September, Kylie had 36.5 million Instagram followers. Today, this number has risen a whopping 47% to 53.6 million. \"King Kylie\" is now the second most followed Kardashian sister, second only to Kim, long the Queen of the clan. But only two years ago, Kylie was almost entirely shadowed by the spotlights that shone on her older sisters. When Keeping Up With The Kardashians began, Kim Kardashian was the center of attention, and Kylie and her older sister Kendall were almost indistinguishable as simply the two twin-like little siblings of the three attractive, desirable older sisters. As the sisters grew, Kendall embarked on her highly successful modeling career, and Kylie was, for a while, alone at the bottom, now outshined by what had been her partner from the very start. So how, exactly, did Kylie rise to such phenomenal levels of success so quickly? In two years, how did Kylie skyrocket from her place as the least significant sister to the second most famous one? To answer this question, I studied Kylie\\'s celebrity timeline alongside celebrity culture and the art of fame. \"Stars are involved in making themselves into commodities,\" says Richard Dyer in his study of fame, Heavenly Bodies, \"they are both labour and the thing labour produces.\" My conclusion is this: through conscious strategy and step-by-step work, Kylie \"produced\" herself into the icon she is today. Kylie used four main branding strategies – becoming the \"new\" Kim, trademarking her lips, connecting with her fans, and marketing an alternative, ethereal image of herself – to create a massive empire for endless public consumption.\\n\\n3\\n\\nBecoming Kim: The Transformation of the Face\\nJust three years ago Kylie was an entirely different entity: a natural beauty, her fresh face was delicate and youthful. She was feminine but reserved, put together and elegant. But unlike her older sisters, this fresh-faced, gentle beauty remained relatively out of the spotlight.\\nIn early 2014, Kylie\\'s face suddenly seemed to have transformed, bearing increasingly uncanny resemblance to Kim. Her fame then snowballed rapidly. But why did she choose to mimic Kim? I believe that by latching onto the most famous and stable sister, Kylie\\'s publicity team aimed to boost Kylie in the eyes of the public, drawing the media attention she needed in order to rise in the ranks among her sisters. Batesian mimicry, a commonly occurring phenomenon in the wild, is \"a relationship where one organism that is harmless… mimics a noxious species… [to] avoid predation\" (Study.com); Kylie\\'s own organic image was not drawing attention, as she was essentially irrelevant relative to the other sisters, so she began to change that image to mimic the most powerful and popular sister of all.\\nLess than three\\nfull years later, Kylie\\'s\\nface has entirely\\ntransformed; two simple\\nsplit-face side-by-sides of\\nthe new Kylie – one split with her old self, one\\nsplit with Kim – reveals\\nthe shocking work that\\n\\n4\\n\\nhas been done onto the delicate face of this 18-year-old girl. Kylie has become Kim Kardashian in a younger body. But this was only the first step in Kylie\\'s strategic plan for fame.\\nAfter the initial rounds of plastic surgery to propel herself towards iconicity, I propose that Kylie Jenner crafted a step-by-step branding and sales strategy. The first step: publicity, through a scandal or otherwise. The second: public obsession. The third: commodification of that obsession. Kylie followed this step-by-step process for each of her next three branding endeavors, which she learned from stars before her: her lips, her \"connection\" to her fans, and (contrastingly), her self-objectification, repeatedly presenting doll-like ethereal image to the public to facilitate her own worship.\\n\\nThe Body-Part Obsession: Kylie\\'s Lips\\nKylie\\'s first branding strategy was the development of her lips into something that could be sold for public consumption. In April 2014, after only a single article was published speculating on Kylie\\'s use of plastic surgery – by OK! Magazine, Australia, of all sources Kylie took to her Twitter page to respond. She capitalized on her chance to cause a media frenzy. \"These plastic surgery rumors hurt my feelings to be honest and are kinda insulting… Just in case anyone forgot… I\\'m 16.\" And so it began: the obsession with Kylie\\'s lips had commenced. Hundreds of articles were published theorizing about Kylie\\'s possible lip augmentation, and Kylie and her team continued to vehemently deny the allegations. As recently as March 2015, a representative told The Daily Mail that the allegations were \"so ridiculous\" and \"not true,\" and her makeup artist stated in an interview that her lips were simply \"created using expertly applied lip liner and a little lip balm\" (Bustle 2015). Nevertheless, the rumors continued, attracting\\n\\n5\\n\\nGoogle\\nenormous amounts of media attention. Kylie had instigated the mass publicity herself as the first step of her branding strategy.\\nIn 2015, the second step of Kylie\\'s branding strategy proved\\ncomplete: the viral Kylie Jenner Lip Challenge began, proving that Kylie\\'s\\nlips had become an obsession. With a shot glass, young girls, \"significantly\\nyounger than Kylie, who [was] only 17\" placed their lips into the shot glass and sucked the air out to create friction (Washington Post 2015); this causes blood to rush into the lips, causing a temporary lip filling. But some results were horrifying; the practice commonly caused extreme bruising and even serious injuries requiring stitches when the shot glass broke under the extreme pressure. But millions took the risk; her lips had become widely coveted – fetishized – and millions modeled themselves after this new icon, internalizing, mimicking her appearance to make it their own. This is not the first time an enlarged body part has caused widespread \"fetishization\": Jennifer Lopez\\'s infamous rear end helped drive her career. Kylie, by enlarging a body part, did the same (Beltrán 2002). It seems Kylie may have even directly learned from Lopez\\'s strategy, as Beltrán notes, \"the discussion of Lopez\\'s body was often initiated by Lopez herself\"; Kylie capitalized on the first published article about her lips to incite media coverage. Like Lopez, she\\n\\n6\\n\\ncontinued to capitalize on this public fixation on her lips, flaunting them in social media pictures and intentionally, glamorously, and intriguingly evading questions about them in interviews. She strategically drew more and more attention to them until her lips had truly become an obsession. In May of 2015, the world gasped and rejoiced when Kylie finally came clean about her injections, stating, \"I have had temporary lip fillers\" (The Daily Mail 2015). And that was that. Thus began the next step of Kylie\\'s branding strategy: turning the public obsession into a commodity for sale. Almost immediately, Kylie began on her next entrepreneurial endeavor, the Kylie Jenner lip kits, which she announced in August of 2015. The three colors of \"Lip Kit by Kylie\" had their first release at 12:00am on November 30 th , 2015, and all the kits, priced at $30 each, sold out immediately. Two weeks later, a second batch was released and similarly sold out instantaneously. Kits were being resold on eBay for over $800. (MTV 2015) Lip Kit by Kylie went on sale for a third time on February 6 th , 2016, this time with three new colors. Within 10 minutes, all six colors had again sold out (Elle Magazine 2015). With the beautiful, attention-drawing face of Kim on a younger body, Kylie went a step further with her brand: she created a brand from her controversy-causing and obsessed-over feature. Kylie and her lips have become iconic, and she has wielded them: at first with shy glamour, and now with reckless abandon. The Kylie Jenner team is making an enormous profit. But the development of Kylie\\'s brand didn\\'t stop there.\\n\\nKylie\\'s Diary: The \"Best Friend\" Celebrity\\nKylie third branding scheme, also developed into a commodity using her step-by-step strategy, elevated her to even more spectacular levels of iconicity: Kylie spends most of her time developing an \"intimate\" relationship with her fans. Kylie began using Instagram for the same\\n\\n7\\n\\nreason that almost every single star does: for publicity. Kylie, however, was innovative in her use of Snapchat. An app previously used just between friends, Kylie began using Snapchat to broadcast her daily life to her fans before almost any other star realized the app\\'s marketing capability. She posted – and continues to post today – clips getting her makeup done, goofing around with friends, going to lunch, going to bed, visiting her sisters, and even (controversially) driving her multi-hundred-thousand-dollar cars. Janice Min, former editor for Us Weekly, commented on the rise of the reality star in celebrity culture in a 2009 interview, stating, \"the whole relationship dynamic between the general population and celebrity has morphed into a belief that there\\'s very little separating you from being like them\" (Davisson 5). And through her seemingly intimate use of social media, Kylie has seized this phenomenon and exploited it to its full potential: watching her Snapchat story, just one story among those of your real-life friends, you almost wonder whether Kylie is one of your best friends, too. In July of 2015, she became the \"Queen of Snapchat,\" the number-one viewed person on the app, which allows users to share everyday clips from their lives to their friends, or in Kylie\\'s case, massive audiences (Elle Magazine 2015). The public had become obsessed with her social media presence, following every trace she left of herself on the Internet.\\nAnd thus came the third step of the strategy: commodification of this public obsession. While announcing that she had become the most followed person on Snapchat and one of the top ten most followed people on Instagram, Kylie announced the development of her website and mobile app, to which a subscription would cost $2.99 per month. Her app, released in September, skyrocketed to the #1 spot on the iTunes charts, surpassing the download counts of her sisters\\' apps astronomically. Kylie\\'s app, though one of the most expensive in the store, was\\ndownloaded 1.75 million times during its first week, followed by Khloe with 498,000 downloads\\n\\n8\\n\\nand Kim with 477,000 downloads (Fortune 2015). \"Part digital diar[y], part lifestyle brand\" (Wired 2015) Kylie\\'s website and accompanying app have Kylie-recommended clothes, accessories, and even foods that users can purchase with a simple click. They feature makeup tutorials to achieve Kylie\\'s infamous coveted look, and photos from A-list celebrity events; for $2.99 a month, Kylie-obsessed fans can feast their eyes on exclusive pictures of Kylie and her boyfriend, rapper Tyga. At surprise moments, users will receive a push notification on their phones when Kylie is broadcasting live on the app. As mentioned, Kylie now retains over 74% of all the subscribers for the sisters\\' apps combined. And the key to its massive success, stated by Kylie herself? Its intimacy: precisely the obsession she cultivated and turned into a commodity. In an interview with Time, who included Kylie in their 30 most influential teens list of 2015, Kylie noted that her app was more successful than hyper-famous older sister Kim\\'s because Kim\\'s is \"simple, everything is black and white. Mine is more, I invite the cameras over and say, \\'Oh, hey, let\\'s just do this.\\' We do makeup tutorials and I\\'m laughing and talking,\" leading the star to an \"unparalleled\" level of openness with her fans. She further explained, \"I just like to show my personality and show my fans my home and my dogs. I get really personal. I\\'m on Snapchat, and I\\'m just like them at the end of the day. Kim\\'s more private. She has kids. It\\'s just different\" (Time Magazine 2015). As Wired puts it, \"Kylie loves to share - and her fans love to follow.\"\\nAs Kylie seemed to have learned from Jennifer Lopez about the power of accentuated body parts, the full utilization of social media appears to have been learned from another star: Lady Gaga previously harnessed such platforms in a wildly successful way. Lady Gaga is a well-known player in the world of fame; she has been open about her extensive studies of previous stars before her to aide in her extraordinary success. In another of Richard Dyer\\'s works, Stars,\\n\\n9\\n\\nhe writes, \"the celebrity…is not distant but attainable – touchable by the multitude. The greatness of the celebrity is something that can be shared.\" Similar to Kylie, Lady Gaga capitalized off of social media, using the most accessible platforms at the time, Facebook and Twitter. Rather than relying on mainstream media to translate her ideas, Gaga used the platforms to communicate directly with her fans (Davisson 18). In 2010, Gaga announced, \"I believe, as an artist, being private in public is at the core of the aesthetic\" (Davisson 6). Gaga was one of the first celebrities to recognize the power of such intimacy and harness social media to create her own brand, to allow fans to connect with her on a (seemingly) deeply personal level. As a result, Gaga become an icon almost overnight. Suddenly, the world was obsessed with Lady Gaga. Like Kylie, Gaga also created her own website, Littlemonsters.com, after her social media platforms became enormously successful. On her site Gaga posted \"random\" photos from her day, thoughts while on tour, advance information about events, comments on current styles, and responses to media scandals about her behavior. She recorded videos just \"consisting of her and her backup dancers sitting around after shows or during downtime, chatting\" (Davisson 113). Kylie, in her day-to-day videos with friends, seems to copy Gaga\\'s technique. As Amber Davisson writes in her study of Gaga, Lady Gaga and the Remaking of Celebrity Culture, such \"constant engagement increases fan awareness of Gaga and makes her more a part of fans\\' day-to-day lives.\" With new technology allowances today, Kylie\\'s spontaneously occurring live feed offered (for a price) from her app takes Gaga\\'s discovery and leaps forward.\\n\\nThe Commodification of the Self: Creating a Cultural Deity\\nThe final branding strategy I will discuss in this paper is Kylie\\'s self-commodification: though she cultivates an \"intimate\" relationship with her fans through her social media platforms,\\n\\n10\\n\\nthere also exists a recurring image of Kylie as an inhuman, too-perfect doll. Perhaps this image could pass as Kylie\\'s \"alter ego,\" an alluring and glamorous addition to her \"open, just-like-you\" image. Either way, Kylie capitalizes on the public obsession with herself by representing herself as a commodity, an inhuman, doll-like object to be consumed.\\nFirst, consider the split-face of half the new Kylie and half the old. The light eye makeup of the past sharply contrasts with Kylie\\'s new, dramatic shadow and false eyelashes, her brow theatrically arched. The highly risen eyebrow hairs to accomplish this arch seem new too, as they exist in places that were simply not possible on her old face. Her youthful appearance has been somehow lifted even more, her skin airbrushed, her jawline widened and defined, her lips enlarged to over twice their old size. Her stare has gone from young, innocent and present to sultry and lifeless. She exists as a real-life wax figure, a perfect porcelain doll; she does not seem human. Thus, under all the marketing and attempts at forming a connection with her fan base, a dichotomy exists: between the intimate, animated Kylie and the Kylie continuously presented as an object for public consumption. This is the commodification of the entire being of the icon. But we will see that the widespread presentation of such a perfected, inhuman image of Kylie has far greater power than simply self-commodification.\\nThis technique, of projecting the image of ethereal facial-perfection, seemed to have been learned from another star: Greta Garbo. Like Garbo, such a representation of Kylie gives her an\\n\\n11\\n\\nessence of \"the Divine… to convey… the essence on her corporeal person, descended from a heaven where all things are formed and perfected in the clearest light… her face was not to have any reality except that of its perfection\" (Barthes 1972). Kylie learned from Garbo: by presenting herself to the public in flawless\\nimages, Kylie has figured out\\nhow to attain the highest level of\\niconicity: by becoming an object\\nfor worship. In numerous photo\\nshoots, Kylie has posed as a\\nliteral (sex) doll; with lifeless\\neyes and skin airbrushed to\\nresemble a perfectly flawless\\nporcelain doll, Kylie has sold out\\nmagazines. In the previously\\ndiscussed comparison photo with\\nKim, Kylie\\'s eyes seem even\\nInterview Magazine 2015\\nmore lifeless than her famous sister\\'s. And thus, Kylie becomes truly a\\ncommodity, an icon to be sold to the worshipping public. She is not a person; she is not real. Like an orthodox Christian icon, she \"remind[s] us what we are and what we should be…show[s] us the importance of matter and of material things… we are challenged to follow in [its] footsteps\" (Orthodoxa.org). And it is true: millions of people follow Kylie Jenner\\'s every move. Images of her perfect face, augmented above those of the common person, are to be admired and mimicked. Every trace she leaves on the Earth is chased after and consumed. From the\\n\\n12\\n\\ndangerous Lip Challenge to the willingness of millions to pay $36 a year to view every moment of how Kylie lives her daily life, Kylie\\'s worshippers take risks and make sacrifices to learn from this \"enlightened\" being.\\nThus, to achieve fame, Kylie has truly become the \"commodity\" of which Dyer spoke. This technique of \"fashioning the star out of the raw material of the person\" is the norm in celebrity culture (Dyer 1986). Dyer states that the degree of change to create the icon varies from person to person, depending on the \"inherent qualities of the material,\" in this case, Kylie transformed dramatically.\\n\\nThe Future of Kylie\\nKylie Jenner is creating her fame in a highly intelligent way. In an interview published in December of 2015, Kylie admits that Lady Gaga, the fame strategist whom Kylie seemed to have imitated, has complimented her, telling Kylie, \"I think you\\'re so amazing, and whatever you\\'re doing, people are just obsessed with it\" (Interview Magazine 2015). Gaga knows: celebrities are an image; their actions and stories for the public are carefully and strategically planned. Before Kylie, Gaga fully understood the power that social media held in its ability to create a seemingly personal connection between the celebrity and her fan base.\\n\\n13\\n\\nKylie Jenner began her rise to iconicity by physically mimicking Kim; with careful marketing strategy and the creation of her own brand, however, she has risen – and will continue to rise – far beyond Kim. Kylie has the (more youthful) face of Kim, but she also has strategized a commercial product: she is the Kim for the people, Kim 2.0. Kris Jenner, the famous \"momager\" of the Kardashian-Jenner clan, is rumored to be moving her attention away from Kim to Kylie, believing Kylie is now \"the family\\'s biggest star\" (Hollywood Life 2015). And in a live stream video on Kylie\\'s app with Kim – suggestively titled, \"You\\'re in Bed with Kim and Kylie\" – Kim asked Kylie, \"How do you feel now that you\\'ve dethroned me?\" (MTV). Kylie \"laughed and brushed off the question,\" without responding. No words needed to be said.\\nAs her self-placed title suggests, \"King Kylie\" is – and always has been - on a mission to take her rightful place as the true leader of the clan. With the \"reborn\" face of Kim and the learned strategy of celebrity geniuses before her, Kylie is unstoppable. As Wired puts it, \"Like It or Not, Kylie Jenner Is the Celebrity of the Future\" (2015).\\n\\n14\\n\\nWorks Cited\\n\\nBailey, Alyssa. \"Kylie Jenner\\'s New Lip Kits Sold Out in Less Than 10 Minutes.\" ELLE. Hearst Communications, 05 Feb. 2016. Web. 11 Mar. 2016.\\n\\nB arthes, Roland. \"The Face of Garbo.\" Epigraf. Web. 12 Mar. 2016. <http://epigraf.fisek.com.tr/?num=236>.\\n\\nB eltrán, Mary. \"The Hollywood Latina Body as Site of Social Sturggle: Media Constructions of Stardom and Jennifer Lopez\\'s \"Cross-over Butt\"\" Quarterly Review of Film and Video 19.1 (2002): 71-86. DOI: 10.1080/10509200214823\\n\\nD\\'Addario, Daniel. \"Kylie Jenner: I Want to Be \"an Inspiration for Young Girls\"\" Time. Time, 29 Oct. 2015. Web. 11 Mar. 2016.\\n\\nDavisson, Amber L. Lady Gaga and the Remaking of Celebrity Culture. Jefferson, North Carolina: McFarland, 2013. Print.\\n\\nDyer, Richard, and Paul McDonald. Stars. London: BFI Pub., 1998. Print.\\nDyer, Richard. \"Introduction.\" Heavenly Bodies: Film Stars and Society. New York: St. Martin\\'s, 1986. 1-7. Print.\\n\\nEnzor, Laura. \"Batesian Mimicry: Examples & Definition.\" Study.com. Web. 28 Jan. 2016. Greenberg, Julia. \"Like It or Not, Kylie Jenner Is the Celebrity of the Future.\" Wired. Conde Nast Digital, 22 Sept. 2015. Web. 11 Mar. 2016.\\n\\nHarwood, Erika. \"I Bought All Three Kylie Jenner Lip Kits And Had No Idea.\" MTV News. Viacom International, 16 Dec. 2015. Web. 11 Mar. 2016.\\n\\n\"Icons.\" Orthodox Church of Estonia. Web. 11 Mar. 2016.\\n\\n15\\n\\n\"Keeping Up with the Kardashians.\" E! Online. Web. 12 Mar. 2016.\\n\"Keeping Up with the Kardashians.\" Wikipedia. Wikimedia Foundation. Web. 12 Mar. 2016. \"Kylie Jenner Net Worth.\" Celebrity Net Worth. Web. 28 Jan. 2016.\\n\\n\"Kylie Jenner, 17, FINALLY Admits to Lip Fillers after Big Sister Khloe Kardashian Urges Her to Stop Lying.\" Daily Mail.com. Associated Newspapers, 6 May 2015. Web. 11 Mar. 2016.\\n\\nLindner, Emily. \"Kim Kardashian Confronts Kylie Jenner About \\'Dethroning\\' Her.\" MTV News. Viacom International, 25 Sept. 2015. Web. 11 Mar. 2016.\\n\\nMoyer, Justin Wm. \"Kylie Jenner Lip Challenge: The Dangers of \\'plumping That Pout\\'.\" Washington Post. The Washington Post, 21 Apr. 2015. Web. 11 Mar. 2016.\\n\\nPhi, Michelle. \"Kylie Jenner: Kris Focused On Her Career — She\\'s The Biggest Family Star.\" Hollywood Life. PMC, 08 Sept. 2015. Web. 11 Mar. 2016.\\n\\nPhi, Michelle. \"Kylie Jenner Wins Vs. Kim Kardashian In App Launch - Tops The iTunes Charts.\" Hollywood Life. PMC, 15 Sept. 2015. Web. 20 Jan. 2016.\\n\\nRao, Leena. \"Which Kardashian Got the Most App Downloads?\" Fortune. 2015. Web. 28 Jan. 2016.\\nSciaretto, Amy. \"Who Is Kylie Jenner\\'s Makeup Artist And What Does He Reveal About Those Overdrawn Lips?\" Bustle. 25 Mar. 2015. Web. 11 Mar. 2016.\\n\\nShe Matches Kim More than She Does Her Own Self. Digital image. Http://www.dailymail.co.uk/femail/article-3172715/Shocking-photo-collage-shows-17-year-old-Kylie-Jenner-s-uncanny-resemblance-Kim-Kardashian-reveals-just-teen-changed-years.html. Associated Newspapers Ltd, 23 July 2015. Web. 20 Jan. 2016.\\n\\nWallace, Chris. \"Kylie Jenner.\" Interview Magazine. Code and Theory, 1 Dec. 2015. Web. 11 Mar. 2016.\\n\\n16\\n\\n'}\n"
     ]
    }
   ],
   "source": [
    "import time, requests, os\n",
    "\n",
    "ocr_texts = {}\n",
    "\n",
    "for file in files:     \n",
    "    response = requests.post(\n",
    "        'http://localhost:1616', \n",
    "        files=file, \n",
    "        data={'lang': 'en'}\n",
    "    )\n",
    "    id = response.json()['id']\n",
    "\n",
    "    while True:\n",
    "        req = requests.get(f\"http://localhost:1616/update/{id}\")\n",
    "        reqAsJson = req.json()\n",
    "        if 'text' in reqAsJson:\n",
    "            break\n",
    "        time.sleep(1)\n",
    "    filename = file['pdf'][0]\n",
    "    ocr_texts[filename] = reqAsJson['text']\n",
    "\n",
    "print(\"The text extracted by pd3f is:\", ocr_texts)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d9fdb383-710a-4e2f-a7e1-3aed8bd2c588",
   "metadata": {},
   "source": [
    "## 2. Load the Model\n",
    "\n",
    "[MistralLite](https://huggingface.co/amazon/MistralLite?library=true) is Amazon's variation of Mistral that allows for contexts of up to 32K. From the HuggingFace page:\n",
    "\n",
    "> MistralLite is a fine-tuned Mistral-7B-v0.1 language model, with enhanced capabilities of processing long context (up to 32K tokens). By utilizing an adapted Rotary Embedding and sliding window during fine-tuning, MistralLite is able to perform significantly better on several long context retrieve and answering tasks, while keeping the simple model structure of the original model. MistralLite is useful for applications such as long context line and topic retrieval, summarization, question-answering, and etc. \n",
    "\n",
    "In this section, we load a 4-bit quantized version of the model so it will fit on a smaller GPU. You can choose to remove the `bnb_config` if you have the compute to load the full version.\n",
    "\n",
    "***Quantization*** in the context of deep learning is the process of reducing the numerical precision of a model's tensors, making the model more compact and the operations faster in execution. This is by nature lossy and usually has some negative effect on accuracy. Mistral's tensors were 16-bit, and we load them in 4-bit, which reduces the bit usage by 75%."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "154e6b22-8d4c-484d-95a8-b5c51f7fe959",
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch\n",
    "from transformers import AutoTokenizer, AutoModelForCausalLM, BitsAndBytesConfig, StoppingCriteriaList, StoppingCriteria\n",
    "\n",
    "base_model_id = \"amazon/MistralLite\"\n",
    "bnb_config = BitsAndBytesConfig(\n",
    "    load_in_4bit=True,\n",
    "    bnb_4bit_use_double_quant=True,\n",
    "    bnb_4bit_quant_type=\"nf4\",\n",
    "    bnb_4bit_compute_dtype=torch.bfloat16\n",
    ")\n",
    "\n",
    "model = AutoModelForCausalLM.from_pretrained(base_model_id, quantization_config=bnb_config)\n",
    "tokenizer = AutoTokenizer.from_pretrained(base_model_id, add_bos_token=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e9098e3c-2f77-4a97-9b99-c71f288615c3",
   "metadata": {},
   "source": [
    "## 3. Get Prompt Ready\n",
    "\n",
    "In this tutorial, since we're using a model that allows for long context lengths, we'll be inserting the text from our entire PDF into the prompt.\n",
    "We want to prompt engineer a bit so that we get the functionality we'd like."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2881231b-4e31-4525-8929-c0b97fcc40b1",
   "metadata": {},
   "source": [
    "Here I've added some code to remove the \"Works Cited\" section of my essay, since it is quite long and I may not need it, in which case I'd rather reserve that context space for something else, like chat history. The code below removes everything after `end_phrase` (\"Works Cited\" in this case) section if `trim=True`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "adcfd044-0f1a-4d5d-b830-7bc9ce363eb4",
   "metadata": {},
   "outputs": [],
   "source": [
    "end_phrase = \"Works Cited\"\n",
    "full_essay = ocr_texts['rba.pdf'] # If you have ≥ 1 pdf, you'll need to alter this code\n",
    "trim = True\n",
    "\n",
    "index = full_essay.find(end_phrase)\n",
    "if trim and index != -1:\n",
    "    # If \"Works Cited:\" is found, remove everything after that index\n",
    "    essay = full_essay[:index]\n",
    "else:\n",
    "    # If \"Works Cited:\" is not found, return the original string\n",
    "    essay = full_essay"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "f5a68fb3-0ffb-4d9b-8ee4-1ebc954e4c38",
   "metadata": {},
   "outputs": [],
   "source": [
    "eval_prompt = \"\"\"Task: Provide detailed answers to questions provided about the following essay, referencing the essay itself.\n",
    "\n",
    "The essay: \"\"\" + essay + \"-----\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "444919bc-7618-400d-bc3c-14f24a66dded",
   "metadata": {},
   "outputs": [],
   "source": [
    "model_input = tokenizer(eval_prompt, return_tensors=\"pt\", return_attention_mask=False).to(\"cuda\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "61523277-5441-4370-b701-d0f8584c9a5b",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'The initial prompt and essay is 5300 tokens.'"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "f\"The initial prompt and essay is {len(model_input[0])} tokens.\""
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d904fdf8-dfcc-4d9b-9c9c-acc157e86b32",
   "metadata": {},
   "source": [
    "## 4. Run the Model!\n",
    "### Add Stopping Criteria List\n",
    "\n",
    "Here is where we teach the model to stop if it sees certain tokens. A common issue with chatbots is that the model will answer its question and then generate another one, assuming the role of the user. For example:\n",
    "\n",
    "You: \n",
    "```\n",
    "Question: How many people live in the United States in 2023? \n",
    "```\n",
    "The model:\n",
    "```\n",
    "Answer: Almost 340 million people.\n",
    "Question: How many people live in Canada in 2023?\n",
    "Answer: About 38.8 million people.\n",
    "Question: ....\n",
    "```\n",
    "\n",
    "To mitigate this issue, we provide stopping criteria, where each \"criteria\" is represented as a list of tokens for the words we'd like the model to stop at. The stopping criteria we will use for this model is the word \"Question: \" - if the model generates \"Question: \", it know it's gone too far. Another common stopping criteria is the presence of a newline.\n",
    "\n",
    "First, we need to get the tokenized form of our stop words (in this case, just \"Question: \")."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "8611bce1-fc17-4d90-a7fc-fea824cc5555",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[1, 22478, 28747, 28705]]\n"
     ]
    }
   ],
   "source": [
    "list_of_stop_words = [\"Question: \"]\n",
    "\n",
    "stop_words_ids = [\n",
    "    tokenizer.encode(stop_word) for stop_word in list_of_stop_words]\n",
    "\n",
    "print(stop_words_ids)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "3787b092-bdbe-4f5d-a4a7-926c07e86545",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Code from https://discuss.huggingface.co/t/implimentation-of-stopping-criteria-list/20040\n",
    "\n",
    "class StoppingCriteriaSub(StoppingCriteria):\n",
    "\n",
    "    def __init__(self, stops = []):\n",
    "      StoppingCriteria.__init__(self), \n",
    "\n",
    "    def __call__(self, input_ids: torch.LongTensor, scores: torch.FloatTensor, stops = []):\n",
    "      self.stops = stops\n",
    "      for i in range(len(stops)):\n",
    "        self.stops = self.stops[i]\n",
    "\n",
    "stopping_criteria = StoppingCriteriaList([StoppingCriteriaSub(stops = stop_words_ids)])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "99c107e1-8dc2-47fc-9df8-ca2cb770a64a",
   "metadata": {},
   "source": [
    "Now, let's define the chatbot loop. This loop takes in a user's question, places \"\\n\\nQuestion: \" as the prefix and \"\\nAnswer: \" as the suffix, and then tokenizes just that new string to save memory. Then, it concatenates it onto the end of the old Encoding object, i.e. the previous tokenized prompt. \n",
    "\n",
    "I noticed that the Encoding's `num_tokens` remains the same - not sure how to fix this. If you know, you'll get $10 of free Brev credits. Just reach out to me. \n",
    "\n",
    "The model will stop if you input one of the `exit_terms` as the question."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "05402bf2-cd82-4d3e-8359-5d73cd3b8a0c",
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Question:  What specific incidents helped Kylie Jenner rise to fame?\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Answer: 1. Kylie's transformation to look like Kim 2. The Kylie Jenner Lip Challenge 3. Kylie's Snapchat story 4. Kylie's website and mobile app\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Question:  How did Kylie Jenner use branding strategies to maintain her celebrity status?\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Answer: 1. Becoming the \"new\" Kim 2. Trademarking her lips 3. Connecting with her fans 4. Marketing an alternative, ethereal image of herself\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Question:  What celebrities did Kylie draw inspiration from?\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Answer: 1. Lady Gaga 2. Jennifer Lopez 3. Greta Garbo\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Question:  stop\n"
     ]
    }
   ],
   "source": [
    "model.eval()\n",
    "\n",
    "exit_terms = [\"stop\", \"exit\"]\n",
    "\n",
    "while True:\n",
    "    q = input(\"Question: \")\n",
    "    if q.lower() in exit_terms: \n",
    "        break\n",
    "    next_prompt = \"\\n\\nQuestion: \" + q + \"\\nAnswer: \"\n",
    "    next_input_tokenized = tokenizer(next_prompt, return_tensors=\"pt\", return_attention_mask=False).to(\"cuda\")   \n",
    "    model_input[\"input_ids\"] = torch.cat((model_input[\"input_ids\"], next_input_tokenized[\"input_ids\"]), dim=1)\n",
    "    model_input[\"attention_mask\"] = torch.ones(model_input[\"input_ids\"].shape)\n",
    "    with torch.no_grad():\n",
    "        gen_tokens = model.generate(**model_input, stopping_criteria=stopping_criteria, max_new_tokens=300, repetition_penalty=1.05, pad_token_id=tokenizer.eos_token_id)\n",
    "        out = tokenizer.decode(gen_tokens[0][len(model_input[\"input_ids\"][0]):], skip_special_tokens=True)\n",
    "        print(\"Answer: \" + out)\n",
    "        model_input[\"input_ids\"] = gen_tokens\n",
    "        model_input[\"attention_mask\"] = torch.ones(model_input[\"input_ids\"].shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ceb7275b-70de-4afb-99f7-4c9612e6f3d8",
   "metadata": {
    "id": "VCJnpZoayRgq"
   },
   "source": [
    "### Sweet... it worked! Epic!!!\n",
    "\n",
    "I hope you enjoyed this tutorial on OCR + building a PDF analysis chatbot using MistralLite. Please join the community on [Discord](https://discord.gg/pnCpkwU3G5)! \n",
    "\n",
    "If you have any questions, please reach out to me on [X](https://x.com/harperscarroll) or in the [Discord channel](https://discord.gg/pnCpkwU3G5).\n",
    "\n",
    "🤙 🤙 🤙 🤙 🤙 🤙 🤙 🤙 🤙 🤙 🤙 🤙 🤙 🤙 🤙 🤙 🤙 🤙 🤙 🤙 🤙 🤙 🤙 🤙 🤙 🤙 🤙 🤙 🤙 🤙 🤙 🤙 🤙 🤙 🤙 🤙 🤙 🤙 🤙 🤙 🤙 🤙 🤙 🤙"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}