FinBERT_Training.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "kernelspec": {
      "display_name": "FinBERT",
      "language": "python",
      "name": "finbert"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.7.3"
    },
    "pycharm": {
      "stem_cell": {
        "cell_type": "raw",
        "metadata": {
          "collapsed": false
        },
        "source": []
      }
    },
    "toc": {
      "base_numbering": 1,
      "nav_menu": {},
      "number_sections": true,
      "sideBar": true,
      "skip_h1_title": true,
      "title_cell": "Table of Contents",
      "title_sidebar": "Contents",
      "toc_cell": false,
      "toc_position": {},
      "toc_section_display": true,
      "toc_window_display": true
    },
    "colab": {
      "name": "FinBERT Training.ipynb",
      "provenance": [],
      "collapsed_sections": [],
      "toc_visible": true,
      "include_colab_link": true
    },
    "accelerator": "GPU"
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/hawc2/Text-Analysis-with-Python/blob/master/FinBERT_Training.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "8m1dOEyQtatw"
      },
      "source": [
        "# FinBERT Notebook for Todd Schifeling's research\n",
        "\n",
        "This notebooks shows how to train and use the FinBERT pre-trained language model for financial sentiment analysis.\n",
        "\n",
        "It was not designed to be used with Google Colab, so alterations listed below need to be made. I've finished some preliminary setup for installing packages and downloading data/models, but it isn't finished and there may be a better approach.\n",
        "\n",
        "# Relevant Resources\n",
        "\n",
        "Github Repo: https://github.com/ProsusAI/finBERT\n",
        "\n",
        "# Steps to Set-Up\n",
        "\n",
        "I've gotten pretty far, but I've encountered a couple confusions.\n",
        "\n",
        "The Models section in Github is not clear about where to put which models, and which config.json file to use. The main error I keep getting, which worries me may be a development issue, is the following, from the Configuring Training Parameters section: \n",
        "\n",
        "Can't load config for '/content/models/language_model/finbertTRC2'. Make sure that:\n",
        "\n",
        "- '/content/models/language_model/finbertTRC2' is a correct model identifier listed on 'https://huggingface.co/models'\n",
        "\n",
        "- or '/content/models/language_model/finbertTRC2' is the correct path to a directory containing a config.json file\n",
        "\n",
        "\n",
        "# Contacting Developer\n",
        "\n",
        "It may be worth contacting the developer and ask them for help setting up a Colab environment for this script\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "JzpSlmyLgFHI"
      },
      "source": [
        ""
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "vHvHaVPxUiMH"
      },
      "source": [
        "# Set-Up Colab Notebook"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "1Ede4QjEUCzT",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "e34807f9-ed28-42f9-e7ba-ce46230f77fb"
      },
      "source": [
        "!pip install oblib\n",
        "!pip install scikit-learn\n",
        "!pip install spacy\n",
        "!pip install torch\n",
        "!pip install textblob\n",
        "!pip install transformers"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Collecting oblib\n",
            "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/9a/df/4e7753f57b41c9b5f4ece65ffe403dd0a2ad730069e193a379e773cc47b8/oblib-1.1.3-py3-none-any.whl (30.7MB)\n",
            "\u001b[K     |████████████████████████████████| 30.7MB 107kB/s \n",
            "\u001b[?25hCollecting lxml==4.2.5\n",
            "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/03/a4/9eea8035fc7c7670e5eab97f34ff2ef0ddd78a491bf96df5accedb0e63f5/lxml-4.2.5-cp36-cp36m-manylinux1_x86_64.whl (5.8MB)\n",
            "\u001b[K     |████████████████████████████████| 5.8MB 42.2MB/s \n",
            "\u001b[?25hCollecting enum-compat==0.0.2\n",
            "  Downloading https://files.pythonhosted.org/packages/95/6e/26bdcba28b66126f66cf3e4cd03bcd63f7ae330d29ee68b1f6b623550bfa/enum-compat-0.0.2.tar.gz\n",
            "Collecting validators==0.12.4\n",
            "  Downloading https://files.pythonhosted.org/packages/70/c3/8f1f923c4ea9c620d0501073d6e2c829db09b717c8b1f3036d6455af3832/validators-0.12.4.tar.gz\n",
            "Requirement already satisfied: six>=1.10.0 in /usr/local/lib/python3.6/dist-packages (from oblib) (1.15.0)\n",
            "Collecting jsondiff==1.1.2\n",
            "  Downloading https://files.pythonhosted.org/packages/33/0c/ddb17571e061c655871ccbf76cdada55a31569327d21517de779d4887241/jsondiff-1.1.2.tar.gz\n",
            "Requirement already satisfied: decorator>=3.4.0 in /usr/local/lib/python3.6/dist-packages (from validators==0.12.4->oblib) (4.4.2)\n",
            "Building wheels for collected packages: enum-compat, validators, jsondiff\n",
            "  Building wheel for enum-compat (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "  Created wheel for enum-compat: filename=enum_compat-0.0.2-cp36-none-any.whl size=1290 sha256=c9cbee887cc5e5192d17b6f55ea2b509f68b9f2d4f4072e1bde1d07ff5c0daa9\n",
            "  Stored in directory: /root/.cache/pip/wheels/b1/69/f4/229af6a49beece0f688c9c73d9188769b89e698361d21ce96a\n",
            "  Building wheel for validators (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "  Created wheel for validators: filename=validators-0.12.4-cp36-none-any.whl size=15010 sha256=f6bf63a53895821128613b2804ba2e3ae01541f782bbd2538af51c5588856bac\n",
            "  Stored in directory: /root/.cache/pip/wheels/6c/aa/81/0ee6b1a0d69ab3db36228d3e621691710a13ed9bb072b6e565\n",
            "  Building wheel for jsondiff (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "  Created wheel for jsondiff: filename=jsondiff-1.1.2-cp36-none-any.whl size=6466 sha256=33271510f98602d3e61d988685401f93d8706064a9a0ec976349aee9942dbebf\n",
            "  Stored in directory: /root/.cache/pip/wheels/46/5f/86/11c6b72b064888e80b98bfcbcdaf2a83517a8cf8f2bb2a3227\n",
            "Successfully built enum-compat validators jsondiff\n",
            "Installing collected packages: lxml, enum-compat, validators, jsondiff, oblib\n",
            "  Found existing installation: lxml 4.2.6\n",
            "    Uninstalling lxml-4.2.6:\n",
            "      Successfully uninstalled lxml-4.2.6\n",
            "Successfully installed enum-compat-0.0.2 jsondiff-1.1.2 lxml-4.2.5 oblib-1.1.3 validators-0.12.4\n",
            "Requirement already satisfied: scikit-learn in /usr/local/lib/python3.6/dist-packages (0.22.2.post1)\n",
            "Requirement already satisfied: numpy>=1.11.0 in /usr/local/lib/python3.6/dist-packages (from scikit-learn) (1.19.5)\n",
            "Requirement already satisfied: scipy>=0.17.0 in /usr/local/lib/python3.6/dist-packages (from scikit-learn) (1.4.1)\n",
            "Requirement already satisfied: joblib>=0.11 in /usr/local/lib/python3.6/dist-packages (from scikit-learn) (1.0.0)\n",
            "Requirement already satisfied: spacy in /usr/local/lib/python3.6/dist-packages (2.2.4)\n",
            "Requirement already satisfied: catalogue<1.1.0,>=0.0.7 in /usr/local/lib/python3.6/dist-packages (from spacy) (1.0.0)\n",
            "Requirement already satisfied: requests<3.0.0,>=2.13.0 in /usr/local/lib/python3.6/dist-packages (from spacy) (2.23.0)\n",
            "Requirement already satisfied: plac<1.2.0,>=0.9.6 in /usr/local/lib/python3.6/dist-packages (from spacy) (1.1.3)\n",
            "Requirement already satisfied: srsly<1.1.0,>=1.0.2 in /usr/local/lib/python3.6/dist-packages (from spacy) (1.0.5)\n",
            "Requirement already satisfied: numpy>=1.15.0 in /usr/local/lib/python3.6/dist-packages (from spacy) (1.19.5)\n",
            "Requirement already satisfied: tqdm<5.0.0,>=4.38.0 in /usr/local/lib/python3.6/dist-packages (from spacy) (4.41.1)\n",
            "Requirement already satisfied: murmurhash<1.1.0,>=0.28.0 in /usr/local/lib/python3.6/dist-packages (from spacy) (1.0.5)\n",
            "Requirement already satisfied: setuptools in /usr/local/lib/python3.6/dist-packages (from spacy) (51.3.3)\n",
            "Requirement already satisfied: cymem<2.1.0,>=2.0.2 in /usr/local/lib/python3.6/dist-packages (from spacy) (2.0.5)\n",
            "Requirement already satisfied: thinc==7.4.0 in /usr/local/lib/python3.6/dist-packages (from spacy) (7.4.0)\n",
            "Requirement already satisfied: blis<0.5.0,>=0.4.0 in /usr/local/lib/python3.6/dist-packages (from spacy) (0.4.1)\n",
            "Requirement already satisfied: wasabi<1.1.0,>=0.4.0 in /usr/local/lib/python3.6/dist-packages (from spacy) (0.8.0)\n",
            "Requirement already satisfied: preshed<3.1.0,>=3.0.2 in /usr/local/lib/python3.6/dist-packages (from spacy) (3.0.5)\n",
            "Requirement already satisfied: importlib-metadata>=0.20; python_version < \"3.8\" in /usr/local/lib/python3.6/dist-packages (from catalogue<1.1.0,>=0.0.7->spacy) (3.3.0)\n",
            "Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.6/dist-packages (from requests<3.0.0,>=2.13.0->spacy) (2020.12.5)\n",
            "Requirement already satisfied: chardet<4,>=3.0.2 in /usr/local/lib/python3.6/dist-packages (from requests<3.0.0,>=2.13.0->spacy) (3.0.4)\n",
            "Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in /usr/local/lib/python3.6/dist-packages (from requests<3.0.0,>=2.13.0->spacy) (1.24.3)\n",
            "Requirement already satisfied: idna<3,>=2.5 in /usr/local/lib/python3.6/dist-packages (from requests<3.0.0,>=2.13.0->spacy) (2.10)\n",
            "Requirement already satisfied: typing-extensions>=3.6.4; python_version < \"3.8\" in /usr/local/lib/python3.6/dist-packages (from importlib-metadata>=0.20; python_version < \"3.8\"->catalogue<1.1.0,>=0.0.7->spacy) (3.7.4.3)\n",
            "Requirement already satisfied: zipp>=0.5 in /usr/local/lib/python3.6/dist-packages (from importlib-metadata>=0.20; python_version < \"3.8\"->catalogue<1.1.0,>=0.0.7->spacy) (3.4.0)\n",
            "Requirement already satisfied: torch in /usr/local/lib/python3.6/dist-packages (1.7.0+cu101)\n",
            "Requirement already satisfied: typing-extensions in /usr/local/lib/python3.6/dist-packages (from torch) (3.7.4.3)\n",
            "Requirement already satisfied: numpy in /usr/local/lib/python3.6/dist-packages (from torch) (1.19.5)\n",
            "Requirement already satisfied: dataclasses in /usr/local/lib/python3.6/dist-packages (from torch) (0.8)\n",
            "Requirement already satisfied: future in /usr/local/lib/python3.6/dist-packages (from torch) (0.16.0)\n",
            "Requirement already satisfied: textblob in /usr/local/lib/python3.6/dist-packages (0.15.3)\n",
            "Requirement already satisfied: nltk>=3.1 in /usr/local/lib/python3.6/dist-packages (from textblob) (3.2.5)\n",
            "Requirement already satisfied: six in /usr/local/lib/python3.6/dist-packages (from nltk>=3.1->textblob) (1.15.0)\n",
            "Collecting transformers\n",
            "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/88/b1/41130a228dd656a1a31ba281598a968320283f48d42782845f6ba567f00b/transformers-4.2.2-py3-none-any.whl (1.8MB)\n",
            "\u001b[K     |████████████████████████████████| 1.8MB 8.6MB/s \n",
            "\u001b[?25hCollecting sacremoses\n",
            "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/7d/34/09d19aff26edcc8eb2a01bed8e98f13a1537005d31e95233fd48216eed10/sacremoses-0.0.43.tar.gz (883kB)\n",
            "\u001b[K     |████████████████████████████████| 890kB 33.0MB/s \n",
            "\u001b[?25hRequirement already satisfied: dataclasses; python_version < \"3.7\" in /usr/local/lib/python3.6/dist-packages (from transformers) (0.8)\n",
            "Requirement already satisfied: regex!=2019.12.17 in /usr/local/lib/python3.6/dist-packages (from transformers) (2019.12.20)\n",
            "Requirement already satisfied: tqdm>=4.27 in /usr/local/lib/python3.6/dist-packages (from transformers) (4.41.1)\n",
            "Requirement already satisfied: packaging in /usr/local/lib/python3.6/dist-packages (from transformers) (20.8)\n",
            "Collecting tokenizers==0.9.4\n",
            "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/0f/1c/e789a8b12e28be5bc1ce2156cf87cb522b379be9cadc7ad8091a4cc107c4/tokenizers-0.9.4-cp36-cp36m-manylinux2010_x86_64.whl (2.9MB)\n",
            "\u001b[K     |████████████████████████████████| 2.9MB 22.1MB/s \n",
            "\u001b[?25hRequirement already satisfied: importlib-metadata; python_version < \"3.8\" in /usr/local/lib/python3.6/dist-packages (from transformers) (3.3.0)\n",
            "Requirement already satisfied: numpy in /usr/local/lib/python3.6/dist-packages (from transformers) (1.19.5)\n",
            "Requirement already satisfied: filelock in /usr/local/lib/python3.6/dist-packages (from transformers) (3.0.12)\n",
            "Requirement already satisfied: requests in /usr/local/lib/python3.6/dist-packages (from transformers) (2.23.0)\n",
            "Requirement already satisfied: six in /usr/local/lib/python3.6/dist-packages (from sacremoses->transformers) (1.15.0)\n",
            "Requirement already satisfied: click in /usr/local/lib/python3.6/dist-packages (from sacremoses->transformers) (7.1.2)\n",
            "Requirement already satisfied: joblib in /usr/local/lib/python3.6/dist-packages (from sacremoses->transformers) (1.0.0)\n",
            "Requirement already satisfied: pyparsing>=2.0.2 in /usr/local/lib/python3.6/dist-packages (from packaging->transformers) (2.4.7)\n",
            "Requirement already satisfied: zipp>=0.5 in /usr/local/lib/python3.6/dist-packages (from importlib-metadata; python_version < \"3.8\"->transformers) (3.4.0)\n",
            "Requirement already satisfied: typing-extensions>=3.6.4; python_version < \"3.8\" in /usr/local/lib/python3.6/dist-packages (from importlib-metadata; python_version < \"3.8\"->transformers) (3.7.4.3)\n",
            "Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.6/dist-packages (from requests->transformers) (2020.12.5)\n",
            "Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in /usr/local/lib/python3.6/dist-packages (from requests->transformers) (1.24.3)\n",
            "Requirement already satisfied: idna<3,>=2.5 in /usr/local/lib/python3.6/dist-packages (from requests->transformers) (2.10)\n",
            "Requirement already satisfied: chardet<4,>=3.0.2 in /usr/local/lib/python3.6/dist-packages (from requests->transformers) (3.0.4)\n",
            "Building wheels for collected packages: sacremoses\n",
            "  Building wheel for sacremoses (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "  Created wheel for sacremoses: filename=sacremoses-0.0.43-cp36-none-any.whl size=893261 sha256=1f06c02d64e2bb16038d987afe6f3c1469860691e9998642f13a74ba6d9f5c8f\n",
            "  Stored in directory: /root/.cache/pip/wheels/29/3c/fd/7ce5c3f0666dab31a50123635e6fb5e19ceb42ce38d4e58f45\n",
            "Successfully built sacremoses\n",
            "Installing collected packages: sacremoses, tokenizers, transformers\n",
            "Successfully installed sacremoses-0.0.43 tokenizers-0.9.4 transformers-4.2.2\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "XiIAA1D6Ukck",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "625a050a-95de-4440-b99d-5dd9dac1ded1"
      },
      "source": [
        "!git clone https://github.com/ProsusAI/finBERT\n",
        "%cd finBERT"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Cloning into 'finBERT'...\n",
            "remote: Enumerating objects: 146, done.\u001b[K\n",
            "remote: Counting objects: 100% (146/146), done.\u001b[K\n",
            "remote: Compressing objects: 100% (93/93), done.\u001b[K\n",
            "remote: Total 146 (delta 67), reused 121 (delta 50), pack-reused 0\u001b[K\n",
            "Receiving objects: 100% (146/146), 54.65 KiB | 10.93 MiB/s, done.\n",
            "Resolving deltas: 100% (67/67), done.\n",
            "/content/finBERT\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "ExecuteTime": {
          "end_time": "2020-03-23T15:55:04.902740Z",
          "start_time": "2020-03-23T15:55:04.876252Z"
        },
        "id": "OaVJYNsAtat6",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "c78f7a16-3bfd-42fc-95d2-fde7b783e61d"
      },
      "source": [
        "from pathlib import Path\n",
        "import shutil\n",
        "import os\n",
        "import logging\n",
        "import sys\n",
        "sys.path.append('..')\n",
        "\n",
        "from textblob import TextBlob\n",
        "from pprint import pprint\n",
        "from sklearn.metrics import classification_report\n",
        "\n",
        "from transformers import AutoModelForSequenceClassification\n",
        "\n",
        "from finbert.finbert import *\n",
        "import finbert.utils as tools\n",
        "\n",
        "%load_ext autoreload\n",
        "%autoreload 2\n",
        "\n",
        "project_dir = Path.cwd().parent\n",
        "pd.set_option('max_colwidth', -1)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "/usr/local/lib/python3.6/dist-packages/ipykernel_launcher.py:21: FutureWarning: Passing a negative integer is deprecated in version 1.0 and will not be supported in future version. Instead, use None to not limit the column width.\n"
          ],
          "name": "stderr"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ZrGrvtgLik83"
      },
      "source": [
        "# Import Models and Datasets"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "YYFJNim8XdKl",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 35
        },
        "outputId": "4029848a-61fc-461d-ba94-27ce0d7a7f16"
      },
      "source": [
        "!mkdir models\n",
        "!mkdir models/sentiment/\n",
        "!mkdir models/language_model/\n",
        "!mkdir models/classifier_model/\n",
        "!mkdir data\n",
        "!mkdir data/sentiment_data\n"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "string"
            },
            "text/plain": [
              "'/content/finBERT'"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 4
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Iq_izOfCVTFj",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "67e1573c-cb87-46ec-864d-ac309e409e44"
      },
      "source": [
        "!wget https://prosus-public.s3-eu-west-1.amazonaws.com/finbert/language-model/pytorch_model.bin -P /models/language_model/\n",
        "!wget https://prosus-public.s3-eu-west-1.amazonaws.com/finbert/finbert-sentiment/pytorch_model.bin -P /models/sentiment/\n",
        "!wget https://huggingface.co/ProsusAI/finbert/raw/main/config.json -P /models/language_models/\n",
        "!wget https://huggingface.co/ProsusAI/finbert/raw/main/config.json -P /models/sentiment"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "--2021-01-21 15:16:46--  https://prosus-public.s3-eu-west-1.amazonaws.com/finbert/language-model/pytorch_model.bin\n",
            "Resolving prosus-public.s3-eu-west-1.amazonaws.com (prosus-public.s3-eu-west-1.amazonaws.com)... 52.218.24.35\n",
            "Connecting to prosus-public.s3-eu-west-1.amazonaws.com (prosus-public.s3-eu-west-1.amazonaws.com)|52.218.24.35|:443... connected.\n",
            "HTTP request sent, awaiting response... 200 OK\n",
            "Length: 440477289 (420M) [application/octet-stream]\n",
            "Saving to: ‘/models/language_model/pytorch_model.bin.1’\n",
            "\n",
            "pytorch_model.bin.1 100%[===================>] 420.07M  45.4MB/s    in 8.6s    \n",
            "\n",
            "2021-01-21 15:16:55 (49.1 MB/s) - ‘/models/language_model/pytorch_model.bin.1’ saved [440477289/440477289]\n",
            "\n",
            "--2021-01-21 15:16:55--  https://prosus-public.s3-eu-west-1.amazonaws.com/finbert/finbert-sentiment/pytorch_model.bin\n",
            "Resolving prosus-public.s3-eu-west-1.amazonaws.com (prosus-public.s3-eu-west-1.amazonaws.com)... 52.218.29.24\n",
            "Connecting to prosus-public.s3-eu-west-1.amazonaws.com (prosus-public.s3-eu-west-1.amazonaws.com)|52.218.29.24|:443... connected.\n",
            "HTTP request sent, awaiting response... 200 OK\n",
            "Length: 437988463 (418M) [application/octet-stream]\n",
            "Saving to: ‘/models/sentiment/pytorch_model.bin.1’\n",
            "\n",
            "pytorch_model.bin.1 100%[===================>] 417.70M  75.9MB/s    in 5.3s    \n",
            "\n",
            "2021-01-21 15:17:01 (78.4 MB/s) - ‘/models/sentiment/pytorch_model.bin.1’ saved [437988463/437988463]\n",
            "\n",
            "--2021-01-21 15:17:01--  https://huggingface.co/ProsusAI/finbert/raw/main/config.json\n",
            "Resolving huggingface.co (huggingface.co)... 34.201.172.85\n",
            "Connecting to huggingface.co (huggingface.co)|34.201.172.85|:443... connected.\n",
            "HTTP request sent, awaiting response... 200 OK\n",
            "Length: 758 [application/json]\n",
            "Saving to: ‘/models/language_models/config.json.1’\n",
            "\n",
            "config.json.1       100%[===================>]     758  --.-KB/s    in 0s      \n",
            "\n",
            "2021-01-21 15:17:01 (117 MB/s) - ‘/models/language_models/config.json.1’ saved [758/758]\n",
            "\n",
            "--2021-01-21 15:17:01--  https://huggingface.co/ProsusAI/finbert/raw/main/config.json\n",
            "Resolving huggingface.co (huggingface.co)... 34.201.172.85\n",
            "Connecting to huggingface.co (huggingface.co)|34.201.172.85|:443... connected.\n",
            "HTTP request sent, awaiting response... 200 OK\n",
            "Length: 758 [application/json]\n",
            "Saving to: ‘/models/sentiment/config.json.1’\n",
            "\n",
            "config.json.1       100%[===================>]     758  --.-KB/s    in 0s      \n",
            "\n",
            "2021-01-21 15:17:02 (122 MB/s) - ‘/models/sentiment/config.json.1’ saved [758/758]\n",
            "\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "1-TYixG-hfb_"
      },
      "source": [
        "## Download FinancialPhraseBank Zip\n",
        "\n",
        "Wget wasn't working, not sure it can be automated\n",
        "\n",
        "Next couple cells:\n",
        "1) Upload zip to Colab\n",
        "2) Unzip with command\n",
        "3) Run the datasets script on the Sentences_50Agree.txt in the unzipped folder"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "0rDN4UGkcV1c",
        "colab": {
          "resources": {
            "http://localhost:8080/nbextensions/google.colab/files.js": {
              "data": "Ly8gQ29weXJpZ2h0IDIwMTcgR29vZ2xlIExMQwovLwovLyBMaWNlbnNlZCB1bmRlciB0aGUgQXBhY2hlIExpY2Vuc2UsIFZlcnNpb24gMi4wICh0aGUgIkxpY2Vuc2UiKTsKLy8geW91IG1heSBub3QgdXNlIHRoaXMgZmlsZSBleGNlcHQgaW4gY29tcGxpYW5jZSB3aXRoIHRoZSBMaWNlbnNlLgovLyBZb3UgbWF5IG9idGFpbiBhIGNvcHkgb2YgdGhlIExpY2Vuc2UgYXQKLy8KLy8gICAgICBodHRwOi8vd3d3LmFwYWNoZS5vcmcvbGljZW5zZXMvTElDRU5TRS0yLjAKLy8KLy8gVW5sZXNzIHJlcXVpcmVkIGJ5IGFwcGxpY2FibGUgbGF3IG9yIGFncmVlZCB0byBpbiB3cml0aW5nLCBzb2Z0d2FyZQovLyBkaXN0cmlidXRlZCB1bmRlciB0aGUgTGljZW5zZSBpcyBkaXN0cmlidXRlZCBvbiBhbiAiQVMgSVMiIEJBU0lTLAovLyBXSVRIT1VUIFdBUlJBTlRJRVMgT1IgQ09ORElUSU9OUyBPRiBBTlkgS0lORCwgZWl0aGVyIGV4cHJlc3Mgb3IgaW1wbGllZC4KLy8gU2VlIHRoZSBMaWNlbnNlIGZvciB0aGUgc3BlY2lmaWMgbGFuZ3VhZ2UgZ292ZXJuaW5nIHBlcm1pc3Npb25zIGFuZAovLyBsaW1pdGF0aW9ucyB1bmRlciB0aGUgTGljZW5zZS4KCi8qKgogKiBAZmlsZW92ZXJ2aWV3IEhlbHBlcnMgZm9yIGdvb2dsZS5jb2xhYiBQeXRob24gbW9kdWxlLgogKi8KKGZ1bmN0aW9uKHNjb3BlKSB7CmZ1bmN0aW9uIHNwYW4odGV4dCwgc3R5bGVBdHRyaWJ1dGVzID0ge30pIHsKICBjb25zdCBlbGVtZW50ID0gZG9jdW1lbnQuY3JlYXRlRWxlbWVudCgnc3BhbicpOwogIGVsZW1lbnQudGV4dENvbnRlbnQgPSB0ZXh0OwogIGZvciAoY29uc3Qga2V5IG9mIE9iamVjdC5rZXlzKHN0eWxlQXR0cmlidXRlcykpIHsKICAgIGVsZW1lbnQuc3R5bGVba2V5XSA9IHN0eWxlQXR0cmlidXRlc1trZXldOwogIH0KICByZXR1cm4gZWxlbWVudDsKfQoKLy8gTWF4IG51bWJlciBvZiBieXRlcyB3aGljaCB3aWxsIGJlIHVwbG9hZGVkIGF0IGEgdGltZS4KY29uc3QgTUFYX1BBWUxPQURfU0laRSA9IDEwMCAqIDEwMjQ7CgpmdW5jdGlvbiBfdXBsb2FkRmlsZXMoaW5wdXRJZCwgb3V0cHV0SWQpIHsKICBjb25zdCBzdGVwcyA9IHVwbG9hZEZpbGVzU3RlcChpbnB1dElkLCBvdXRwdXRJZCk7CiAgY29uc3Qgb3V0cHV0RWxlbWVudCA9IGRvY3VtZW50LmdldEVsZW1lbnRCeUlkKG91dHB1dElkKTsKICAvLyBDYWNoZSBzdGVwcyBvbiB0aGUgb3V0cHV0RWxlbWVudCB0byBtYWtlIGl0IGF2YWlsYWJsZSBmb3IgdGhlIG5leHQgY2FsbAogIC8vIHRvIHVwbG9hZEZpbGVzQ29udGludWUgZnJvbSBQeXRob24uCiAgb3V0cHV0RWxlbWVudC5zdGVwcyA9IHN0ZXBzOwoKICByZXR1cm4gX3VwbG9hZEZpbGVzQ29udGludWUob3V0cHV0SWQpOwp9CgovLyBUaGlzIGlzIHJvdWdobHkgYW4gYXN5bmMgZ2VuZXJhdG9yIChub3Qgc3VwcG9ydGVkIGluIHRoZSBicm93c2VyIHlldCksCi8vIHdoZXJlIHRoZXJlIGFyZSBtdWx0aXBsZSBhc3luY2hyb25vdXMgc3RlcHMgYW5kIHRoZSBQeXRob24gc2lkZSBpcyBnb2luZwovLyB0byBwb2xsIGZvciBjb21wbGV0aW9uIG9mIGVhY2ggc3RlcC4KLy8gVGhpcyB1c2VzIGEgUHJvbWlzZSB0byBibG9jayB0aGUgcHl0aG9uIHNpZGUgb24gY29tcGxldGlvbiBvZiBlYWNoIHN0ZXAsCi8vIHRoZW4gcGFzc2VzIHRoZSByZXN1bHQgb2YgdGhlIHByZXZpb3VzIHN0ZXAgYXMgdGhlIGlucHV0IHRvIHRoZSBuZXh0IHN0ZXAuCmZ1bmN0aW9uIF91cGxvYWRGaWxlc0NvbnRpbnVlKG91dHB1dElkKSB7CiAgY29uc3Qgb3V0cHV0RWxlbWVudCA9IGRvY3VtZW50LmdldEVsZW1lbnRCeUlkKG91dHB1dElkKTsKICBjb25zdCBzdGVwcyA9IG91dHB1dEVsZW1lbnQuc3RlcHM7CgogIGNvbnN0IG5leHQgPSBzdGVwcy5uZXh0KG91dHB1dEVsZW1lbnQubGFzdFByb21pc2VWYWx1ZSk7CiAgcmV0dXJuIFByb21pc2UucmVzb2x2ZShuZXh0LnZhbHVlLnByb21pc2UpLnRoZW4oKHZhbHVlKSA9PiB7CiAgICAvLyBDYWNoZSB0aGUgbGFzdCBwcm9taXNlIHZhbHVlIHRvIG1ha2UgaXQgYXZhaWxhYmxlIHRvIHRoZSBuZXh0CiAgICAvLyBzdGVwIG9mIHRoZSBnZW5lcmF0b3IuCiAgICBvdXRwdXRFbGVtZW50Lmxhc3RQcm9taXNlVmFsdWUgPSB2YWx1ZTsKICAgIHJldHVybiBuZXh0LnZhbHVlLnJlc3BvbnNlOwogIH0pOwp9CgovKioKICogR2VuZXJhdG9yIGZ1bmN0aW9uIHdoaWNoIGlzIGNhbGxlZCBiZXR3ZWVuIGVhY2ggYXN5bmMgc3RlcCBvZiB0aGUgdXBsb2FkCiAqIHByb2Nlc3MuCiAqIEBwYXJhbSB7c3RyaW5nfSBpbnB1dElkIEVsZW1lbnQgSUQgb2YgdGhlIGlucHV0IGZpbGUgcGlja2VyIGVsZW1lbnQuCiAqIEBwYXJhbSB7c3RyaW5nfSBvdXRwdXRJZCBFbGVtZW50IElEIG9mIHRoZSBvdXRwdXQgZGlzcGxheS4KICogQHJldHVybiB7IUl0ZXJhYmxlPCFPYmplY3Q+fSBJdGVyYWJsZSBvZiBuZXh0IHN0ZXBzLgogKi8KZnVuY3Rpb24qIHVwbG9hZEZpbGVzU3RlcChpbnB1dElkLCBvdXRwdXRJZCkgewogIGNvbnN0IGlucHV0RWxlbWVudCA9IGRvY3VtZW50LmdldEVsZW1lbnRCeUlkKGlucHV0SWQpOwogIGlucHV0RWxlbWVudC5kaXNhYmxlZCA9IGZhbHNlOwoKICBjb25zdCBvdXRwdXRFbGVtZW50ID0gZG9jdW1lbnQuZ2V0RWxlbWVudEJ5SWQob3V0cHV0SWQpOwogIG91dHB1dEVsZW1lbnQuaW5uZXJIVE1MID0gJyc7CgogIGNvbnN0IHBpY2tlZFByb21pc2UgPSBuZXcgUHJvbWlzZSgocmVzb2x2ZSkgPT4gewogICAgaW5wdXRFbGVtZW50LmFkZEV2ZW50TGlzdGVuZXIoJ2NoYW5nZScsIChlKSA9PiB7CiAgICAgIHJlc29sdmUoZS50YXJnZXQuZmlsZXMpOwogICAgfSk7CiAgfSk7CgogIGNvbnN0IGNhbmNlbCA9IGRvY3VtZW50LmNyZWF0ZUVsZW1lbnQoJ2J1dHRvbicpOwogIGlucHV0RWxlbWVudC5wYXJlbnRFbGVtZW50LmFwcGVuZENoaWxkKGNhbmNlbCk7CiAgY2FuY2VsLnRleHRDb250ZW50ID0gJ0NhbmNlbCB1cGxvYWQnOwogIGNvbnN0IGNhbmNlbFByb21pc2UgPSBuZXcgUHJvbWlzZSgocmVzb2x2ZSkgPT4gewogICAgY2FuY2VsLm9uY2xpY2sgPSAoKSA9PiB7CiAgICAgIHJlc29sdmUobnVsbCk7CiAgICB9OwogIH0pOwoKICAvLyBXYWl0IGZvciB0aGUgdXNlciB0byBwaWNrIHRoZSBmaWxlcy4KICBjb25zdCBmaWxlcyA9IHlpZWxkIHsKICAgIHByb21pc2U6IFByb21pc2UucmFjZShbcGlja2VkUHJvbWlzZSwgY2FuY2VsUHJvbWlzZV0pLAogICAgcmVzcG9uc2U6IHsKICAgICAgYWN0aW9uOiAnc3RhcnRpbmcnLAogICAgfQogIH07CgogIGNhbmNlbC5yZW1vdmUoKTsKCiAgLy8gRGlzYWJsZSB0aGUgaW5wdXQgZWxlbWVudCBzaW5jZSBmdXJ0aGVyIHBpY2tzIGFyZSBub3QgYWxsb3dlZC4KICBpbnB1dEVsZW1lbnQuZGlzYWJsZWQgPSB0cnVlOwoKICBpZiAoIWZpbGVzKSB7CiAgICByZXR1cm4gewogICAgICByZXNwb25zZTogewogICAgICAgIGFjdGlvbjogJ2NvbXBsZXRlJywKICAgICAgfQogICAgfTsKICB9CgogIGZvciAoY29uc3QgZmlsZSBvZiBmaWxlcykgewogICAgY29uc3QgbGkgPSBkb2N1bWVudC5jcmVhdGVFbGVtZW50KCdsaScpOwogICAgbGkuYXBwZW5kKHNwYW4oZmlsZS5uYW1lLCB7Zm9udFdlaWdodDogJ2JvbGQnfSkpOwogICAgbGkuYXBwZW5kKHNwYW4oCiAgICAgICAgYCgke2ZpbGUudHlwZSB8fCAnbi9hJ30pIC0gJHtmaWxlLnNpemV9IGJ5dGVzLCBgICsKICAgICAgICBgbGFzdCBtb2RpZmllZDogJHsKICAgICAgICAgICAgZmlsZS5sYXN0TW9kaWZpZWREYXRlID8gZmlsZS5sYXN0TW9kaWZpZWREYXRlLnRvTG9jYWxlRGF0ZVN0cmluZygpIDoKICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgJ24vYSd9IC0gYCkpOwogICAgY29uc3QgcGVyY2VudCA9IHNwYW4oJzAlIGRvbmUnKTsKICAgIGxpLmFwcGVuZENoaWxkKHBlcmNlbnQpOwoKICAgIG91dHB1dEVsZW1lbnQuYXBwZW5kQ2hpbGQobGkpOwoKICAgIGNvbnN0IGZpbGVEYXRhUHJvbWlzZSA9IG5ldyBQcm9taXNlKChyZXNvbHZlKSA9PiB7CiAgICAgIGNvbnN0IHJlYWRlciA9IG5ldyBGaWxlUmVhZGVyKCk7CiAgICAgIHJlYWRlci5vbmxvYWQgPSAoZSkgPT4gewogICAgICAgIHJlc29sdmUoZS50YXJnZXQucmVzdWx0KTsKICAgICAgfTsKICAgICAgcmVhZGVyLnJlYWRBc0FycmF5QnVmZmVyKGZpbGUpOwogICAgfSk7CiAgICAvLyBXYWl0IGZvciB0aGUgZGF0YSB0byBiZSByZWFkeS4KICAgIGxldCBmaWxlRGF0YSA9IHlpZWxkIHsKICAgICAgcHJvbWlzZTogZmlsZURhdGFQcm9taXNlLAogICAgICByZXNwb25zZTogewogICAgICAgIGFjdGlvbjogJ2NvbnRpbnVlJywKICAgICAgfQogICAgfTsKCiAgICAvLyBVc2UgYSBjaHVua2VkIHNlbmRpbmcgdG8gYXZvaWQgbWVzc2FnZSBzaXplIGxpbWl0cy4gU2VlIGIvNjIxMTU2NjAuCiAgICBsZXQgcG9zaXRpb24gPSAwOwogICAgd2hpbGUgKHBvc2l0aW9uIDwgZmlsZURhdGEuYnl0ZUxlbmd0aCkgewogICAgICBjb25zdCBsZW5ndGggPSBNYXRoLm1pbihmaWxlRGF0YS5ieXRlTGVuZ3RoIC0gcG9zaXRpb24sIE1BWF9QQVlMT0FEX1NJWkUpOwogICAgICBjb25zdCBjaHVuayA9IG5ldyBVaW50OEFycmF5KGZpbGVEYXRhLCBwb3NpdGlvbiwgbGVuZ3RoKTsKICAgICAgcG9zaXRpb24gKz0gbGVuZ3RoOwoKICAgICAgY29uc3QgYmFzZTY0ID0gYnRvYShTdHJpbmcuZnJvbUNoYXJDb2RlLmFwcGx5KG51bGwsIGNodW5rKSk7CiAgICAgIHlpZWxkIHsKICAgICAgICByZXNwb25zZTogewogICAgICAgICAgYWN0aW9uOiAnYXBwZW5kJywKICAgICAgICAgIGZpbGU6IGZpbGUubmFtZSwKICAgICAgICAgIGRhdGE6IGJhc2U2NCwKICAgICAgICB9LAogICAgICB9OwogICAgICBwZXJjZW50LnRleHRDb250ZW50ID0KICAgICAgICAgIGAke01hdGgucm91bmQoKHBvc2l0aW9uIC8gZmlsZURhdGEuYnl0ZUxlbmd0aCkgKiAxMDApfSUgZG9uZWA7CiAgICB9CiAgfQoKICAvLyBBbGwgZG9uZS4KICB5aWVsZCB7CiAgICByZXNwb25zZTogewogICAgICBhY3Rpb246ICdjb21wbGV0ZScsCiAgICB9CiAgfTsKfQoKc2NvcGUuZ29vZ2xlID0gc2NvcGUuZ29vZ2xlIHx8IHt9OwpzY29wZS5nb29nbGUuY29sYWIgPSBzY29wZS5nb29nbGUuY29sYWIgfHwge307CnNjb3BlLmdvb2dsZS5jb2xhYi5fZmlsZXMgPSB7CiAgX3VwbG9hZEZpbGVzLAogIF91cGxvYWRGaWxlc0NvbnRpbnVlLAp9Owp9KShzZWxmKTsK",
              "ok": true,
              "headers": [
                [
                  "content-type",
                  "application/javascript"
                ]
              ],
              "status": 200,
              "status_text": ""
            }
          },
          "base_uri": "https://localhost:8080/",
          "height": 89
        },
        "outputId": "5e921866-913f-4c0f-edc9-79eea68ac839"
      },
      "source": [
        "from google.colab import files\n",
        "\n",
        "uploaded = files.upload()\n",
        "\n",
        "for fn in uploaded.keys():\n",
        "  print('User uploaded file \"{name}\" with length {length} bytes'.format(\n",
        "      name=fn, length=len(uploaded[fn])))"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "\n",
              "     <input type=\"file\" id=\"files-b0437913-262a-4229-901c-89b162ed3939\" name=\"files[]\" multiple disabled\n",
              "        style=\"border:none\" />\n",
              "     <output id=\"result-b0437913-262a-4229-901c-89b162ed3939\">\n",
              "      Upload widget is only available when the cell has been executed in the\n",
              "      current browser session. Please rerun this cell to enable.\n",
              "      </output>\n",
              "      <script src=\"/nbextensions/google.colab/files.js\"></script> "
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "stream",
          "text": [
            "Saving FinancialPhraseBank-v1.0.zip to FinancialPhraseBank-v1.0.zip\n",
            "User uploaded file \"FinancialPhraseBank-v1.0.zip\" with length 681890 bytes\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "63EK1oCmcUKp",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "fdfcbc5e-d6c8-4b7b-e3c5-7f20e3cc7b45"
      },
      "source": [
        "!unzip FinancialPhraseBank-v1.0.zip"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Archive:  FinancialPhraseBank-v1.0.zip\n",
            "   creating: FinancialPhraseBank-v1.0/\n",
            "  inflating: FinancialPhraseBank-v1.0/License.txt  \n",
            "   creating: __MACOSX/\n",
            "   creating: __MACOSX/FinancialPhraseBank-v1.0/\n",
            "  inflating: __MACOSX/FinancialPhraseBank-v1.0/._License.txt  \n",
            "  inflating: FinancialPhraseBank-v1.0/README.txt  \n",
            "  inflating: __MACOSX/FinancialPhraseBank-v1.0/._README.txt  \n",
            "  inflating: FinancialPhraseBank-v1.0/Sentences_50Agree.txt  \n",
            "  inflating: FinancialPhraseBank-v1.0/Sentences_66Agree.txt  \n",
            "  inflating: FinancialPhraseBank-v1.0/Sentences_75Agree.txt  \n",
            "  inflating: FinancialPhraseBank-v1.0/Sentences_AllAgree.txt  \n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "VA3P09Jjc1LQ",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "df6eebd1-220a-4218-cdbd-374124cdc8c6"
      },
      "source": [
        "!python scripts/datasets.py --data_path /content/finBERT/FinancialPhraseBank-v1.0/Sentences_50Agree.txt"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "scripts/datasets.py:13: ParserWarning: Falling back to the 'python' engine because the 'c' engine does not support regex separators (separators > 1 char and different from '\\s+' are interpreted as regex); you can avoid this warning by specifying engine='python'.\n",
            "  data = pd.read_csv(args.data_path, sep='.@', names=['text','label'])\n",
            "Traceback (most recent call last):\n",
            "  File \"scripts/datasets.py\", line 13, in <module>\n",
            "    data = pd.read_csv(args.data_path, sep='.@', names=['text','label'])\n",
            "  File \"/usr/local/lib/python3.6/dist-packages/pandas/io/parsers.py\", line 688, in read_csv\n",
            "    return _read(filepath_or_buffer, kwds)\n",
            "  File \"/usr/local/lib/python3.6/dist-packages/pandas/io/parsers.py\", line 454, in _read\n",
            "    parser = TextFileReader(fp_or_buf, **kwds)\n",
            "  File \"/usr/local/lib/python3.6/dist-packages/pandas/io/parsers.py\", line 948, in __init__\n",
            "    self._make_engine(self.engine)\n",
            "  File \"/usr/local/lib/python3.6/dist-packages/pandas/io/parsers.py\", line 1191, in _make_engine\n",
            "    self._engine = klass(self.f, **self.options)\n",
            "  File \"/usr/local/lib/python3.6/dist-packages/pandas/io/parsers.py\", line 2389, in __init__\n",
            "    memory_map=self.memory_map,\n",
            "  File \"/usr/local/lib/python3.6/dist-packages/pandas/io/common.py\", line 496, in get_handle\n",
            "    f = open(path_or_buf, mode, errors=\"replace\", newline=\"\")\n",
            "FileNotFoundError: [Errno 2] No such file or directory: '/content/finBERT/FinancialPhraseBank-v1.0/Sentences_50Agree.txt'\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "m46GvF2Qtat5"
      },
      "source": [
        "## Modules "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "ExecuteTime": {
          "end_time": "2020-03-23T15:55:05.711210Z",
          "start_time": "2020-03-23T15:55:05.693609Z"
        },
        "id": "4xpaPtqxtat8"
      },
      "source": [
        "logging.basicConfig(format = '%(asctime)s - %(levelname)s - %(name)s -   %(message)s',\n",
        "                    datefmt = '%m/%d/%Y %H:%M:%S',\n",
        "                    level = logging.ERROR)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "VFg0Y5GKtat9"
      },
      "source": [
        "## Prepare the model"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "3jvTYlpmtat-"
      },
      "source": [
        "### Setting path variables:\n",
        "1. `lm_path`: the path for the pre-trained language model (If vanilla Bert is used then no need to set this one).\n",
        "2. `cl_path`: the path where the classification model is saved.\n",
        "3. `cl_data_path`: the path of the directory that contains the data files of `train.csv`, `validation.csv`, `test.csv`.\n",
        "---\n",
        "\n",
        "In the initialization of `bertmodel`, we can either use the original pre-trained weights from Google by giving `bm = 'bert-base-uncased`, or our further pre-trained language model by `bm = lm_path`\n",
        "\n",
        "\n",
        "---\n",
        "All of the configurations with the model is controlled with the `config` variable. "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "ExecuteTime": {
          "end_time": "2020-03-23T15:55:07.405597Z",
          "start_time": "2020-03-23T15:55:07.386378Z"
        },
        "id": "syfkYYUctat-"
      },
      "source": [
        "lm_path = project_dir/'models'/'language_model'/'finbertTRC2'\n",
        "cl_path = project_dir/'models'/'classifier_model'/'finbert-sentiment'\n",
        "cl_data_path = project_dir/'data'/'sentiment_data'"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "KTFN_fr5tat_"
      },
      "source": [
        "###  Configuring training parameters"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "O_eTbCmqtat_"
      },
      "source": [
        "You can find the explanations of the training parameters in the class docsctrings. "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "ExecuteTime": {
          "end_time": "2020-03-23T15:55:12.378583Z",
          "start_time": "2020-03-23T15:55:09.196746Z"
        },
        "id": "AOe5OjBntauA",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 650
        },
        "outputId": "62344bc5-7750-4429-c434-0a068d7acd67"
      },
      "source": [
        "# Clean the cl_path\n",
        "try:\n",
        "    shutil.rmtree(cl_path) \n",
        "except:\n",
        "    pass\n",
        "\n",
        "bertmodel = AutoModelForSequenceClassification.from_pretrained(lm_path,cache_dir=None, num_labels=3)\n",
        "\n",
        "\n",
        "config = Config(   data_dir=cl_data_path,\n",
        "                   bert_model=bertmodel,\n",
        "                   num_train_epochs=4,\n",
        "                   model_dir=cl_path,\n",
        "                   max_seq_length = 48,\n",
        "                   train_batch_size = 32,\n",
        "                   learning_rate = 2e-5,\n",
        "                   output_mode='classification',\n",
        "                   warm_up_proportion=0.2,\n",
        "                   local_rank=-1,\n",
        "                   discriminate=True,\n",
        "                   gradual_unfreeze=True)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "404 Client Error: Not Found for url: https://huggingface.co//content/finBERT/finBERT/models/language_model/finbertTRC2/resolve/main/config.json\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "error",
          "ename": "OSError",
          "evalue": "ignored",
          "traceback": [
            "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
            "\u001b[0;31mHTTPError\u001b[0m                                 Traceback (most recent call last)",
            "\u001b[0;32m/usr/local/lib/python3.6/dist-packages/transformers/configuration_utils.py\u001b[0m in \u001b[0;36mget_config_dict\u001b[0;34m(cls, pretrained_model_name_or_path, **kwargs)\u001b[0m\n\u001b[1;32m    419\u001b[0m                 \u001b[0mlocal_files_only\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mlocal_files_only\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 420\u001b[0;31m                 \u001b[0muse_auth_token\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0muse_auth_token\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    421\u001b[0m             )\n",
            "\u001b[0;32m/usr/local/lib/python3.6/dist-packages/transformers/file_utils.py\u001b[0m in \u001b[0;36mcached_path\u001b[0;34m(url_or_filename, cache_dir, force_download, proxies, resume_download, user_agent, extract_compressed_file, force_extract, use_auth_token, local_files_only)\u001b[0m\n\u001b[1;32m   1055\u001b[0m             \u001b[0muse_auth_token\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0muse_auth_token\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 1056\u001b[0;31m             \u001b[0mlocal_files_only\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mlocal_files_only\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m   1057\u001b[0m         )\n",
            "\u001b[0;32m/usr/local/lib/python3.6/dist-packages/transformers/file_utils.py\u001b[0m in \u001b[0;36mget_from_cache\u001b[0;34m(url, cache_dir, force_download, proxies, etag_timeout, resume_download, user_agent, use_auth_token, local_files_only)\u001b[0m\n\u001b[1;32m   1185\u001b[0m             \u001b[0mr\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mrequests\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mhead\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0murl\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mheaders\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mheaders\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mallow_redirects\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;32mFalse\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mproxies\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mproxies\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mtimeout\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0metag_timeout\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 1186\u001b[0;31m             \u001b[0mr\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mraise_for_status\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m   1187\u001b[0m             \u001b[0metag\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mr\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mheaders\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mget\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\"X-Linked-Etag\"\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;32mor\u001b[0m \u001b[0mr\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mheaders\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mget\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\"ETag\"\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/local/lib/python3.6/dist-packages/requests/models.py\u001b[0m in \u001b[0;36mraise_for_status\u001b[0;34m(self)\u001b[0m\n\u001b[1;32m    940\u001b[0m         \u001b[0;32mif\u001b[0m \u001b[0mhttp_error_msg\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 941\u001b[0;31m             \u001b[0;32mraise\u001b[0m \u001b[0mHTTPError\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mhttp_error_msg\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mresponse\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    942\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;31mHTTPError\u001b[0m: 404 Client Error: Not Found for url: https://huggingface.co//content/finBERT/finBERT/models/language_model/finbertTRC2/resolve/main/config.json",
            "\nDuring handling of the above exception, another exception occurred:\n",
            "\u001b[0;31mOSError\u001b[0m                                   Traceback (most recent call last)",
            "\u001b[0;32m<ipython-input-21-0e942551ef2c>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m()\u001b[0m\n\u001b[1;32m      5\u001b[0m     \u001b[0;32mpass\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m      6\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 7\u001b[0;31m \u001b[0mbertmodel\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mAutoModelForSequenceClassification\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mfrom_pretrained\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mlm_path\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0mcache_dir\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;32mNone\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mnum_labels\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;36m3\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m      8\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m      9\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/local/lib/python3.6/dist-packages/transformers/models/auto/modeling_auto.py\u001b[0m in \u001b[0;36mfrom_pretrained\u001b[0;34m(cls, pretrained_model_name_or_path, *model_args, **kwargs)\u001b[0m\n\u001b[1;32m   1310\u001b[0m         \u001b[0;32mif\u001b[0m \u001b[0;32mnot\u001b[0m \u001b[0misinstance\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mconfig\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mPretrainedConfig\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1311\u001b[0m             config, kwargs = AutoConfig.from_pretrained(\n\u001b[0;32m-> 1312\u001b[0;31m                 \u001b[0mpretrained_model_name_or_path\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mreturn_unused_kwargs\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;32mTrue\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m   1313\u001b[0m             )\n\u001b[1;32m   1314\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/local/lib/python3.6/dist-packages/transformers/models/auto/configuration_auto.py\u001b[0m in \u001b[0;36mfrom_pretrained\u001b[0;34m(cls, pretrained_model_name_or_path, **kwargs)\u001b[0m\n\u001b[1;32m    358\u001b[0m             \u001b[0;34m{\u001b[0m\u001b[0;34m'foo'\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0;32mFalse\u001b[0m\u001b[0;34m}\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    359\u001b[0m         \"\"\"\n\u001b[0;32m--> 360\u001b[0;31m         \u001b[0mconfig_dict\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0m_\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mPretrainedConfig\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mget_config_dict\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mpretrained_model_name_or_path\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    361\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    362\u001b[0m         \u001b[0;32mif\u001b[0m \u001b[0;34m\"model_type\"\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mconfig_dict\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/local/lib/python3.6/dist-packages/transformers/configuration_utils.py\u001b[0m in \u001b[0;36mget_config_dict\u001b[0;34m(cls, pretrained_model_name_or_path, **kwargs)\u001b[0m\n\u001b[1;32m    430\u001b[0m                 \u001b[0;34mf\"- or '{pretrained_model_name_or_path}' is the correct path to a directory containing a {CONFIG_NAME} file\\n\\n\"\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    431\u001b[0m             )\n\u001b[0;32m--> 432\u001b[0;31m             \u001b[0;32mraise\u001b[0m \u001b[0mEnvironmentError\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mmsg\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    433\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    434\u001b[0m         \u001b[0;32mexcept\u001b[0m \u001b[0mjson\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mJSONDecodeError\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;31mOSError\u001b[0m: Can't load config for '/content/finBERT/finBERT/models/language_model/finbertTRC2'. Make sure that:\n\n- '/content/finBERT/finBERT/models/language_model/finbertTRC2' is a correct model identifier listed on 'https://huggingface.co/models'\n\n- or '/content/finBERT/finBERT/models/language_model/finbertTRC2' is the correct path to a directory containing a config.json file\n\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "-IagArA0tauC"
      },
      "source": [
        "`finbert` is our main class that encapsulates all the functionality. The list of class labels should be given in the prepare_model method call with label_list parameter."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "ExecuteTime": {
          "end_time": "2020-03-23T15:55:16.657078Z",
          "start_time": "2020-03-23T15:55:16.639644Z"
        },
        "id": "j0DVhx0FtauD"
      },
      "source": [
        "finbert = FinBert(config)\n",
        "finbert.base_model = 'bert-base-uncased'\n",
        "finbert.config.discriminate=True\n",
        "finbert.config.gradual_unfreeze=True"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "ExecuteTime": {
          "end_time": "2020-03-23T15:55:17.850734Z",
          "start_time": "2020-03-23T15:55:17.368073Z"
        },
        "id": "Ja2Ug4autauE"
      },
      "source": [
        "finbert.prepare_model(label_list=['positive','negative','neutral'])"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "kkUsGlqXtauE"
      },
      "source": [
        "## Fine-tune the model"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "ExecuteTime": {
          "end_time": "2020-03-23T15:55:19.395707Z",
          "start_time": "2020-03-23T15:55:19.349642Z"
        },
        "scrolled": true,
        "id": "VyZQWr6ltauF"
      },
      "source": [
        "# Get the training examples\n",
        "train_data = finbert.get_data('train')"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "ExecuteTime": {
          "end_time": "2020-03-23T15:55:25.912424Z",
          "start_time": "2020-03-23T15:55:20.065887Z"
        },
        "id": "-bASZW60tauF"
      },
      "source": [
        "model = finbert.create_the_model()"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "_wCA8uYCtauF"
      },
      "source": [
        "### [Optional] Fine-tune only a subset of the model\n",
        "The variable `freeze` determines the last layer (out of 12) to be freezed. You can skip this part if you want to fine-tune the whole model.\n",
        "\n",
        "<span style=\"color:red\">Important: </span>\n",
        "Execute this step if you want a shorter training time in the expense of accuracy."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "WcVpUgDStauF"
      },
      "source": [
        "# This is for fine-tuning a subset of the model.\n",
        "\n",
        "freeze = 6\n",
        "\n",
        "for param in model.bert.embeddings.parameters():\n",
        "    param.requires_grad = False\n",
        "    \n",
        "for i in range(freeze):\n",
        "    for param in model.bert.encoder.layer[i].parameters():\n",
        "        param.requires_grad = False"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "3odoFQXXtauH"
      },
      "source": [
        "### Training"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "ExecuteTime": {
          "end_time": "2020-03-23T15:58:35.486890Z",
          "start_time": "2020-03-23T15:55:27.293772Z"
        },
        "scrolled": true,
        "id": "yU3233n1tauH"
      },
      "source": [
        "trained_model = finbert.train(train_examples = train_data, model = model)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "fa683E3ktauI"
      },
      "source": [
        "## Test the model\n",
        "\n",
        "`bert.evaluate` outputs the DataFrame, where true labels and logit values for each example is given"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "ExecuteTime": {
          "end_time": "2020-03-23T15:58:40.056789Z",
          "start_time": "2020-03-23T15:58:40.023198Z"
        },
        "id": "GnJwGjwptauI"
      },
      "source": [
        "test_data = finbert.get_data('test')"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "ExecuteTime": {
          "end_time": "2020-03-23T15:58:48.248044Z",
          "start_time": "2020-03-23T15:58:41.699009Z"
        },
        "scrolled": true,
        "id": "AjAfmFAJtauJ"
      },
      "source": [
        "results = finbert.evaluate(examples=test_data, model=trained_model)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Ag_yjM0ytauJ"
      },
      "source": [
        "### Prepare the classification report"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "ExecuteTime": {
          "end_time": "2020-03-23T15:58:51.361079Z",
          "start_time": "2020-03-23T15:58:51.339548Z"
        },
        "id": "Q39wjsUetauJ"
      },
      "source": [
        "def report(df, cols=['label','prediction','logits']):\n",
        "    #print('Validation loss:{0:.2f}'.format(metrics['best_validation_loss']))\n",
        "    cs = CrossEntropyLoss(weight=finbert.class_weights)\n",
        "    loss = cs(torch.tensor(list(df[cols[2]])),torch.tensor(list(df[cols[0]])))\n",
        "    print(\"Loss:{0:.2f}\".format(loss))\n",
        "    print(\"Accuracy:{0:.2f}\".format((df[cols[0]] == df[cols[1]]).sum() / df.shape[0]) )\n",
        "    print(\"\\nClassification Report:\")\n",
        "    print(classification_report(df[cols[0]], df[cols[1]]))"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "ExecuteTime": {
          "end_time": "2020-03-23T15:58:53.190447Z",
          "start_time": "2020-03-23T15:58:53.166729Z"
        },
        "id": "YDzFK1NptauJ"
      },
      "source": [
        "results['prediction'] = results.predictions.apply(lambda x: np.argmax(x,axis=0))"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "ExecuteTime": {
          "end_time": "2020-03-23T15:58:54.436270Z",
          "start_time": "2020-03-23T15:58:54.399174Z"
        },
        "scrolled": true,
        "id": "M5zqzpWKtauK"
      },
      "source": [
        "report(results,cols=['labels','prediction','predictions'])"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "DpVTBz_htauK"
      },
      "source": [
        "### Get predictions"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Mbc9wkm1tauL"
      },
      "source": [
        "With the `predict` function, given a piece of text, we split it into a list of sentences and then predict sentiment for each sentence. The output is written into a dataframe. Predictions are represented in three different columns: \n",
        "\n",
        "1) `logit`: probabilities for each class\n",
        "\n",
        "2) `prediction`: predicted label\n",
        "\n",
        "3) `sentiment_score`: sentiment score calculated as: probability of positive - probability of negative\n",
        "\n",
        "Below we analyze a paragraph taken out of [this](https://www.economist.com/finance-and-economics/2019/01/03/a-profit-warning-from-apple-jolts-markets) article from The Economist. For comparison purposes, we also put the sentiments predicted with TextBlob.\n",
        "> Later that day Apple said it was revising down its earnings expectations in the fourth quarter of 2018, largely because of lower sales and signs of economic weakness in China. The news rapidly infected financial markets. Apple’s share price fell by around 7% in after-hours trading and the decline was extended to more than 10% when the market opened. The dollar fell by 3.7% against the yen in a matter of minutes after the announcement, before rapidly recovering some ground. Asian stockmarkets closed down on January 3rd and European ones opened lower. Yields on government bonds fell as investors fled to the traditional haven in a market storm."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "ExecuteTime": {
          "end_time": "2020-03-23T15:59:03.875213Z",
          "start_time": "2020-03-23T15:59:03.857612Z"
        },
        "id": "_6pz52lvtauL"
      },
      "source": [
        "text = \"Later that day Apple said it was revising down its earnings expectations in \\\n",
        "the fourth quarter of 2018, largely because of lower sales and signs of economic weakness in China. \\\n",
        "The news rapidly infected financial markets. Apple’s share price fell by around 7% in after-hours \\\n",
        "trading and the decline was extended to more than 10% when the market opened. The dollar fell \\\n",
        "by 3.7% against the yen in a matter of minutes after the announcement, before rapidly recovering \\\n",
        "some ground. Asian stockmarkets closed down on January 3rd and European ones opened lower. \\\n",
        "Yields on government bonds fell as investors fled to the traditional haven in a market storm.\""
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "ExecuteTime": {
          "end_time": "2020-03-23T15:59:16.246963Z",
          "start_time": "2020-03-23T15:59:13.285393Z"
        },
        "scrolled": true,
        "id": "eFX-TveytauM"
      },
      "source": [
        "cl_path = project_dir/'models'/'classifier_model'/'finbert-sentiment'\n",
        "model = AutoModelForSequenceClassification.from_pretrained(cl_path, cache_dir=None, num_labels=3)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "wQzDgiPYtauM"
      },
      "source": [
        "import nltk\n",
        "nltk.download('punkt')"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "ExecuteTime": {
          "end_time": "2020-03-23T15:59:19.531663Z",
          "start_time": "2020-03-23T15:59:17.744984Z"
        },
        "scrolled": true,
        "id": "oLCX5nvVtauM"
      },
      "source": [
        "result = predict(text,model)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "ExecuteTime": {
          "end_time": "2020-03-23T15:59:20.519047Z",
          "start_time": "2020-03-23T15:59:20.440450Z"
        },
        "id": "2Joh8C5-tauN"
      },
      "source": [
        "blob = TextBlob(text)\n",
        "result['textblob_prediction'] = [sentence.sentiment.polarity for sentence in blob.sentences]\n",
        "result"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "ExecuteTime": {
          "end_time": "2020-03-23T15:59:38.737969Z",
          "start_time": "2020-03-23T15:59:38.718255Z"
        },
        "id": "vgMe0WmKtauO"
      },
      "source": [
        "print(f'Average sentiment is %.2f.' % (result.sentiment_score.mean()))"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "pfTMS2hAtauO"
      },
      "source": [
        "Here is another example"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "ExecuteTime": {
          "end_time": "2020-03-23T15:59:45.922058Z",
          "start_time": "2020-03-23T15:59:45.904622Z"
        },
        "id": "n4zMLJl5tauO"
      },
      "source": [
        "text2 = \"Shares in the spin-off of South African e-commerce group Naspers surged more than 25% \\\n",
        "in the first minutes of their market debut in Amsterdam on Wednesday. Bob van Dijk, CEO of \\\n",
        "Naspers and Prosus Group poses at Amsterdam's stock exchange, as Prosus begins trading on the \\\n",
        "Euronext stock exchange in Amsterdam, Netherlands, September 11, 2019. REUTERS/Piroschka van de Wouw \\\n",
        "Prosus comprises Naspers’ global empire of consumer internet assets, with the jewel in the crown a \\\n",
        "31% stake in Chinese tech titan Tencent. There is 'way more demand than is even available, so that’s \\\n",
        "good,' said the CEO of Euronext Amsterdam, Maurice van Tilburg. 'It’s going to be an interesting \\\n",
        "hour of trade after opening this morning.' Euronext had given an indicative price of 58.70 euros \\\n",
        "per share for Prosus, implying a market value of 95.3 billion euros ($105 billion). The shares \\\n",
        "jumped to 76 euros on opening and were trading at 75 euros at 0719 GMT.\""
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "ExecuteTime": {
          "end_time": "2020-03-23T15:59:48.152474Z",
          "start_time": "2020-03-23T15:59:47.028417Z"
        },
        "id": "PGXrzZNrtauP"
      },
      "source": [
        "result2 = predict(text2,model)\n",
        "blob = TextBlob(text2)\n",
        "result2['textblob_prediction'] = [sentence.sentiment.polarity for sentence in blob.sentences]"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "ExecuteTime": {
          "end_time": "2020-03-23T15:59:50.428951Z",
          "start_time": "2020-03-23T15:59:50.402385Z"
        },
        "id": "81AyGPmetauP"
      },
      "source": [
        "result2"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "ExecuteTime": {
          "end_time": "2020-03-23T16:00:27.031491Z",
          "start_time": "2020-03-23T16:00:27.012639Z"
        },
        "id": "z7i3hGk8tauP"
      },
      "source": [
        "print(f'Average sentiment is %.2f.' % (result2.sentiment_score.mean()))"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "0iKql9cstauP"
      },
      "source": [
        ""
      ],
      "execution_count": null,
      "outputs": []
    }
  ]
}