training-log.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# A training log"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- `step 1000`: train loss 5.9037, `val loss 5.8993`\n",
    "\n",
    "![At 1000 epochs](./training-log/1000-epochs.png \"1000 Epochs\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- `step 1500`: train loss 5.6095, `val loss 5.5964`\n",
    "- interestingly, more gibberish. will run more epochs before checking again.\n",
    "\n",
    "![At 1500 epochs](./training-log/1500-epochs.png \"1500 Epochs\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- `step 3000`: train loss 5.0643, `val loss 5.0579`\n",
    "- seems very \"news\"-y?\n",
    "\n",
    "![At 3000 epochs](./training-log/3000-epochs.png \"3000 Epochs\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- `step 5500`: train loss 4.6092, `val loss 4.6042`\n",
    "- output is looking more coherent, but still very \"news\"-y\n",
    "- let's wait until 10k epochs and then see if we need to bump up the learning rate\n",
    "\n",
    "![At 5500 epochs](./training-log/5500-epochs.png \"5500 Epochs\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- `step 50k`, `val loss: 3.628270149230957`\n",
    "- ramped up training to 600k overall steps and left overnight\n",
    "- brought in cosine annealing with linear warmup on the first 2k steps (per paper)\n",
    "- implemented checkpoints so I can resume training across instances\n",
    "- switched to the A10G GPU instances with 4 cores\n",
    "- pulled entire training/val datasets into memory which greatly increased GPU throughput (see images below):\n",
    "\n",
    "![Train to Val before Data in Mem](./training-log/train-to-val-before.png \"Train to Val before data in mem\")\n",
    "\n",
    "Before data pulled into memory. Left is GPU utilization seeking across the 18GB `train.bin` file, right is from 85MB `val.bin` file. You can see the huge discrepancy in throughput randomly reading from each via `np.memmap`.\n",
    "\n",
    "![Train to Val after Data in Mem](./training-log/train-to-val-after.png \"Train to Val after data in mem\")\n",
    "\n",
    "Utilization results after pulling the whole dataset into memory. Notes from left-to-right with dips between (representing code execution moving to a new stage):\n",
    "\n",
    "1. GPU Utilization of pulling from `train.bin` for loss estimation - much higher without the dips going to 0% utilization.\n",
    "2. GPU Utilization of pulling from `val.bin` for loss estimation - notice that this is the same as randomly pulling from `train.bin`.\n",
    "3. GPU Utilization of randomly pulling from `train.bin` for training. Somehow this is even higher than randomly pulling from the file for loss estimation with dips sometimes not dropping all the way to 50%. I'm guessing this is because we are also saving to the checkpoint file vs. not in the training case.\n",
    "\n",
    "![At 50k epochs](./training-log/50k-epochs.png \"50k Epochs\")\n",
    "\n",
    "After training for some time, you can see much better results with a loss that's in the `~3.6` range. The output seems a lot more conversational. I'm going to keep this running (likely for a full day or 2) until I get diminishing returns on loss. I would love to see if I could get away with even faster using an A100 or H100, but I don't want to spend that kind of cash on idle given how noobish I still am at all of this stuff and because I'd have to either use a 3rd party like Lambda (which is more of a pain) and/or upgrade to the next pricing tier."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- `step 73k-93k`: seems like our loss has stalled a bit. Not sure why. Thinking.\n",
    "\n",
    "![Stalled loss](./training-log/73k-93k.png \"Stalled loss\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- after getting stalled out at `~3.5 val loss` in the previous model, I went through the final course of Zero to Hero and made some modifications:\n",
    "  - gradient accumulation with 0.5M batch sizes\n",
    "  - std 0.2 initialization\n",
    "  - 1/sqrt(n) residual normalization\n",
    "  - several optimizations:\n",
    "    - DDP\n",
    "    - weight sharing between token embedding and lm_head\n",
    "    - enabled tf32 types\n",
    "    - autocast mixed precision\n",
    "    - increase vocab_size to a power of 2\n",
    "    - adamw with weight decay\n",
    "    - gradient clipping\n",
    "    - torch.compile\n",
    "- since there were so many changes, created an empty commit of `cc0a0c606d6c8de9a7cb4c0e7751d1d38c318563` which marks where I built things on my own vs. started using the course material\n",
    "- GPU throughput has increased massively\n",
    "- overnight there is a val loss of `3.34238338470459` after 15k steps (note that a step here is over a batch of 0.5M)\n",
    "\n",
    "![GPU 100](./training-log/gpu-utilization-100.png \"GPU utilization is now 100\")\n",
    "\n",
    "Here are some outputs from the new model (with slightly better val loss):\n",
    "\n",
    "- Hello, I'm a language model, eyey. We're in arts too; I use poetic translation above (there are several of these, but three of them never appear nearly any further than the rest); most of the shots on my back do no more than what was once sleep-the task of sentiment. I have written two take-shots of hobbling my fingers very deliberately into my thoughts, the two of which have varied features, and where my true theories differ: and what I will argue is minimal because those are both concept\n",
    "- Hello, I'm a language model, so I've built anything to be used at minimal rates. It's a very nice boot package, but with this service I can keep everyone up-to-date up-to-date and still bring people down-to-date when doing anything.\n",
    "\n",
    "  In 2015 -\n",
    "\n",
    "  Given an 8,000,000 LI requests (75,000 Y taxes and 10,000 terms) -in total, I have to be able to write without any problems.\n",
    "\n",
    "  Last week I\n",
    "- Hello, I'm a language model, but I'm speaking with a Seattle-based open source programming design company and really enjoy this gender. It just fit with my criteria of being an active project leader.\n",
    "\n",
    "  Gay people should not be asked the question, but rather be led to believe them so eloquently that it will change before the industry starts. I'll add more people to the process before I invades the term \"hostile\" again.\n",
    "\n",
    "  The question is: what kind of a sociable working relationship are we all\n",
    "- Hello, I'm a language model, but this world is formally a posterior state of the Highway. You'll see how my life happens, as long as it gets back to the way plain-world languages like worst learners come online. Often, I describe my landscape in terms of the usual “in- or out”-tier types of people at least partially because that's important to me if you're posting pictures of material — I just call it it enlightenment, while shunning other stuff just to set up a foundation for creating\n",
    "\n",
    "It does appear the model is starting to take on an understanding of narrative continuity. I really have no idea, besides the val loss, why this is happening. Is it because of the weight sharing? The std and 1/sqrt(n) normalization? It's not any of the perf stuff, so we can rule that out. I will never know because I'm not spending $100's to find out."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- added `checkpoint/chckpt-31k.pt` which has a `val loss: 3.272`\n",
    "- Left lightning studio running overnight and ran out of credits. I feel like, if I train for the full 600k cycles, I'll get something pretty good. I would like to see the project to the end and get a lower loss, but I really don't want to spend hundreds of dollars.\n",
    "- Because of that, this is the end folks 👋🏾"
   ]
  }
 ],
 "metadata": {
  "language_info": {
   "name": "python"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}