Chapter 13 - Text Summarization.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "lF2YIAKhKsd3"
   },
   "source": [
    "# Text Summarization via Deep Reinforcement Learning\n",
    "\n",
    "In this case study, we will apply the deep reinforcement learning concepts of this chapter to the task of text summarization. We will use the Cornell NewsRoom Sum- marization dataset. The goal here is to show readers how we can use deep reinforce- ment learning algorithms to train an agent that can learn to generate summaries of these articles. For the case study, we will focus on deep policy gradient and double deep Q-network agents. We will use the following packages in this case study:\n",
    "\n",
    "* **TensorFlow** is an open-source software library for dataflow programming across a range of tasks. It is a symbolic math library, and is also used for machine learning applications such as neural networks. It is used for both research and production at Google.\n",
    "* **RLSeq2Seq** is an open-source library which implements various RL techniques for text summarization using sequence-to-sequence models https://github.com/yaserkl/RLSeq2Seq.\n",
    "* **pyrouge** is a python interface to the perl-based ROUGE-1.5.5 package that computes ROUGE scores of text summaries https://github.com/andersjo/pyrouge.\n",
    "\n",
    "To measure the performance of machine generated summaries, we will use ROUGE, which stands for Recall-Oriented Understudy for Gisting Evaluation. It is a set of metrics used to evaluate automatic summarization of texts as well as machine trans- lation. It works by comparing an automatically produced summary or translation against a set of reference summaries (typically human-produced).\n",
    "\n",
    "ROUGE-N, ROUGE-S, and ROUGE-L are measures of the granularity of texts when comparing between the system predicted summaries and reference summaries. For example, ROUGE-1 refers to overlap of unigrams between the system summary and reference summary. ROUGE-2 refers to the overlap of bigrams between the system and reference summaries. Let’s take the example from above. Let us say we want to compute the ROUGE-2 precision and recall scores. For ROUGE, recall is a measure of how much of the reference summary is the captured by the system summary."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "2v6vXrdgL2Rx"
   },
   "source": [
    "## Cornell Newsroom Dataset\n",
    "\n",
    "The Cornell Newsroom dataset consists of 1.3 million articles and summaries writ- ten by news authors and editors from 38 major publications between 1998 and 2017. The dataset is split into train, dev, and test sets of 1.1 m, 100 k, and 100 k samples.\n",
    "\n",
    "For our case study, we will use subsets of 10,000/1000/1000 articles and sum- maries from the Cornell Newsroom dataset for our training, validation, and test sets, respectively. We will tokenize and map these data sets using 100-dim embeddings generated with word2vec. For memory considerations, we limit our vocabulary to 50,000 words.\n",
    "\n",
    "A sample story and summary are below:\n",
    "\n",
    ">**Story:** Coinciding with Mary Shelley’s birthday week, this Scott family affair produced by Ridley for director son Luke is another runout for the old story about scientists who cre- ate new life only to see it lurch bloodily away from them. Frosty risk assessor Kate Mara’s investigations into the mishandling of the eponymous hybrid intelligence (The Witch’s still- eerie Anya Taylor-Joy) permits Scott Jr a good hour of existential unease: is it the placid Morgan or her intemperate human overseers (Toby Jones, Michelle Yeoh, Paul Giamatti) who pose the greater threat to this shadowy corporation’s safe operation? Alas, once that question is resolved, the film turns into a passably schlocky runaround, bound for a guess- able last-minute twist that has an obvious precedent in the Scott canon. The capable cast yank us through the chicanery, making welcome gestures towards a number of science- fiction ideas, but cranked-up Frankenstein isn’t one of the film’s smarter or more original ones.\n",
    "\n",
    "> **Summary:** Ridley and son Luke turn in a passable sci-fi thriller, but the horror turns to shlock as the film heads for a predictable twist ending.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "qd9AyNTrZG4c"
   },
   "source": [
    "## Seq2Seq Model\n",
    "\n",
    "Our first task is to train a deep policy gradient agent that can produce summaries of the articles. Before we do so, we pre-train the seq2seq model using maximum likelihood loss, an encoder and decoder layer size of 256, batch size of 20, and adagrad with gradient clipping for 10 epochs (5000 iterations). (*NOTE this will take a long time to train.*)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "HROay83LhsXp",
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "!python RLSeq2Seq/src/run_summarization.py --mode=train \\\n",
    "                                           --data_path=data/processed_train.bin \\\n",
    "                                           --vocab_path=data/vocab-50k \\\n",
    "                                           --log_root=. \\\n",
    "                                           --exp_name=seq2seq_pg \\\n",
    "                                           --batch_size=20 \\\n",
    "                                           --max_iter=5000 \\\n",
    "                                           --use_temporal_attention=True \\\n",
    "                                           --emb_dim=100"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "q-vqOfkuh3-J"
   },
   "source": [
    "Let's calculate ROUGE scores:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "KFzyJhurhzEY",
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "!python RLSeq2Seq/src/run_summarization.py  --mode=decode \\\n",
    "                                            --data_path=data/processed_test.bin \\\n",
    "                                            --vocab_path=data/vocab-50k \\\n",
    "                                            --log_root=. \\\n",
    "                                            --exp_name=seq2seq_pg \\\n",
    "                                            --batch_size=20 \\\n",
    "                                            --emb_dim=100 \\\n",
    "                                            --use_temporal_attention=True \\\n",
    "                                            --single_pass=True"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Policy Gradient\n",
    "\n",
    "Let’s apply a deep policy gradient algorithm to improve our summaries. We switch from MLE loss to RL loss:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "!python RLSeq2Seq/src/run_summarization.py  --mode=train \\\n",
    "                                            --data_path=data/processed_train.bin  \\\n",
    "                                            --vocab_path=data/vocab-50k \\\n",
    "                                            --log_root=. \\\n",
    "                                            --exp_name=seq2seq_pg \\\n",
    "                                            --batch_size=20 \\\n",
    "                                            --emb_dim=100 \\\n",
    "                                            --use_temporal_attention=True \\\n",
    "                                            --eta=2.5E-05 \\\n",
    "                                            --rl_training=True \\\n",
    "                                            --convert_to_reinforce_model=True"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We then continue training for 8 epochs (4000 iterations):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!python RLSeq2Seq/src/run_summarization.py  --mode=train \\\n",
    "                                            --data_path=data/processed_train.bin  \\\n",
    "                                            --vocab_path=data/vocab-50k \\\n",
    "                                            --log_root=. \\\n",
    "                                            --exp_name=seq2seq_pg \\\n",
    "                                            --batch_size=20 \\\n",
    "                                            --emb_dim=100 \\\n",
    "                                            --max_iter=9000 \\\n",
    "                                            --use_temporal_attention=True \\\n",
    "                                            --eta=2.5E-05 \\\n",
    "                                            --rl_training=True \\"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let us evaluate the RL-trained model on the test data:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "!python RLSeq2Seq/src/run_summarization.py  --mode=decode \\\n",
    "                                            --data_path=data/processed_test.bin \\\n",
    "                                            --vocab_path=data/vocab-50k \\\n",
    "                                            --log_root=. \\\n",
    "                                            --exp_name=seq2seq_pg \\\n",
    "                                            --emb_dim=100 \\\n",
    "                                            --rl_training=True \\\n",
    "                                            --use_temporal_attention=True \\\n",
    "                                            --single_pass=1 \\\n",
    "                                            --beam_size=4 \\\n",
    "                                            --decode_after=0"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## DDQN\n",
    "\n",
    "Let’s see if we can improve on the results above using a double deep Q-learning agent. We start as before by pre-training the seq2seq language model using maximum likelihood loss for 10 epochs:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "!python RLSeq2Seq/src/run_summarization.py --mode=train \\\n",
    "                                           --data_path=data/processed_train.bin \\\n",
    "                                           --vocab_path=data/vocab-50k \\\n",
    "                                           --log_root=. \\\n",
    "                                           --exp_name=ddqn \\\n",
    "                                           --batch_size=20 \\\n",
    "                                           --max_iter=5000 \\\n",
    "                                           --emb_dim=100 \\\n",
    "                                           --use_temporal_attention=True"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We switch from MLE to RL loss:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "!python RLSeq2Seq/src/run_summarization.py --mode=train \\\n",
    "                                           --data_path=data/processed_train.bin \\\n",
    "                                           --vocab_path=data/vocab-50k \\\n",
    "                                           --log_root=. \\\n",
    "                                           --exp_name=ddqn \\\n",
    "                                           --batch_size=20 \\\n",
    "                                           --emb_dim=100 \\\n",
    "                                           --ac_training=True \\\n",
    "                                           --dueling_net=True \\\n",
    "                                           --dqn_target_update=500 \\\n",
    "                                           --convert_to_reinforce_model=True  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will first pre-train the DDQN with a fixed actor model:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "!python RLSeq2Seq/src/run_summarization.py --mode=train \\\n",
    "                                           --data_path=data/processed_train.bin \\\n",
    "                                           --vocab_path=data/vocab-50k \\\n",
    "                                           --log_root=. \\\n",
    "                                           --exp_name=ddqn \\\n",
    "                                           --batch_size=20 \\\n",
    "                                           --emb_dim=100 \\\n",
    "                                           --dqn_replay_buffer_size=5000 \\\n",
    "                                           --dqn_target_update=500 \\\n",
    "                                           --ac_training=True \\\n",
    "                                           --dqn_pretrain=True \\\n",
    "                                           --dueling_net=True \\\n",
    "                                           --dqn_pretrain_steps=500"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Then we will train the DDQN for 8 epochs using a batch size of 20, replay buffer of 5000 samples and updating the target network every 500 iterations. We first start training with true Q-estimates:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "!python RLSeq2Seq/src/run_summarization.py --mode=train \\\n",
    "                                           --data_path=data/processed_train.bin \\\n",
    "                                           --vocab_path=data/vocab-50k \\\n",
    "                                           --log_root=. \\\n",
    "                                           --exp_name=ddqn \\\n",
    "                                           --batch_size=20 \\\n",
    "                                           --max_iter=5500 \\\n",
    "                                           --emb_dim=100 \\\n",
    "                                           --dqn_replay_buffer_size=5000 \\\n",
    "                                           --dqn_target_update=500 \\\n",
    "                                           --ac_training=True \\\n",
    "                                           --dueling_net=True \\\n",
    "                                           --calculate_true_q=True "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We then switch to Q-estimates after the warm start:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "!python RLSeq2Seq/src/run_summarization.py --mode=train \\\n",
    "                                           --data_path=data/processed_train.bin \\\n",
    "                                           --vocab_path=data/vocab-50k \\\n",
    "                                           --log_root=. \\\n",
    "                                           --exp_name=ddqn \\\n",
    "                                           --batch_size=20 \\\n",
    "                                           --max_iter=9000 \\\n",
    "                                           --emb_dim=100 \\\n",
    "                                           --dqn_replay_buffer_size=5000 \\\n",
    "                                           --dqn_target_update=500 \\\n",
    "                                           --ac_training=True \\\n",
    "                                           --dueling_net=True \\\n",
    "                                           --calculate_true_q=False "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finally, we calculate ROUGE scores:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "!python RLSeq2Seq/src/run_summarization.py  --mode=decode \\\n",
    "                                            --data_path=data/processed_test.bin \\\n",
    "                                            --vocab_path=data/vocab-50k \\\n",
    "                                            --log_root=. \\\n",
    "                                            --exp_name=ddqn \\\n",
    "                                            --emb_dim=100 \\\n",
    "                                            --ac_training=True \\\n",
    "                                            --dueling_net=True \\\n",
    "                                            --dqn_replay_buffer_size=5000 \\\n",
    "                                            --dqn_target_update=500 \\\n",
    "                                            --single_pass=1 \\\n",
    "                                            --beam_size=4"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The DDQN agent outperforms the policy gradient agent for the chosen parameters. There are a myriad of possibilities to improve results further—we could use scheduled or prioritized sampling, intermediate rewards, and attention at the encoder or decoder."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "accelerator": "GPU",
  "colab": {
   "name": "Chapter 13: Text Summarization.ipynb",
   "provenance": [],
   "version": "0.3.2"
  },
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.16"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}