docs/content/web_navigation/WebNavigation-Difficulty-01-v0.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Web Navigation\n",
    "\n",
    "This environment is included in A2Perf.\n",
    "\n",
    "![The Ariane RISC-V CPU](../../../media/gminiwob_scene.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Description\n",
    "The web navigation environment aims at enabling the development of compositional tasks that can be represented by a dependency graph. Using Compositional Design of Environments (CoDE), propesed by Google Research in \n",
    "['Environment Generation for Zero-Shot Compositional Reinforcement Learning'](https://openreview.net/pdf?id=CeByDMy0YTL \"Environment Generation for Zero-Shot Compositional Reinforcement Learning\"), websites are generated automatically, after which the policy has to complete the proposed webpages.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Action Space\n",
    "The action should be passed as a scalar. Two types of actions are possible. Firstly, abstract navigation allows to directly refer to an element, and the profile is irrelevant. In this case, the action is converted to a tuple. If abstract navigation is desired, we have to pass `use_conceptual=True` when initializing the environment. Secondly, the action can refer to a pair of elements and profile fields. The agent will then enter the value of the profile key corresponding to the selected DOM element.\n",
    "\n",
    "For example, with abstract navigation, `action=5` refers to the 5-th\n",
    "element in the DOM tree where the tree is linearized using the\n",
    "`get_dom_elements` function. Without abstract navigation, '5' refers to\n",
    "both profile and element indices, i.e., (element_index, profile_index)\n",
    "where `action=profile_index*number_of_dom_elements+element_index`."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Observation Space\n",
    "When performing one step in the environment, we are given the option to return the raw state of wrapped state. To return the raw state, we pass `raw_state=True`, by default, the observation is wrapped. \\\n",
    "The wrapped structure will return a dictionary with the following keys: `profile_key, profile_value, profile_key_mask, profile_value_mask, dom_elements, dom_profile_joint_mask, time_step, dom_attribute_mask, dom_profile_intersection, dom_profile_intersection_mask, dom_features`, where the values are arrays.\n",
    "- `profile_key`, `profile_value`, `profile_key_mask`, `profile_value_mask`: \\\n",
    "The profile arrays are 2D arrays with the shape (number of fields, sequence length), dom arrays have the shape. The first keys of the wrapped observation relate to the profile, while the last few relate to the DOM tree. The profile of the webpage is the user profile, which contains keys and values which need to be filled in. When booking flights, this could be `{\"Departure Date\": \"Friday\", \"Destination Airport\": \"Los Angeles (LAX)\"}`. These keys and values are subsequently embedded into vectors. All keys and values are embedded in fixed length vectors, which raises the need for padding the shorter embeddings. This is where the masks comes in, the mask contains ones where the embedding relates to the original key or value, and zeros where the embedding is padded. \n",
    "- `dom_elements`, `dom_elements_mask`, `dom_attribute_mask`, `dom_features`: \\\n",
    "Next the webpage is returned as DOM elements, again embedded. Examples of DOM elements are eg. `<div>`, `<header>` etc. \n",
    "- `dom_profile_intersection`, `dom_profile_intersection_mask`: \\\n",
    "Next, the intersection between the profile and DOM elements is embedded and returned. For each profile field key and value tokens (such as `[\"first\", \"name\"]`) and for each element attribute tokens (such as `[\"initial\", \"name\", \":\"]`), the overlapping tokens are embedded. The intersection is a 5D tensor of shape `(number of elements, max number of attributes, number of profile fields, number of action types (2), max sequence length)`.\n",
    "- `time_step`: \\\n",
    "The timestep is calculated as the number of steps taken, divided by the maximum number of steps allowed.\n",
    "\n",
    "\\\n",
    "When `raw_state=True`, the raw state is returned as a `MiniWoBState` object. This object stores the raw website information, which can be accessed with the following attributes:\n",
    "\n",
    "- `obs.utterance`: returns the task utterance, a dictionary providing the profile key and value pairs.\n",
    "- `obs.phrase`: returns the Phrase object of the utterance.\n",
    "- `obs.tokens`: returns the tokens used for the encoding of the utterance.\n",
    "- `obs.fields`: returns the key-value pairs extracted from the utterance.\n",
    "- `obs.dom`: returns the root DOM structure.\n",
    "- `obs.dom_elements`: returns a flattened lsit of all DOM elements "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Rewards\n",
    "At every timestep, the agent receives a small penalty to encourage efficient navigation. Next, a potential-based positive reward is given to the agent after successfully linking a key to a field and entering the correct value. Lastly, the agent receives a task success reward (1.0 or -1.0) when the final state is reached or a time out is encountered. \n",
    "\n",
    "To give an example, consider booking flights where the agent has the following profile:\n",
    "`{\"Departure Date\": \"Friday\", \"Destination Airport\": \"Los Angeles (LAX)\"}`, which has two fields (Departure Date and Destination Airport). The agent starts by picking a field, and tries to find the correpsonding text box in the page. The value corresponding to the field (eg. Destination Airport), is subsequently typed into the text box. If this is correct, the agent will receive a positive reward of 1/2, where the denominator, 2, is the number of fields in the profile."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Starting State\n",
    "Upon resetting the website, all episode fields are emptied and a clean webpage is returned."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Episode End\n",
    "The episode can be ended under multiple conditions.\n",
    "Firstly, if the number of steps is greater then the maximum allowed number of steps, the environment is terminated. The maximum number of steps can be defined when initializing the environment, by default, 6 steps is the limit."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Arguments\n",
    "When creating the web navigation environment, there is parameters we have to define and optional parameters. Firstly, we have to define the number of websites that needs to be created, with the `num_websites` parameter. Next, we have to either specify a difficulty for the websites or we can provide a design. When creating the environment we thus either specify `difficulty` or `designs`.\n",
    "```python\n",
    "import gymnasium as gym\n",
    "import a2perf.domains.web_navigation\n",
    "\n",
    "env = gym.make('WebNavigation-v0', num_websites=1, difficutly=1, ...)\n",
    "\n",
    "```\n",
    "#### Required parameters:\n",
    "\n",
    "| Parameter          | Type  | Default | Description|\n",
    "|--------------------|------|---|---|\n",
    "| `num_websites` | int | `None` | The number of websites to be created. |\n",
    "| `difficulty` | Optional, int | `1` | Defines the difficulty of the webpage(s) that are being created. A random agent has a >=50% chance of completing a level 1 website, a >=25% chance of completing a level 2 website and a >=10% chance of completing a level 3 website. You either define `difficulty` or `designs`, not both.|\n",
    "| `designs` | Optional, list[dict[str, Any]] | `None` | You can pass the design for a number of websites, where each website corresponds to one dictionary in the list. If you specify designs, note that you need to specify at least the number of websites defined with `num_websites`. The designs returned is then `num_websites` randomly sampled from `designs`. You either define `difficulty` or `designs`, not both.|\n",
    "\n",
    "#### Optional parameters:\n",
    "\n",
    "| Parameter          | Type  | Default | Description|\n",
    "|--------------------|------|---|---|\n",
    "| `seed` | int | `0` | Defines the seed for the random number generator and the `reset` method.|\n",
    "| `data_dir` | str | `\"a2perf/domains/web_navigation/environment_generation/data\"` | Path to the directory which contains a zipfile with json files describing difficulty levels for webpages.|\n",
    "| `global_vocabulary` | Vocabulary | `vocabulary_node.LockedThreadedVocabulary()`| The global_vocabulary gathers all characters and corresponding tokens. Is used to create embeddings for profile dictionaries and DOM dictionaries. |\n",
    "| `use_legacy_reset` | bool | `False` | If `True`, the `reset` method returns only the observation. If `False` both the observation and info are returned.|\n",
    "| `use_legacy_step` | bool | `False` | If `True`, the `step` method returns the observation, reward, (terminated or truncated), info. If `False`, both terminated and truncated are returned.|\n",
    "| `step_limit` | int | `25` | Defines the maximum number of steps that can be taken by the environment. |\n",
    "| `render_mode` | str | `image` | Possible render modes are `test`, which saves screenshots of the website in the screenshots attribute, `rgb_array` which returns a 3D array, and `image`, which returns a screenshot of the webstie.|\n",
    "| `raw_state` | bool | `False` | If `True`, the raw observation is returned, else the observation is wrapped, for more info, see the Observation section of this notebook.|\n",
    "| `use_conceptual` | bool | `False` | If true, the action spac expects abstract navigation, else an action refers to a pair of elements and profile fields. |\n",
    "| `**kwargs` | dict | `None` | |"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Reset you can pass kwargs in the options argument, not possible in step though"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Version History\n",
    "- v0: Initial versions release"
   ]
  }
 ],
 "metadata": {
  "language_info": {
   "name": "python"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}