dota2_hero_lore_connectivity.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Dota 2 Hero Lore Connnectivity\n",
    "---\n",
    "#### Data:\n",
    "Dota 2 hero lores including associated heroes.\n",
    "#### Source: \n",
    "Gamepedia [Dota 2 Heroes](https://dota2.gamepedia.com/Dota_2_Wiki) and respective [hero lores](https://dota2.gamepedia.com/Category:Hero_lore).\n",
    "#### Method[<sup>PS</sup>](#Postscript-down-low):\n",
    "[MediaWiki API](https://help.gamepedia.com/Bots) for Gamepedia.\n",
    "\n",
    "See [here](https://help.gamepedia.com/Logging_in_to_third-party_tools#Using_Special:BotPasswords) for API registration and usage. Gamepedia also offers [Cargo data](https://help.gamepedia.com/Extension:Cargo) access. Dota-specific Cargo tables are found [here](https://dota2.gamepedia.com/Special:CargoTables).\n",
    "\n",
    "#### Visualization:\n",
    "A [variation on a radial dendogram](https://beta.observablehq.com/@youmikoh/dota-2-lore-connectivity) using [d3v5.js](https://d3js.org/).\n",
    "\n",
    "$G = \\{V, E\\} : $ `heroes` $\\mapsto V, $ `connections_set` $\\mapsto E$ \n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Postscript down-low\n",
    "Although Gamepedia's infrastructure allows for wiki contributors and users to get pretty fancy (see above links for details), it isn't really warranted for these ad-hoc micro projects. Since read-only data can be easily compiled using barebone MediaWiki API requests and [Cargo table queries](https://dota2.gamepedia.com/api.php?action=help&modules=cargoquery), skip on the extensions and third-party tools.\n",
    "\n",
    "For a more thourough example, see similar here: [Dota 2 Pro Circuit Connectivity](http://nbviewer.jupyter.org/github/youmikoh/dpc-connectivity/blob/master/part1_collect.ipynb).\n",
    "\n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import re\n",
    "import requests\n",
    "import requests_cache\n",
    "from urllib.parse import urlencode\n",
    "\n",
    "import time\n",
    "from datetime import datetime\n",
    "\n",
    "import wikitextparser as wtp"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "requests_cache.install_cache(\n",
    "    cache_name='dota2_hero_lore',\n",
    "    backend='sqlite',\n",
    "    expire_after=60*60*24\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "headers = {\n",
    "    'User-Agent': 'Dota 2 Hero Lore Connectivity 1.0/youmikoh@github',\n",
    "    'Accept-Encoding': 'gzip'\n",
    "}\n",
    "auth = ('playumiko@bot', 'mvk41h8nj3a4hbovohs146utr48kc996')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "def throttle(f):\n",
    "    def wrap(*args, **kwds):\n",
    "        now = datetime.now()\n",
    "        if (now - wrap.last).seconds < 2: \n",
    "            time.sleep(2)\n",
    "        wrap.last = now\n",
    "        return f(*args, **kwds)\n",
    "    wrap.last = datetime.now()\n",
    "    return wrap"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "single = lambda c: len(c)==1\n",
    "\n",
    "@throttle\n",
    "def gamepedia_api(query):\n",
    "    api = f'https://dota2.gamepedia.com/api.php?{urlencode(query)}'\n",
    "    response = requests.get(api, headers=headers, auth=auth)\n",
    "    return response.json()\n",
    "\n",
    "def gamepedia_content(query):\n",
    "    k, v = query\n",
    "    query = {'action': 'query', 'prop': 'revisions', 'rvprop': 'content', 'format': 'json'}\n",
    "    query[k] = v\n",
    "    \n",
    "    data = gamepedia_api(query)\n",
    "    pages = list(data['query']['pages'].values())\n",
    "    \n",
    "    source = lambda page: page['revisions'].pop().get('*')\n",
    "    content = [source(p) for p in pages]\n",
    "    \n",
    "    if single(content): content = content.pop()\n",
    "    return content\n",
    "    \n",
    "def gamepedia_cargo_content(query):\n",
    "    query['action'] = 'cargoquery'\n",
    "    query['format'] = 'json'\n",
    "    data = gamepedia_api(query)\n",
    "    return data.get('cargoquery')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "heroes_query = {'tables': 'heroes', 'fields': 'title,primary_attribute,page_id', 'where': 'game is null', 'limit': 200}\n",
    "heroes = gamepedia_cargo_content(heroes_query)\n",
    "\n",
    "hero = lambda h: h['title']\n",
    "heroes = {hero(h)['title']:hero(h) for h in heroes}\n",
    "heroes_set = set(heroes.keys()) "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "heroes_bio_query = {'tables': 'heroes_bio', 'fields': 'name,title,quote,lore', 'limit': 200}\n",
    "heroes_bio = gamepedia_cargo_content(heroes_bio_query)\n",
    "\n",
    "key = lambda h: h['title'] +' '+ h['name']\n",
    "heroes_bio = {key(h['title']):h['title'] for h in heroes_bio}\n",
    "\n",
    "for bio in heroes_bio.values(): bio['quote'] = re.sub('&quot;', '', bio['quote'])\n",
    "\n",
    "finicky = set()\n",
    "\n",
    "for key, bio in heroes_bio.items():\n",
    "    similar = lambda key: [h for h in heroes_set if h in key]\n",
    "    k = similar(key)\n",
    "    if len(k)==1:\n",
    "        k = k.pop()\n",
    "        heroes[k]['lore'] = bio.pop('lore')\n",
    "        heroes[k]['quote'] = bio.pop('quote')\n",
    "    else: finicky.add(key)\n",
    "\n",
    "for f in finicky:\n",
    "    remaining_without_bio = {k for k in heroes_set if not heroes[k].get('lore')}\n",
    "    keys = f.split(' ')\n",
    "    similar_remaining = lambda key: [h for h in remaining_without_bio if key in h] \n",
    "    for key in keys:\n",
    "        k = similar_remaining(key)\n",
    "        if len(k)==1:\n",
    "            k = k.pop()\n",
    "            heroes[k]['lore'] = heroes_bio[f].pop('lore')\n",
    "            heroes[k]['quote'] = heroes_bio[f].pop('quote')\n",
    "\n",
    "not {k for k in heroes_set if not heroes[k].get('lore')}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "def hero_factions(faction_page_id):\n",
    "    faction = gamepedia_content(('pageids', faction_page_id))\n",
    "    parsed = wtp.parse(faction)\n",
    "    affiliated = [p for p in parsed.sections if 'Affiliated' in p][-1]\n",
    "    return re.findall(r'{{H\\|(.*?)}}', affiliated.string)\n",
    "\n",
    "dire_heroes = hero_factions(127924)\n",
    "radiant_heroes = hero_factions(137875)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "def hero_connections(hero):\n",
    "    lore = gamepedia_content(('titles', f'{hero}/Lore'))\n",
    "    parsed = wtp.parse(lore)\n",
    "    parsed = [p for p in parsed.templates if 'Lore infobox' in p].pop()\n",
    "    parsed = [p for p in parsed.arguments if 'heroes' in p].pop()\n",
    "    return set(re.findall(r'{{H.?\\|(.*?)}}', parsed.string))\n",
    "\n",
    "connections_list = []\n",
    "\n",
    "for hero in heroes_set:\n",
    "    connected = hero_connections(hero)\n",
    "    heroes[hero]['count'] = len(connected)\n",
    "    heroes[hero]['connections'] = connected & heroes_set\n",
    "    connections_list += [frozenset([hero, c]) for c in connected]\n",
    "    heroes[hero]['faction'] = 'radiant' if hero in radiant_heroes else 'dire'\n",
    "    \n",
    "connections_set = {c for c in connections_list}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "graph = {'nodes':heroes, 'links':connections_set}\n",
    "\n",
    "import json\n",
    "from random import shuffle\n",
    "\n",
    "class DPCEncoder(json.JSONEncoder):\n",
    "    def default(self, obj):\n",
    "        if isinstance(obj, set): \n",
    "            return list(obj)\n",
    "        if isinstance(obj, frozenset): \n",
    "            listset = list(obj)\n",
    "            shuffle(listset)\n",
    "            return listset\n",
    "        elif isinstance(obj, datetime): \n",
    "            return obj.isoformat()\n",
    "        else:\n",
    "            return super(DPCEncoder, self).default(obj)\n",
    "        \n",
    "with open('data/hero_lore_connectivity_data.json', 'w') as outfile:  \n",
    "    json.dump(graph, outfile, cls=DPCEncoder)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "## Continue to [Visualization](https://beta.observablehq.com/@youmikoh/dota-2-lore-connectivity)\n",
    "___\n",
    "<br><br>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<style>\n",
       "@import url(http://fonts.googleapis.com/css?family=Source+Code+Pro|Open+Sans|Montserrat:300,500);\n",
       "\n",
       "#notebook-container {\n",
       "-webkit-box-shadow: none;\n",
       "box-shadow: none;\n",
       "}\n",
       "\n",
       ".rendered_html h1 { font-size: 3rem !important; }\n",
       ".rendered_html h2 { font-size: 2rem !important; }\n",
       ".rendered_html h3 { font-size: 1.8rem !important; }\n",
       ".rendered_html h4 { font-size: 1.5rem !important; }\n",
       ".rendered_html h5 { font-size: 1.5rem !important; }\n",
       ".rendered_html h6 { font-size: 1.5rem !important; }\n",
       "\n",
       ".rendered_html h1,\n",
       ".rendered_html h2,\n",
       ".rendered_html h3,\n",
       ".rendered_html h4,\n",
       ".rendered_html h5,\n",
       ".rendered_html h6 {\n",
       "font-family: 'Montserrat', sans-serif !important;\n",
       "line-height: 0.5em !important;\n",
       "}\n",
       "\n",
       ".rendered_html p,\n",
       ".rendered_html li,\n",
       ".rendered_html blockquote,\n",
       ".rendered_html summary {\n",
       "font-family: 'Open+Sans', sans-serif !important;\n",
       "line-height: 20px !important;\n",
       "font-weight: 300 !important;\n",
       "}\n",
       "\n",
       ".rendered_html summary { margin-top: 1.3em !important; }\n",
       "\n",
       "pre {\n",
       "font-family: 'Source Code Pro', sans-serif !important;\n",
       "font-size: 0.9em !important;\n",
       "}\n",
       "\n",
       "div.input_area {\n",
       "border: none !important;\n",
       "background: whitesmoke !important;\n",
       "}\n",
       "\n",
       "</style>\n"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from IPython.core.display import HTML\n",
    "HTML(open(\"css/lore_ipynb.css\", \"r\").read()) #IPYNB STYLING"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}