labs/final_project/TextWorld.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "C5w08Bz0m2Ra"
   },
   "source": [
    "# TextWorld 101\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "kT9ZurVxn3oa"
   },
   "source": [
    "## Environment setup\n",
    "\n",
    "* TextWorld (I used my branch instead of microsoft upstream version [here](https://github.com/Microsoft/TextWorld) to freeze TextWorld, as it is being actively developed)\n",
    "* pytorch\n",
    "* ptan\n",
    "\n",
    "Note: you should not restart the runtime after the next cell (ignore if it will ask you to do so), but you **have to** restart the runtime after the following cell, otherwise textworld won't work.\n",
    "\n",
    "Those cells need to be run only once after the runtime reset or fresh load."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "collapsed": true,
    "id": "2m-ULnq5DxLO"
   },
   "outputs": [],
   "source": [
    "!pip install git+https://github.com/Shmuma/TextWorld > /dev/null 2>&1\n",
    "!pip install ptan > /dev/null 2>&1\n",
    "!pip3 install torch torchvision > /dev/null 2>&1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 276
    },
    "colab_type": "code",
    "collapsed": false,
    "id": "5Tsi-vvKD9hu",
    "outputId": "bec33154-c2b8-40bf-f5e2-827ae7891ec3"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Collecting prompt-toolkit==1.0.16\n",
      "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/57/a8/a151b6c61718eabe6b4672b6aa760b734989316d62ec1ba4996765e602d4/prompt_toolkit-1.0.16-py3-none-any.whl (244kB)\n",
      "\u001b[K     |████████████████████████████████| 245kB 6.8MB/s \n",
      "\u001b[?25hRequirement already satisfied: six>=1.9.0 in /usr/local/lib/python3.6/dist-packages (from prompt-toolkit==1.0.16) (1.12.0)\n",
      "Requirement already satisfied: wcwidth in /usr/local/lib/python3.6/dist-packages (from prompt-toolkit==1.0.16) (0.1.7)\n",
      "\u001b[31mERROR: textworld 1.1.1 has requirement prompt-toolkit<2.1.0,>=2.0.0, but you'll have prompt-toolkit 1.0.16 which is incompatible.\u001b[0m\n",
      "\u001b[31mERROR: jupyter-console 6.0.0 has requirement prompt-toolkit<2.1.0,>=2.0.0, but you'll have prompt-toolkit 1.0.16 which is incompatible.\u001b[0m\n",
      "Installing collected packages: prompt-toolkit\n",
      "Successfully installed prompt-toolkit-1.0.16\n"
     ]
    },
    {
     "data": {
      "application/vnd.colab-display-data+json": {
       "pip_warning": {
        "packages": [
         "prompt_toolkit"
        ]
       }
      }
     },
     "metadata": {
      "tags": []
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Juggling around colab requirement\n",
    "!pip uninstall -y prompt-toolkit > /dev/null 2>&1\n",
    "!pip install prompt-toolkit==1.0.16\n",
    "# DO NOT FORGET to restart runtime as suggested"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "NTWnBRj_tpd3"
   },
   "source": [
    "## Imports"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "collapsed": true,
    "id": "1YDwj_PTtxP3"
   },
   "outputs": [],
   "source": [
    "import gym\n",
    "import textworld.gym as tw_gym"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "GLvzoUN8VLRM"
   },
   "source": [
    "## Build games we'll play with\n",
    "\n",
    "TextWorld allows to generate games of specific complexity, which makes it great for experimentation, learning and reasearch, as we can gradually increase complexity of problems.\n",
    "\n",
    "Below we generate the problem with 10 objects, 5 rooms and quest length 5"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 136
    },
    "colab_type": "code",
    "collapsed": false,
    "id": "iw_vwZ_gVe9J",
    "outputId": "ccdd360b-4e19-41dd-cf78-01b9e6c5b45a"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Global seed: 1234\n",
      "Game generated: /content/simple1.ulx\n",
      "total 660\n",
      "drwxr-xr-x 1 root root   4096 May 31 16:17 sample_data\n",
      "-rw-r--r-- 1 root root  32067 Jun 19 14:10 simple1.json\n",
      "-rw-r--r-- 1 root root  26000 Jun 19 14:10 simple1.ni\n",
      "-rw-r--r-- 1 root root 608512 Jun 19 14:10 simple1.ulx\n"
     ]
    }
   ],
   "source": [
    "!tw-make custom --world-size 5 --nb-objects 10 --quest-length 5 --seed 1234 --output simple1.ulx\n",
    "!ls -l"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "rYkXHfF6o957"
   },
   "source": [
    "\n",
    "## Env capabilities\n",
    "\n",
    "Explore textworld gym environment\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "collapsed": true,
    "id": "7TKWGF-ppAcY"
   },
   "outputs": [],
   "source": [
    "env_id = tw_gym.register_game(\"simple1.ulx\")\n",
    "env = gym.make(env_id)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 34
    },
    "colab_type": "code",
    "collapsed": false,
    "id": "Ig7o6cMPEGqg",
    "outputId": "cf57f2a3-7503-46d2-a52f-55f29b419e74"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(Word(L=200, V=1249), Word(L=8, V=1249))"
      ]
     },
     "execution_count": 4,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "env.observation_space, env.action_space"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 34
    },
    "colab_type": "code",
    "collapsed": false,
    "id": "4GCyCNXdEhTB",
    "outputId": "0dc68448-d857-4448-9661-567836058d1f"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['<PAD>', '<UNK>', '</S>', '<S>', '<|>', '', '0', '1', '10', '100']"
      ]
     },
     "execution_count": 5,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "env.observation_space.vocab[:10]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 71
    },
    "colab_type": "code",
    "collapsed": false,
    "id": "yqS77YN1ZTeM",
    "outputId": "316ebb44-81b9-4602-840f-500aac7c2a6e"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(\"\\n\\n\\n                    ________  ________  __    __  ________        \\n                   |        \\\\|        \\\\|  \\\\  |  \\\\|        \\\\       \\n                    \\\\$$$$$$$$| $$$$$$$$| $$  | $$ \\\\$$$$$$$$       \\n                      | $$   | $$__     \\\\$$\\\\/  $$   | $$          \\n                      | $$   | $$  \\\\     >$$  $$    | $$          \\n                      | $$   | $$$$$    /  $$$$\\\\    | $$          \\n                      | $$   | $$_____ |  $$ \\\\$$\\\\   | $$          \\n                      | $$   | $$     \\\\| $$  | $$   | $$          \\n                       \\\\$$    \\\\$$$$$$$$ \\\\$$   \\\\$$    \\\\$$          \\n              __       __   ______   _______   __        _______  \\n             |  \\\\  _  |  \\\\ /      \\\\ |       \\\\ |  \\\\      |       \\\\ \\n             | $$ / \\\\ | $$|  $$$$$$\\\\| $$$$$$$\\\\| $$      | $$$$$$$\\\\\\n             | $$/  $\\\\| $$| $$  | $$| $$__| $$| $$      | $$  | $$\\n             | $$  $$$\\\\ $$| $$  | $$| $$    $$| $$      | $$  | $$\\n             | $$ $$\\\\$$\\\\$$| $$  | $$| $$$$$$$\\\\| $$      | $$  | $$\\n             | $$$$  \\\\$$$$| $$__/ $$| $$  | $$| $$_____ | $$__/ $$\\n             | $$$    \\\\$$$ \\\\$$    $$| $$  | $$| $$     \\\\| $$    $$\\n              \\\\$$      \\\\$$  \\\\$$$$$$  \\\\$$   \\\\$$ \\\\$$$$$$$$ \\\\$$$$$$$ \\n\\nWho's got a virtual machine and is about to play through an life changing round of TextWorld? You do! Here is your task for today. First off, if it's not too much trouble, I need you to check that the gate inside the scullery is unlocked with the keycard. And then, open the gate within the scullery. And then, go east. Then, take the cookie from the counter. Then, eat the cookie. That's it!\\n\\n-= Scullery =-\\nYou've just shown up in a scullery. You begin to take stock of what's here.\\n\\nHey, want to see a rack? Look over there, a rack. The rack is ordinary. Unfortunately, there isn't a thing on it.\\n\\nThere is a closed gate leading east. There is a closed door leading south.\\n\\n\\n\\n\",\n",
       " {})"
      ]
     },
     "execution_count": 6,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "r = env.reset()\n",
    "r"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 615
    },
    "colab_type": "code",
    "collapsed": false,
    "id": "BzWIsqg5ZYd6",
    "outputId": "147cc42f-73d2-4266-dfd2-8c8b275aeb27"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "\n",
      "\n",
      "                    ________  ________  __    __  ________        \n",
      "                   |        \\|        \\|  \\  |  \\|        \\       \n",
      "                    \\$$$$$$$$| $$$$$$$$| $$  | $$ \\$$$$$$$$       \n",
      "                      | $$   | $$__     \\$$\\/  $$   | $$          \n",
      "                      | $$   | $$  \\     >$$  $$    | $$          \n",
      "                      | $$   | $$$$$    /  $$$$\\    | $$          \n",
      "                      | $$   | $$_____ |  $$ \\$$\\   | $$          \n",
      "                      | $$   | $$     \\| $$  | $$   | $$          \n",
      "                       \\$$    \\$$$$$$$$ \\$$   \\$$    \\$$          \n",
      "              __       __   ______   _______   __        _______  \n",
      "             |  \\  _  |  \\ /      \\ |       \\ |  \\      |       \\ \n",
      "             | $$ / \\ | $$|  $$$$$$\\| $$$$$$$\\| $$      | $$$$$$$\\\n",
      "             | $$/  $\\| $$| $$  | $$| $$__| $$| $$      | $$  | $$\n",
      "             | $$  $$$\\ $$| $$  | $$| $$    $$| $$      | $$  | $$\n",
      "             | $$ $$\\$$\\$$| $$  | $$| $$$$$$$\\| $$      | $$  | $$\n",
      "             | $$$$  \\$$$$| $$__/ $$| $$  | $$| $$_____ | $$__/ $$\n",
      "             | $$$    \\$$$ \\$$    $$| $$  | $$| $$     \\| $$    $$\n",
      "              \\$$      \\$$  \\$$$$$$  \\$$   \\$$ \\$$$$$$$$ \\$$$$$$$ \n",
      "\n",
      "Who's got a virtual machine and is about to play through an life changing round of TextWorld? You do! Here is your task for today. First off, if it's not too much trouble, I need you to check that the gate inside the scullery is unlocked with the keycard. And then, open the gate within the scullery. And then, go east. Then, take the cookie from the counter. Then, eat the cookie. That's it!\n",
      "\n",
      "-= Scullery =-\n",
      "You've just shown up in a scullery. You begin to take stock of what's here.\n",
      "\n",
      "Hey, want to see a rack? Look over there, a rack. The rack is ordinary. Unfortunately, there isn't a thing on it.\n",
      "\n",
      "There is a closed gate leading east. There is a closed door leading south.\n",
      "\n",
      "\n",
      "\n",
      "\n"
     ]
    }
   ],
   "source": [
    "print(r[0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 34
    },
    "colab_type": "code",
    "collapsed": false,
    "id": "3idrRH3PZml_",
    "outputId": "84dca56c-03e6-4f5c-8d43-b19ffb55bd88"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "({},)"
      ]
     },
     "execution_count": 8,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "r[1:]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "Abpy20zpZrKA"
   },
   "source": [
    "So, TextWorld env is **incompatible** with gym api, as `reset()` returns tuple with observation string and extended info dict. This is not a bug, but a feature -- gym provides no way to get the extended info for initial observation.\n",
    "\n",
    "We need to take this into account on observation preprocessing."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 34
    },
    "colab_type": "code",
    "collapsed": false,
    "id": "Q2jxVnVxayyt",
    "outputId": "31bf6f6a-e85f-46e3-9ea5-422af68d78a7"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "('You have to open the gate first.\\n\\n', 0, False, {})"
      ]
     },
     "execution_count": 9,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "r1 = env.step('go east')\n",
    "r1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 68
    },
    "colab_type": "code",
    "collapsed": false,
    "id": "OolFbdCta6JE",
    "outputId": "c1baf3a9-278b-4a7b-a275-1e14a989fb68"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "You have to open the gate first.\n",
      "\n",
      "\n"
     ]
    }
   ],
   "source": [
    "print(r1[0])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "eGZzRkM4bDKa"
   },
   "source": [
    "## Extended attributes\n",
    "\n",
    "TextWorld can provide us information in a more structured way, giving information about admissible commands, intermediate reward, inventory and even path to solution.\n",
    "\n",
    "To get it, we need to pass specific flags on environment creation.\n",
    "\n",
    "Below we'll ask about specific flags I find useful, full set of them is here: https://github.com/microsoft/TextWorld/blob/04952aa64c4612309a7296aef99867ecfa5c7db1/textworld/envs/wrappers/filter.py\n",
    "\n",
    "In addition, we can limit amount of steps agent can do and redefine the name of gym env which will be registered."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 34
    },
    "colab_type": "code",
    "collapsed": false,
    "id": "nPPknG5ObE93",
    "outputId": "ce1cd0e5-58c8-4903-9024-9a1bf9fe5400"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'tw-simple1-v0'"
      ]
     },
     "execution_count": 11,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from textworld.envs.wrappers.filter import EnvInfos\n",
    "\n",
    "EXTRA_GAME_INFO = {\n",
    "    \"inventory\": True,\n",
    "    \"description\": True,\n",
    "    \"intermediate_reward\": True,\n",
    "    \"admissible_commands\": True,\n",
    "    \"policy_commands\": True,\n",
    "}\n",
    "\n",
    "env_id = tw_gym.register_game(\"simple1.ulx\", max_episode_steps=10,\n",
    "                                  name=\"simple1\", request_infos=EnvInfos(**EXTRA_GAME_INFO))\n",
    "env_id"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "collapsed": true,
    "id": "dwnG3haCdJnK"
   },
   "outputs": [],
   "source": [
    "env = gym.make(env_id)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 394
    },
    "colab_type": "code",
    "collapsed": false,
    "id": "v2sIRq7WdNlL",
    "outputId": "fc1fdece-9016-43fe-c3a3-40a30cc92468"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'admissible_commands': ['drop keycard',\n",
       "  'drop ladle',\n",
       "  'examine door',\n",
       "  'examine gate',\n",
       "  'examine keycard',\n",
       "  'examine ladle',\n",
       "  'examine rack',\n",
       "  'inventory',\n",
       "  'look',\n",
       "  'open door',\n",
       "  'put keycard on rack',\n",
       "  'put ladle on rack',\n",
       "  'unlock gate with keycard'],\n",
       " 'description': \"-= Scullery =-\\nYou've just shown up in a scullery. You begin to take stock of what's here.\\n\\nHey, want to see a rack? Look over there, a rack. The rack is ordinary. Unfortunately, there isn't a thing on it.\\n\\nThere is a closed gate leading east. There is a closed door leading south.\\n\\n\\n\",\n",
       " 'intermediate_reward': 0,\n",
       " 'inventory': 'You are carrying:\\n  a keycard\\n  a ladle\\n\\n\\n',\n",
       " 'policy_commands': ['unlock gate with keycard',\n",
       "  'open gate',\n",
       "  'go east',\n",
       "  'take cookie from counter',\n",
       "  'eat cookie']}"
      ]
     },
     "execution_count": 13,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "r = env.reset()\n",
    "r[1]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "hZrRB2_-F8kK"
   },
   "source": [
    "The dict returned by env contains the following fields:\n",
    "\n",
    "\n",
    "*   `admissible_commands`: list of commands (=actions) we can execute from the current state\n",
    "*   `description`: generic description of the scene (observation)\n",
    "* `intermediate_reward`: additional reward showing us we're on the right or wrong track upon solution. TextWorld description says it equals +1 every time path to the solution becomes shorter and -1 if we do something which pushes us away from final game goal\n",
    "* `inventory`: description of things you're carrying\n",
    "* `policy_commands`: list of commands we need to execute to get to the goal from the current state. We're not going to use this information during the training.\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "G69OTcDLfXLW"
   },
   "source": [
    "## Observations and actions\n",
    "\n",
    "Both observation and action space are of new `Word` class, which is an extension of TextWorld, representing variable-length sequence of words from some vocabulary.\n",
    "\n",
    "Code is here: https://github.com/microsoft/TextWorld/blob/40e13d9bd1d3bb64a98504a8d50d8873884cc08c/textworld/gym/spaces/text_spaces.py#L86"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 34
    },
    "colab_type": "code",
    "collapsed": false,
    "id": "E3JCA48UfUG9",
    "outputId": "5c69b9c3-e2a7-4f5d-869f-0974dcc98665"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Word(L=200, V=1249)"
      ]
     },
     "execution_count": 15,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "env.observation_space"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 34
    },
    "colab_type": "code",
    "collapsed": false,
    "id": "4NUnm7QsgZsS",
    "outputId": "b44ef205-8a76-41ce-9410-15f26f3eb626"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([   3, 1239,  407, 1245,  558,   35,  919,    2])"
      ]
     },
     "execution_count": 16,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tokens = env.observation_space.tokenize(\"You find yourself in a scullery\")\n",
    "tokens"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 34
    },
    "colab_type": "code",
    "collapsed": false,
    "id": "4mLXl0UDgwaB",
    "outputId": "5ec1ccc5-d72e-4893-d2a9-1af64c77720a"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['<PAD>', '<UNK>', '</S>', '<S>']"
      ]
     },
     "execution_count": 17,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "env.observation_space.vocab[:4]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 34
    },
    "colab_type": "code",
    "collapsed": false,
    "id": "SAA2c3bWhBlI",
    "outputId": "019111eb-e9b9-4338-a907-b9dc598cbb94"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['<S>', 'you', 'find', 'yourself', 'in', 'a', 'scullery', '</S>']"
      ]
     },
     "execution_count": 18,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "[env.observation_space.vocab[t] for t in tokens]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "mydhPrcWhaDQ"
   },
   "source": [
    "# Baseline DQN agent\n",
    "\n",
    "First step will be the DQN which takes the state description and one of the commands from 'admissible commands' list and estimate Q-value using the Bellman equation.\n",
    "\n",
    "---\n",
    "\n",
    "![alt text](https://raw.githubusercontent.com/Shmuma/rlss2019-hands-on/textworld/imgs/RLSS-TW-arch.png)\n",
    "\n",
    "## Preprocessor pipeline\n",
    "\n",
    "Before we can start training the agent, we need to convert the data into NN-friendly form. DQN net accepts the fixed-sized vector which encodes state description and command we're going to execute.\n",
    "\n",
    "Preprocessing consists of several steps:\n",
    "* from raw strings given by environment to sequence of token ids -- indices in underlying vocabulary. This step is already implemented by TextWorld in `Word.tokenize()` method, but we need to implement its call in a convenient and flexible way.\n",
    "* every token in sequence should be encoded into dense vector using so called *Embeddings*, which is a standard NLP practice to get from individual sequence entries into NN representation. You can take pre-trained embeddings or train your own using word2vec or glove methods. In our demo we go with third way: train embeddings as part of our model.\n",
    "* Embeddings convert every token in sequence into dense vector of fixed size, but the sequence as a whole still has variable length. To deal with that variability we'll use RNNs, which, being applied to input sequence can encode it into fixed size vector, capturing the squence as a whole. This step is labelled as Encoders on the diagram above. We'll train invidiual encoder for every input sequence.\n",
    "* Every encoder outputs fixed-size vector capturing the specifics from the sequence, so we just concatenate them together to get single fixed-size input to DQN network.\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "collapsed": true,
    "id": "oOjCmk0lMZrL"
   },
   "outputs": [],
   "source": [
    "import ptan\n",
    "import random\n",
    "import pathlib\n",
    "import logging\n",
    "import numpy as np\n",
    "from typing import Tuple, List, Iterable, Optional, Any\n",
    "from textworld.gym import spaces as tw_spaces\n",
    "from textworld.envs.wrappers.filter import EnvInfos"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "r_JZHhbL-3gL"
   },
   "source": [
    "## Env encoder wrapper\n",
    "\n",
    "To simplify further experiments, below is the wrapper around TextWorld environment which transforms the observations using `Word.tokenize()` method\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "collapsed": true,
    "id": "kcAZQQk8CYjd"
   },
   "outputs": [],
   "source": [
    "STEPS_LIMIT = 50\n",
    "GAME_FILE = \"simple1.ulx\"\n",
    "\n",
    "EXTRA_GAME_INFO = {\n",
    "  \"inventory\": True,\n",
    "  \"description\": True,\n",
    "  \"intermediate_reward\": True,\n",
    "  \"admissible_commands\": True,\n",
    "}\n",
    "\n",
    "\n",
    "def make_env():\n",
    "  game_path = pathlib.Path(GAME_FILE)\n",
    "  env_id = tw_gym.register_game(\n",
    "      str(game_path), max_episode_steps=STEPS_LIMIT,\n",
    "      name=game_path.stem, request_infos=EnvInfos(**EXTRA_GAME_INFO))\n",
    "  env = gym.make(env_id)\n",
    "  return env"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 34
    },
    "colab_type": "code",
    "collapsed": false,
    "id": "BmoNnM_BBemy",
    "outputId": "55118855-0e1d-4da3-a763-9a7486c84248"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<TimeLimit<TextworldGamesEnv<tw-simple1-v1>>>"
      ]
     },
     "execution_count": 21,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "env = make_env()\n",
    "env\n",
    "  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "collapsed": true,
    "id": "dPe6Yp4__dBP"
   },
   "outputs": [],
   "source": [
    "class TextWorldPreproc(gym.Wrapper):\n",
    "    \"\"\"\n",
    "    Simple wrapper to preprocess text_world game observation\n",
    "    \"\"\"\n",
    "    log = logging.getLogger(\"TextWorldPreproc\")\n",
    "    \n",
    "    def __init__(self, env: gym.Env, encode_raw_text: bool = False,\n",
    "                 encode_extra_fields: Iterable[str] = ('description', 'inventory'),\n",
    "                 use_admissible_commands: bool = True,\n",
    "                 use_intermediate_reward: bool = True,\n",
    "                 tokens_limit: Optional[int] = None):\n",
    "        \"\"\"\n",
    "        :param env: env to be wrapped. Has to provide Word observations\n",
    "        :param encode_raw_text: do we need to encode raw observation from environment, if true, adds an extra encoder\n",
    "        :param encode_extra_fields: tuple of field names to be encoded, expected to be string values\n",
    "        :param use_admissible_commands: if true, admissible commands used for action wrapping\n",
    "        :param use_intermediate_reward: take intermediate reward into account\n",
    "        :param tokens_limit: optional limit of tokens in the encoded fields\n",
    "        \"\"\"\n",
    "        super(TextWorldPreproc, self).__init__(env)\n",
    "        if not isinstance(env.observation_space, tw_spaces.Word):\n",
    "            raise ValueError(\"Env should expose text_world compatible observation space, \"\n",
    "                             \"this one gives %s\" % env.observation_space)\n",
    "        self._encode_raw_text = encode_raw_text\n",
    "        self._encode_extra_field = tuple(encode_extra_fields)\n",
    "        self._use_admissible_commands = use_admissible_commands\n",
    "        self._use_intermedate_reward = use_intermediate_reward\n",
    "        self._num_fields = len(self._encode_extra_field) + int(self._encode_raw_text)\n",
    "        self._last_admissible_commands = None\n",
    "        self._last_extra_info = None\n",
    "        self._tokens_limit = tokens_limit\n",
    "        self._cmd_hist = []\n",
    "\n",
    "    @property\n",
    "    def num_fields(self):\n",
    "        return self._num_fields\n",
    "\n",
    "    def _encode(self, obs: str, extra_info: dict) -> dict:\n",
    "        obs_result = []\n",
    "        if self._encode_raw_text:\n",
    "            tokens = self.env.observation_space.tokenize(obs)\n",
    "            if self._tokens_limit is not None:\n",
    "                tokens = tokens[:self._tokens_limit]\n",
    "            obs_result.append(tokens)\n",
    "        for field in self._encode_extra_field:\n",
    "            tokens = self.env.observation_space.tokenize(extra_info[field])\n",
    "            if self._tokens_limit is not None:\n",
    "                tokens = tokens[:self._tokens_limit]\n",
    "            obs_result.append(tokens)\n",
    "        result = {\"obs\": obs_result}\n",
    "        if self._use_admissible_commands:\n",
    "            adm_result = []\n",
    "            for cmd in extra_info['admissible_commands']:\n",
    "                adm_result.append(self.env.action_space.tokenize(cmd))\n",
    "            result['admissible_commands'] = adm_result\n",
    "            self._last_admissible_commands = extra_info['admissible_commands']\n",
    "        self._last_extra_info = extra_info\n",
    "        return result\n",
    "\n",
    "    # TextWorld environment has a workaround of gym drawback: \n",
    "    # reset returns tuple with raw observation and extra dict\n",
    "    def reset(self):\n",
    "        res = self.env.reset()\n",
    "        self._cmd_hist = []\n",
    "        return self._encode(res[0], res[1])\n",
    "\n",
    "    def step(self, action):\n",
    "        if self._use_admissible_commands:\n",
    "            action = self._last_admissible_commands[action]\n",
    "            self._cmd_hist.append(action)\n",
    "        obs, r, is_done, extra = self.env.step(action)\n",
    "        if self._use_intermedate_reward:\n",
    "            r += extra.get('intermediate_reward', 0)\n",
    "        new_extra = dict(extra)\n",
    "        for f in self._encode_extra_field + ('admissible_commands', 'intermediate_reward'):\n",
    "            if f in new_extra:\n",
    "                new_extra.pop(f)\n",
    "        # if is_done:\n",
    "        #     self.log.info(\"Commands: %s\", self._cmd_hist)\n",
    "        #     self.log.info(\"Reward: %s, extra: %s\", r, new_extra)\n",
    "        return self._encode(obs, extra), r, is_done, new_extra\n",
    "\n",
    "    @property\n",
    "    def last_admissible_commands(self):\n",
    "        return tuple(self._last_admissible_commands) if self._last_admissible_commands else None\n",
    "\n",
    "    @property\n",
    "    def last_extra_info(self):\n",
    "        return self._last_extra_info\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 34
    },
    "colab_type": "code",
    "collapsed": false,
    "id": "MXY1rvQDDqKx",
    "outputId": "711539c5-7e4f-4e80-bdc8-72dc0f3c8571"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<TextWorldPreproc<TimeLimit<TextworldGamesEnv<tw-simple1-v1>>>>"
      ]
     },
     "execution_count": 23,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "e = TextWorldPreproc(env)\n",
    "e"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "cbaYMcpwI8JM"
   },
   "source": [
    "Preprocessor applies tokenisation to original environment observations and repacks it into dict with two fileds: `admissible_commands`, which is a list of commands to take and `obs` list, keeping tokenised sequences building up our observation. \n",
    "\n",
    "Let's check how it works"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 374
    },
    "colab_type": "code",
    "collapsed": false,
    "id": "ZyRzLN3CDun-",
    "outputId": "bf798fab-5377-44a7-f331-2793693fa468"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'admissible_commands': [array([  3, 341, 589,   2]),\n",
       "  array([  3, 341, 599,   2]),\n",
       "  array([  3, 382, 333,   2]),\n",
       "  array([  3, 382, 452,   2]),\n",
       "  array([  3, 382, 589,   2]),\n",
       "  array([  3, 382, 599,   2]),\n",
       "  array([  3, 382, 860,   2]),\n",
       "  array([  3, 574,   2]),\n",
       "  array([  3, 647,   2]),\n",
       "  array([  3, 762, 333,   2]),\n",
       "  array([  3, 854, 589, 755, 860,   2]),\n",
       "  array([  3, 854, 599, 755, 860,   2]),\n",
       "  array([   3, 1137,  452, 1211,  589,    2])],\n",
       " 'obs': [array([   3,  919,    1,  584,  948, 1144,  558,   35,  919,    2,    3,\n",
       "         1239,  126, 1102, 1052, 1012,  746,    1,  523,    2,    3,  525,\n",
       "         1177, 1102,  925,   35,  860,  647,  776, 1081,   35,  860,    2,\n",
       "            3, 1078,  860,  577,  770,    2,    3, 1135, 1081,    1,   35,\n",
       "         1084,  755,  579,    2,    3, 1081,  577,   35,  235,  452,  613,\n",
       "          352,    2,    3, 1081,  577,   35,  235,  333,  613,  986,    2,\n",
       "            3,    2]),\n",
       "  array([   3, 1239,   78,    1,   35,  589,   35,  599,    2])]}"
      ]
     },
     "execution_count": 24,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "r = e.reset()\n",
    "r"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "7PmXI6oeJXKv"
   },
   "source": [
    "Besides transformed observations, the wrapper keeps two fields: `last_extra_info` with raw untransformed state from TextWorld environment and `last_admissible_commands` with list of text commands available. They are useful for debugging"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 309
    },
    "colab_type": "code",
    "collapsed": false,
    "id": "nXdKj8rbECJt",
    "outputId": "df5392aa-6ede-4e9b-9547-9fc81d093490"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'admissible_commands': ['drop keycard',\n",
       "  'drop ladle',\n",
       "  'examine door',\n",
       "  'examine gate',\n",
       "  'examine keycard',\n",
       "  'examine ladle',\n",
       "  'examine rack',\n",
       "  'inventory',\n",
       "  'look',\n",
       "  'open door',\n",
       "  'put keycard on rack',\n",
       "  'put ladle on rack',\n",
       "  'unlock gate with keycard'],\n",
       " 'description': \"-= Scullery =-\\nYou've just shown up in a scullery. You begin to take stock of what's here.\\n\\nHey, want to see a rack? Look over there, a rack. The rack is ordinary. Unfortunately, there isn't a thing on it.\\n\\nThere is a closed gate leading east. There is a closed door leading south.\\n\\n\\n\",\n",
       " 'intermediate_reward': 0,\n",
       " 'inventory': 'You are carrying:\\n  a keycard\\n  a ladle\\n\\n\\n'}"
      ]
     },
     "execution_count": 11,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "e.last_extra_info"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 442
    },
    "colab_type": "code",
    "collapsed": false,
    "id": "VbIREqE5ELyj",
    "outputId": "de35b190-cb31-47a9-f8b6-2c4c8865e062"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "({'admissible_commands': [array([  3, 341, 589,   2]),\n",
       "   array([  3, 341, 599,   2]),\n",
       "   array([  3, 382, 333,   2]),\n",
       "   array([  3, 382, 452,   2]),\n",
       "   array([  3, 382, 589,   2]),\n",
       "   array([  3, 382, 599,   2]),\n",
       "   array([  3, 382, 860,   2]),\n",
       "   array([  3, 574,   2]),\n",
       "   array([   3,  638,  452, 1211,  589,    2]),\n",
       "   array([  3, 647,   2]),\n",
       "   array([  3, 762, 333,   2]),\n",
       "   array([  3, 762, 452,   2]),\n",
       "   array([  3, 854, 589, 755, 860,   2]),\n",
       "   array([  3, 854, 599, 755, 860,   2])],\n",
       "  'obs': [array([   3,  919,    1,  584,  948, 1144,  558,   35,  919,    2,    3,\n",
       "          1239,  126, 1102, 1052, 1012,  746,    1,  523,    2,    3,  525,\n",
       "          1177, 1102,  925,   35,  860,  647,  776, 1081,   35,  860,    2,\n",
       "             3, 1078,  860,  577,  770,    2,    3, 1135, 1081,    1,   35,\n",
       "          1084,  755,  579,    2,    3, 1081,  577,   35,  235,  452,  613,\n",
       "           352,    2,    3, 1081,  577,   35,  235,  333,  613,  986,    2,\n",
       "             3,    2]),\n",
       "   array([   3, 1239,   78,    1,   35,  589,   35,  599,    2])]},\n",
       " 1,\n",
       " False,\n",
       " {})"
      ]
     },
     "execution_count": 12,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "e.step(e.last_admissible_commands.index('unlock gate with keycard'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "AfxgzqxuXrCJ"
   },
   "source": [
    "Please note reward=1 (second entry in returned tuple) in the result above. That's due to `intermediate_reward` option set in the env, which gives us reward every time we do a step in the right direction. You can disable this to make problem harder."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 326
    },
    "colab_type": "code",
    "collapsed": false,
    "id": "5NjiXKxWIq_t",
    "outputId": "21694d65-fe3d-4c7d-83ed-0641b6341d19"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'admissible_commands': ['drop keycard',\n",
       "  'drop ladle',\n",
       "  'examine door',\n",
       "  'examine gate',\n",
       "  'examine keycard',\n",
       "  'examine ladle',\n",
       "  'examine rack',\n",
       "  'inventory',\n",
       "  'lock gate with keycard',\n",
       "  'look',\n",
       "  'open door',\n",
       "  'open gate',\n",
       "  'put keycard on rack',\n",
       "  'put ladle on rack'],\n",
       " 'description': \"-= Scullery =-\\nYou've just shown up in a scullery. You begin to take stock of what's here.\\n\\nHey, want to see a rack? Look over there, a rack. The rack is ordinary. Unfortunately, there isn't a thing on it.\\n\\nThere is a closed gate leading east. There is a closed door leading south.\\n\\n\\n\",\n",
       " 'intermediate_reward': 1,\n",
       " 'inventory': 'You are carrying:\\n  a keycard\\n  a ladle\\n\\n\\n'}"
      ]
     },
     "execution_count": 13,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "e.last_extra_info"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 442
    },
    "colab_type": "code",
    "collapsed": false,
    "id": "ygHkW18sIvML",
    "outputId": "5b0c569c-1fcc-4aaa-9aab-917084a7f703"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "({'admissible_commands': [array([  3, 234, 452,   2]),\n",
       "   array([  3, 341, 589,   2]),\n",
       "   array([  3, 341, 599,   2]),\n",
       "   array([  3, 382, 333,   2]),\n",
       "   array([  3, 382, 452,   2]),\n",
       "   array([  3, 382, 589,   2]),\n",
       "   array([  3, 382, 599,   2]),\n",
       "   array([  3, 382, 860,   2]),\n",
       "   array([  3, 465, 352,   2]),\n",
       "   array([  3, 574,   2]),\n",
       "   array([  3, 647,   2]),\n",
       "   array([  3, 762, 333,   2]),\n",
       "   array([  3, 854, 589, 755, 860,   2]),\n",
       "   array([  3, 854, 599, 755, 860,   2])],\n",
       "  'obs': [array([   3,  919,    1,  584,  948, 1144,  558,   35,  919,    2,    3,\n",
       "          1239,  126, 1102, 1052, 1012,  746,    1,  523,    2,    3,  525,\n",
       "          1177, 1102,  925,   35,  860,  647,  776, 1081,   35,  860,    2,\n",
       "             3, 1078,  860,  577,  770,    2,    3, 1135, 1081,    1,   35,\n",
       "          1084,  755,  579,    2,    3, 1081,  577,   63,  762,  452,  613,\n",
       "           352,    2,    3, 1081,  577,   35,  235,  333,  613,  986,    2,\n",
       "             3,    2]),\n",
       "   array([   3, 1239,   78,    1,   35,  589,   35,  599,    2])]},\n",
       " 1,\n",
       " False,\n",
       " {})"
      ]
     },
     "execution_count": 14,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "e.step(e.last_admissible_commands.index('open gate'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 326
    },
    "colab_type": "code",
    "collapsed": false,
    "id": "Wbu2jyaYJCbj",
    "outputId": "b05b1ed4-f51c-4a62-fe0d-6734ecbdf3e4"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'admissible_commands': ['close gate',\n",
       "  'drop keycard',\n",
       "  'drop ladle',\n",
       "  'examine door',\n",
       "  'examine gate',\n",
       "  'examine keycard',\n",
       "  'examine ladle',\n",
       "  'examine rack',\n",
       "  'go east',\n",
       "  'inventory',\n",
       "  'look',\n",
       "  'open door',\n",
       "  'put keycard on rack',\n",
       "  'put ladle on rack'],\n",
       " 'description': \"-= Scullery =-\\nYou've just shown up in a scullery. You begin to take stock of what's here.\\n\\nHey, want to see a rack? Look over there, a rack. The rack is ordinary. Unfortunately, there isn't a thing on it.\\n\\nThere is an open gate leading east. There is a closed door leading south.\\n\\n\\n\",\n",
       " 'intermediate_reward': 1,\n",
       " 'inventory': 'You are carrying:\\n  a keycard\\n  a ladle\\n\\n\\n'}"
      ]
     },
     "execution_count": 15,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "e.last_extra_info"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 544
    },
    "colab_type": "code",
    "collapsed": false,
    "id": "8Dn1YIrgJPPV",
    "outputId": "d9f7b40b-334d-41fb-80f1-3a16ddb1a8cf"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "({'admissible_commands': [array([  3, 234, 452,   2]),\n",
       "   array([  3, 341, 589,   2]),\n",
       "   array([  3, 341, 599,   2]),\n",
       "   array([  3, 382, 269,   2]),\n",
       "   array([  3, 382, 276,   2]),\n",
       "   array([  3, 382, 452,   2]),\n",
       "   array([  3, 382, 589,   2]),\n",
       "   array([  3, 382, 599,   2]),\n",
       "   array([  3, 382, 935,   2]),\n",
       "   array([   3,  382, 1108,  506,    2]),\n",
       "   array([  3, 465, 986,   2]),\n",
       "   array([   3,  465, 1195,    2]),\n",
       "   array([  3, 574,   2]),\n",
       "   array([  3, 647,   2]),\n",
       "   array([  3, 854, 589, 755, 276,   2]),\n",
       "   array([  3, 854, 599, 755, 276,   2]),\n",
       "   array([   3, 1052,  269,  441,  276,    2]),\n",
       "   array([   3, 1052,  935,  441,  276,    2]),\n",
       "   array([   3, 1052, 1108,  506,    2])],\n",
       "  'obs': [array([   3,   92, 1239,  407, 1245,  558,   63,   92,    2,    3,   35,\n",
       "           727,  590,  746,  810,    2,    3, 1239,  916, 1078,  893,  428,\n",
       "            35,  276,   65, 1239,  407,   35,  276,    2,    3,  755, 1078,\n",
       "           276, 1239,  199,  661,  774,   35,  269,   65,   35,  935,    2,\n",
       "             3, 1081,  577,   63,  762,  452,  613, 1195,    2,    3, 1081,\n",
       "           577,   63,  387, 1102, 1078,  986,    2,    3,    1, 1229,  579,\n",
       "           577, 1132,    2,    3, 1081,  577,   35, 1108,  506,  755, 1078,\n",
       "           419,    2,    3,    2]),\n",
       "   array([   3, 1239,   78,    1,   35,  589,   35,  599,    2])]},\n",
       " 1,\n",
       " False,\n",
       " {})"
      ]
     },
     "execution_count": 16,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "e.step(e.last_admissible_commands.index('go east'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "hW_03R41bt_G"
   },
   "source": [
    "So, the observation from `TextWorldPreproc` is a dict with two keys: \n",
    "* `admissible_commands`: list of commands available in the current state. Index in this array is the action to be pased to `step()` function\n",
    "* `obs`: list of arrays with sequences, by default `description` and `inventory`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "T3jMq6NeeHM8"
   },
   "source": [
    "## Preprocessor (practice)\n",
    "\n",
    "The next steps in our pipeline are embeddings and encoders, you should implement using template code below. Class `Encoder` is simple wrapper around LSTM layer, and already provided.\n",
    "\n",
    "In **tests** section, you find several use-cases of this class to check your functionality. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "collapsed": true,
    "id": "U9OWcgSQwhlJ"
   },
   "outputs": [],
   "source": [
    "import logging\n",
    "import torch\n",
    "import torch.nn as nn\n",
    "import torch.nn.utils.rnn as rnn_utils\n",
    "import torch.nn.functional as F\n",
    "import numpy as np"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "collapsed": true,
    "id": "iA0u5YeCwdgr"
   },
   "outputs": [],
   "source": [
    "class Encoder(nn.Module):\n",
    "    \"\"\"\n",
    "    Takes input sequences (after embeddings) and returns the hidden state from LSTM\n",
    "    \"\"\"\n",
    "    def __init__(self, emb_size: int, out_size: int):\n",
    "        super(Encoder, self).__init__()\n",
    "\n",
    "        self.net = nn.LSTM(input_size=emb_size, hidden_size=out_size, batch_first=True)\n",
    "\n",
    "    def forward(self, x):\n",
    "        self.net.flatten_parameters()\n",
    "        _, hid_cell = self.net(x)\n",
    "        # Warn: if bidir=True or several layers, sequeeze has to be changed!\n",
    "        return hid_cell[0].squeeze(0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "GIkU9onpKtYo"
   },
   "source": [
    "Function `order_batch` is used to sort the batch of input sequences by descending of sequence length. This is a PyTorch requirement, which itself is imposed by low-level cudnn implementation.\n",
    "\n",
    "`order_batch` sorts the input batch, returns it and in addition returns the indices of sequences to return the batch to the original order."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "collapsed": true,
    "id": "muixwId92Klh"
   },
   "outputs": [],
   "source": [
    "def order_batch(batch):\n",
    "    \"\"\"\n",
    "    Order batch of sequences by sequence len and keep inverse\n",
    "    :param batch: list of lists of items\n",
    "    :return: Tuple[sorted batch, numpy array to return the order]\n",
    "    \"\"\"\n",
    "    lens = list(map(len, batch))\n",
    "    ord_idx = np.flip(np.argsort(lens, kind='stable'))\n",
    "    ord_inv = np.argsort(ord_idx, kind='stable')\n",
    "    ord_batch = [batch[idx] for idx in ord_idx]\n",
    "    return ord_batch, ord_inv"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "mqtHtoYkMw5z"
   },
   "source": [
    "Now, this is the class, you need to implement. Function `_apply_encoder` is the core of class functionality and should apply one `Encoder` instance to the batch of sequences. You should use function [`rnn_utils.pack_sequence`](https://pytorch.org/docs/stable/nn.html#torch.nn.utils.rnn.pack_sequence) to obtain instance of [`PackedSequence` class](https://pytorch.org/docs/stable/nn.html#packedsequence). \n",
    "\n",
    "This might be tricky if you haven't played with PyTorch RNNs before, so, you probably should first try to apply simple RNN to small toy tensors to understand they idea. Those links are also could be useful:\n",
    "* https://stackoverflow.com/questions/51030782/why-do-we-pack-the-sequences-in-pytorch\n",
    "* https://discuss.pytorch.org/t/simple-working-example-how-to-use-packing-for-variable-length-sequence-inputs-for-rnn/2120\n",
    "* https://www.kdnuggets.com/2018/06/taming-lstms-variable-sized-mini-batches-pytorch.html"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "collapsed": true,
    "id": "XqYjp6G0xAG4"
   },
   "outputs": [],
   "source": [
    "class Preprocessor(nn.Module):\n",
    "    \"\"\"\n",
    "    Takes batch of several input sequences and outputs their summary from one or many encoders\n",
    "    \"\"\"\n",
    "    def __init__(self, dict_size: int, emb_size: int, num_sequences: int, enc_output_size: int):\n",
    "        \"\"\"\n",
    "        :param dict_size: amount of words is our vocabulary\n",
    "        :param emb_size: dimensionality of embeddings\n",
    "        :param num_sequences: count of sequences\n",
    "        :param enc_output_size: output from single encoder\n",
    "        \"\"\"\n",
    "        super(Preprocessor, self).__init__()\n",
    "\n",
    "        # TODO: construct the class\n",
    "        # 1. create nn.Embedding layer\n",
    "        # 2. build a list of Encoder instances for input sequences\n",
    "        # 3. register them in the module using self.add_module() call\n",
    "        # 4. create separate Encoder instance for command encoding\n",
    "        raise NotImplementedError\n",
    "        \n",
    "    def encode_sequences(self, batches):\n",
    "        \"\"\"\n",
    "        Encode batches of sequences. Input is a list of observations, each \n",
    "        observation is a list of sequences. Corresponding encoders should\n",
    "        be applied to corresponding sequence.\n",
    "        \n",
    "        More concretely, if we have two sequences: 'description' and 'inventory',\n",
    "        input could be:\n",
    "        \n",
    "        [\n",
    "          [\n",
    "            [tokens of \"you are standing on a cliff\"],\n",
    "            [tokens of \"you have an axe\"]\n",
    "\n",
    "          ],\n",
    "          [\n",
    "            [tokens of \"you're in cosy chair\"],\n",
    "            [tokens of \"you're holding cup of tea\"]\n",
    "          ]\n",
    "        ]\n",
    "        \n",
    "        The output from every encoder should be concatenated, keeping batches \n",
    "        over the first dimension.\n",
    "        \n",
    "        Implemented properly, should be just call to _apply_encoder in a loop \n",
    "        with torch.cat() call\n",
    "        \n",
    "        :param batches: list of tuples with variable-length sequences of word ids\n",
    "        :return: tensor with concatenated encoder outputs for every batch sample\n",
    "        \"\"\"\n",
    "        raise NotImplementedError\n",
    "\n",
    "    def encode_commands(self, batch):\n",
    "        \"\"\"\n",
    "        Apply encoder to list of commands sequence.\n",
    "        \n",
    "        Should be just call to _apply_encoder()\n",
    "        \n",
    "        :param batch: list of lists of idx\n",
    "        :return: tensor with encoded commands in original order\n",
    "        \"\"\"\n",
    "        raise NotImplementedError\n",
    "        \n",
    "    def _apply_encoder(self, batch, encoder):\n",
    "        \"\"\"\n",
    "        A bit tricky internal method which applies encoder to the batch (which\n",
    "        is a list of variable-length sequences).\n",
    "        \n",
    "        You should use order_batch() function above (to properly sort the sequence),\n",
    "        and rnn_utils.pack_sequence and rnn_utils.PackedSequence. Alternatively,\n",
    "        you can use padding, but this will make the implementation less memory-efficient.\n",
    "        \n",
    "        And don't forget about embeddings!\n",
    "        \n",
    "        Output is a tensor of size (len(batch), encoder_out_dim)\n",
    "        \"\"\"\n",
    "        raise NotImplementedError\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "VtwuS0jB23xC"
   },
   "source": [
    "### Tests\n",
    "\n",
    "Several test cases for the code you've just written. Might be helpful to understand the inputs and outputs of the methods to be implemented, so, don't hesitate to study test cases!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 221
    },
    "colab_type": "code",
    "collapsed": false,
    "id": "GDzgwXfg3G0i",
    "outputId": "d49a8c5c-bb16-45f9-806e-578944ec2903"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Preprocessor(\n",
       "  (emb): Embedding(10, 5)\n",
       "  (enc_0): Encoder(\n",
       "    (net): LSTM(5, 7, batch_first=True)\n",
       "  )\n",
       "  (enc_1): Encoder(\n",
       "    (net): LSTM(5, 7, batch_first=True)\n",
       "  )\n",
       "  (enc_commands): Encoder(\n",
       "    (net): LSTM(5, 7, batch_first=True)\n",
       "  )\n",
       ")"
      ]
     },
     "execution_count": 19,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# construct our preprocessor, vocab has 10 tokens, embeddings of size 5, two input sequences, Encoder should return vectors of size 7\n",
    "prep = Preprocessor(dict_size=10, emb_size=5, num_sequences=2, enc_output_size=7)\n",
    "prep"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 68
    },
    "colab_type": "code",
    "collapsed": false,
    "id": "EYPH8nH83ftL",
    "outputId": "76f8d178-d87d-441a-e062-9e718e791ad0"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([[ 0.2047, -0.0103, -0.1646,  0.1190,  0.2260, -0.0132,  0.2552],\n",
      "        [ 0.0952,  0.1719, -0.0816,  0.0340,  0.0970, -0.0745,  0.2033]],\n",
      "       grad_fn=<IndexBackward>)\n"
     ]
    }
   ],
   "source": [
    "# apply encoder to two sequences, should return 2x7 tensor\n",
    "enc = Encoder(emb_size=5, out_size=7)\n",
    "t = prep._apply_encoder([\n",
    "    [1, 2, 3], [2],\n",
    "], enc)\n",
    "print(t)\n",
    "assert t.size() == (2, 7)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 68
    },
    "colab_type": "code",
    "collapsed": false,
    "id": "8eJ2wwHH4miR",
    "outputId": "c66c6bcd-bb39-4a66-db19-f66c516a1865"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([[ 0.0952,  0.1719, -0.0816,  0.0340,  0.0970, -0.0745,  0.2033],\n",
      "        [ 0.2047, -0.0103, -0.1646,  0.1190,  0.2260, -0.0132,  0.2552]],\n",
      "       grad_fn=<IndexBackward>)\n"
     ]
    }
   ],
   "source": [
    "# same batch reversed should have reversed tensor\n",
    "tt = prep._apply_encoder([\n",
    "    [2],\n",
    "    [1, 2, 3], \n",
    "], enc)\n",
    "print(tt)\n",
    "tn = t.detach().numpy()\n",
    "ttn = tt.detach().numpy()\n",
    "np.testing.assert_allclose(tn[0], ttn[1])\n",
    "np.testing.assert_allclose(tn[1], ttn[0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 68
    },
    "colab_type": "code",
    "collapsed": false,
    "id": "quD35tPU5aGl",
    "outputId": "43af221c-55a9-4d23-8d3b-216ca9b6e0ca"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([[ 0.0931, -0.1730,  0.0878,  0.2355,  0.1928,  0.0060, -0.0471],\n",
      "        [ 0.0445,  0.1387,  0.1410,  0.0521, -0.0650,  0.0325, -0.0029]],\n",
      "       grad_fn=<IndexBackward>)\n"
     ]
    }
   ],
   "source": [
    "# the same semantics as just encoder application, but encoder is different, so, numbers shouldn't match\n",
    "ttt = prep.encode_commands([[1, 2, 3], [2]])\n",
    "print(ttt)\n",
    "assert t.size() == (2, 7)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 102
    },
    "colab_type": "code",
    "collapsed": false,
    "id": "mk9_2CiD7jSI",
    "outputId": "407eb847-6225-4e09-9c8e-2cb44325b1c5"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([[-0.2038, -0.1217,  0.0930,  0.2409,  0.3689,  0.1423, -0.0894,  0.0599,\n",
      "          0.0747, -0.0321,  0.1589,  0.0231,  0.0087, -0.1703],\n",
      "        [-0.2038, -0.1217,  0.0930,  0.2409,  0.3689,  0.1423, -0.0894,  0.0369,\n",
      "          0.1102, -0.0396,  0.3131,  0.0297,  0.0061, -0.3218]],\n",
      "       grad_fn=<CatBackward>)\n"
     ]
    }
   ],
   "source": [
    "# encode sequences, now we have list of batches for every sequence\n",
    "t = prep.encode_sequences([\n",
    "    # obs1\n",
    "    [[1,2,3], [2]],\n",
    "    # obs2\n",
    "    [[1,2,3], [2,2,2]],\n",
    "])\n",
    "print(t)\n",
    "\n",
    "# the output shape should be [batch_size=2, encoder_size*num_sequences=7*2=14]\n",
    "assert t.size() == (2, 14)\n",
    "# another subtle property is that both observation have the same first sequence, \n",
    "# first 7 numbers in both rows should be the same\n",
    "torch.testing.assert_allclose(t[0][:7], t[1][:7])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 102
    },
    "colab_type": "code",
    "collapsed": false,
    "id": "K8FRLCYWQOjb",
    "outputId": "19c12481-07da-491f-f522-3aae654ebb9f"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([[-0.2038, -0.1217,  0.0930,  0.2409,  0.3689,  0.1423, -0.0894,  0.0369,\n",
      "          0.1102, -0.0396,  0.3131,  0.0297,  0.0061, -0.3218],\n",
      "        [-0.2038, -0.1217,  0.0930,  0.2409,  0.3689,  0.1423, -0.0894,  0.0599,\n",
      "          0.0747, -0.0321,  0.1589,  0.0231,  0.0087, -0.1703]],\n",
      "       grad_fn=<CatBackward>)\n"
     ]
    }
   ],
   "source": [
    "# try inversed sequences\n",
    "tt = prep.encode_sequences([\n",
    "    # obs1\n",
    "    [[1,2,3], [2,2,2]],\n",
    "    # obs2\n",
    "    [[1,2,3], [2]],\n",
    "\n",
    "])\n",
    "print(tt)\n",
    "\n",
    "# the output shape should be [batch_size=2, encoder_size*num_sequences=7*2=14]\n",
    "assert tt.size() == (2, 14)\n",
    "torch.testing.assert_allclose(tt[0][:7], tt[1][:7])\n",
    "torch.testing.assert_allclose(t[0], tt[1])\n",
    "torch.testing.assert_allclose(t[1], tt[0])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "eyqqc-mhkxuL"
   },
   "source": [
    "## Preprocessor (solution)\n",
    "\n",
    "**SPOILER Alert!** If you are not completely lost in previous excercise, it would be better to keep trying (or ask TAs for hints). Below is the complete solution, which might ruin some fun from the excercise :)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "collapsed": true,
    "id": "OYnQOJQTlGwk"
   },
   "outputs": [],
   "source": [
    "class Preprocessor(nn.Module):\n",
    "    \"\"\"\n",
    "    Takes batch of several input sequences and outputs their summary from one or many encoders\n",
    "    \"\"\"\n",
    "    def __init__(self, dict_size: int, emb_size: int, num_sequences: int, enc_output_size: int):\n",
    "        \"\"\"\n",
    "        :param dict_size: amount of words is our vocabulary\n",
    "        :param emb_size: dimensionality of embeddings\n",
    "        :param num_sequences: count of sequences\n",
    "        :param enc_output_size: output from single encoder\n",
    "        \"\"\"\n",
    "        super(Preprocessor, self).__init__()\n",
    "\n",
    "        self.emb = nn.Embedding(num_embeddings=dict_size, embedding_dim=emb_size)\n",
    "        self.encoders = []\n",
    "        for idx in range(num_sequences):\n",
    "            enc = Encoder(emb_size, enc_output_size)\n",
    "            self.encoders.append(enc)\n",
    "            self.add_module(f\"enc_{idx}\", enc)\n",
    "        self.enc_commands = Encoder(emb_size, enc_output_size)\n",
    "\n",
    "    def encode_sequences(self, batches):\n",
    "        \"\"\"\n",
    "        Forward pass of Preprocessor\n",
    "        :param batches: list of tuples with variable-length sequences of word ids\n",
    "        :return: tensor with concatenated encoder outputs for every batch sample\n",
    "        \"\"\"\n",
    "        data = []\n",
    "        for enc, enc_batch in zip(self.encoders, zip(*batches)):\n",
    "            data.append(self._apply_encoder(enc_batch, enc))\n",
    "        res_t = torch.cat(data, dim=1)\n",
    "        return res_t\n",
    "\n",
    "    def _apply_encoder(self, batch, encoder):\n",
    "        ord_batch, inv_batch = order_batch(batch)\n",
    "        dev = self.emb.weight.device\n",
    "        ord_batch_t = [torch.tensor(sample).to(dev) for sample in ord_batch]\n",
    "        batch_seq = rnn_utils.pack_sequence(ord_batch_t)\n",
    "        emb_seq_t = rnn_utils.PackedSequence(data=self.emb(batch_seq.data), batch_sizes=batch_seq.batch_sizes)\n",
    "        res = encoder(emb_seq_t)\n",
    "        res = res[inv_batch]\n",
    "        return res\n",
    "\n",
    "    def encode_commands(self, batch):\n",
    "        \"\"\"\n",
    "        Apply encoder to list of commands sequence\n",
    "        :param batch: list of lists of idx\n",
    "        :return: tensor with encoded commands in original order\n",
    "        \"\"\"\n",
    "        return self._apply_encoder(batch, self.enc_commands)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "qmqRvRvI58qB"
   },
   "source": [
    "## DQN model\n",
    "\n",
    "Final step in our diagram is DQN model, which takes the observation representation produced by encoders + encoded command to be executed and produces Q(s, a). \n",
    "\n",
    "The code is trivial and straghtforward, but of course, you can tweak the model if you want (layers, activations, etc).\n",
    "\n",
    "For convenience, in addition to `forward()` method, we implement `q_values()` which applies vector of observations and batch of commands to the model to obtain Q-values for all admissible commands we have in the particular state."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "collapsed": true,
    "id": "yFbryTit45f8"
   },
   "outputs": [],
   "source": [
    "class DQNModel(nn.Module):\n",
    "    def __init__(self, obs_size: int, cmd_size: int, hid_size: int = 256):\n",
    "        super(DQNModel, self).__init__()\n",
    "\n",
    "        self.net = nn.Sequential(\n",
    "            nn.Linear(obs_size + cmd_size, hid_size),\n",
    "            nn.ReLU(),\n",
    "            nn.Linear(hid_size, 1)\n",
    "        )\n",
    "\n",
    "    def forward(self, obs, cmd):\n",
    "        x = torch.cat((obs, cmd), dim=1)\n",
    "        return self.net(x)\n",
    "\n",
    "    def q_values(self, obs_t, commands_t):\n",
    "        \"\"\"\n",
    "        Calculate q-values for observation and tensor of commands\n",
    "        :param obs_t: preprocessed observation, need to be of [1, obs_size] shape\n",
    "        :param commands_t: commands to be evaluated, shape is [N, cmd_size]\n",
    "        :return: list of q-values for commands\n",
    "        \"\"\"\n",
    "        result = []\n",
    "        for cmd_t in commands_t:\n",
    "            result.append(self(obs_t, cmd_t.unsqueeze(0))[0].cpu().item())\n",
    "        return result\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "5euCik0qBDVG"
   },
   "source": [
    "## Ptan Agent (practice)\n",
    "\n",
    "Now, with all components of our pipeline implemented, we can start working on training process. We'll use Ptan library (https://github.com/Shmuma/ptan) simple PyTorch-based lib for RL. We've covered ptan on the previous tutorial (to be written), so, you might remember that the **agent** is one of the central concepts in ptan.\n",
    "\n",
    "As a reminder, the goal of agent is to convert batch of environment states into batch of actions to be executed. During this conversion, agent could apply exploration policy (by sampling random actions from time to time, for example) and can have an optional state (which is useful for PoMDP cases).\n",
    "\n",
    "In TextWorld we don't need states yet (as with inventory description our env is MDP). Below is the outline of the agent you need to write.\n",
    "\n",
    "Additional function our agent need to implement is epsilon-greedy exploration, which just means taking random action with given probability (denoted by epsilon). If epsilon=0, we always take action with maximum Q value. In case of epsilon=1, actions are always randomly sampled. We will decrease the epsilon during the training to allow agent to explore the environment."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "collapsed": true,
    "id": "uFq7sBRwDFcX"
   },
   "outputs": [],
   "source": [
    "class DQNAgent(ptan.agent.BaseAgent):\n",
    "    def __init__(self, net: DQNModel, preprocessor: Preprocessor, epsilon: float = 0.0, device=\"cpu\"):\n",
    "        \"\"\"\n",
    "        Construct the agent. We pass the model we've just defined, preprocessor\n",
    "        to convert observations and commands into input vector.\n",
    "        In addition, we allow agent to perform exploration with \n",
    "        given epsilon probability.\n",
    "        \"\"\"\n",
    "        self.net = net\n",
    "        self.preprocessor = preprocessor\n",
    "        self._epsilon = epsilon\n",
    "        self.device = device\n",
    "\n",
    "    @property\n",
    "    def epsilon(self):\n",
    "        return self._epsilon\n",
    "\n",
    "    @epsilon.setter\n",
    "    def epsilon(self, value: float):\n",
    "        if 0.0 <= value <= 1.0:\n",
    "            self._epsilon = value\n",
    "\n",
    "    @torch.no_grad()\n",
    "    def __call__(self, states: List[dict], agent_states: Optional[Any] = None) -> Tuple[List[int], List[Any]]:\n",
    "        \"\"\"\n",
    "        Argument `states` is batch of observations returned by \n",
    "        TextWorldPreproc, which is a dict with two keys: \n",
    "        'obs' and 'admissible_commands'\n",
    "        \"\"\"\n",
    "        if agent_states is None:\n",
    "            agent_states = [None] * len(states)\n",
    "\n",
    "        # for every state in the batch, calculate\n",
    "        actions = []\n",
    "        for state in states:\n",
    "            # for every state, you need to:\n",
    "            # 1. encode observations using preprocessor,\n",
    "            # 2. encode admissible commands\n",
    "            # 3. calculate q-values \n",
    "            # 4. choose the best action to do (index in admissible commands list)\n",
    "            # NB: do no forget about exploration!\n",
    "            raise NotImplementedError\n",
    "        return actions, agent_states"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "uHj48ULXI8ir"
   },
   "source": [
    "### Tests\n",
    "\n",
    "With code below you can check you implementation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "collapsed": true,
    "id": "56Qi-nV6JHPp"
   },
   "outputs": [],
   "source": [
    "# this code cell is also useful to understand how're going to use our classes\n",
    "env = make_env()\n",
    "env = TextWorldPreproc(env)\n",
    "\n",
    "ENC_SIZE = 5\n",
    "prep = Preprocessor(dict_size=env.observation_space.vocab_size, emb_size=20, \n",
    "                    num_sequences=env.num_fields, enc_output_size=ENC_SIZE)\n",
    "\n",
    "net = DQNModel(obs_size=env.num_fields*ENC_SIZE, cmd_size=ENC_SIZE)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "collapsed": true,
    "id": "04bBSp7ALBWd"
   },
   "outputs": [],
   "source": [
    "s = env.reset()\n",
    "\n",
    "# check the epsilon first - make the net to return zeroes all the time\n",
    "net.q_values = lambda obs, cmd: [0.0 for c in cmd]\n",
    "\n",
    "# no exploration, should be zeros\n",
    "agent = DQNAgent(net, prep, epsilon=0.0)\n",
    "acts, _ = agent([s for _ in range(10)])\n",
    "assert acts == [0]*10\n",
    "\n",
    "# exploration all the time, should not be only zeros\n",
    "agent = DQNAgent(net, prep, epsilon=1.0)\n",
    "acts, _ = agent([s for _ in range(10)])\n",
    "assert acts != [0]*10"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 34
    },
    "colab_type": "code",
    "collapsed": false,
    "id": "ir_CH5t9MNly",
    "outputId": "00c7ba4b-c37c-44a5-f5e3-8675d1045637"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[11, 9, 4, 10, 2, 0, 3, 1, 3, 8]"
      ]
     },
     "execution_count": 63,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "acts"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "zThuJ03sDIOE"
   },
   "source": [
    "## Ptan Agent (solution)\n",
    "\n",
    "**Spoiler alert**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "collapsed": true,
    "id": "ACM5Y1ajDM8m"
   },
   "outputs": [],
   "source": [
    "class DQNAgent(ptan.agent.BaseAgent):\n",
    "    def __init__(self, net: DQNModel, preprocessor: Preprocessor, epsilon: float = 0.0, device=\"cpu\"):\n",
    "        self.net = net\n",
    "        self.preprocessor = preprocessor\n",
    "        self._epsilon = epsilon\n",
    "        self.device = device\n",
    "\n",
    "    @property\n",
    "    def epsilon(self):\n",
    "        return self._epsilon\n",
    "\n",
    "    @epsilon.setter\n",
    "    def epsilon(self, value: float):\n",
    "        if 0.0 <= value <= 1.0:\n",
    "            self._epsilon = value\n",
    "\n",
    "    @torch.no_grad()\n",
    "    def __call__(self, states, agent_states=None):\n",
    "        if agent_states is None:\n",
    "            agent_states = [None] * len(states)\n",
    "\n",
    "        # for every state in the batch, calculate\n",
    "        actions = []\n",
    "        for state in states:\n",
    "            commands = state['admissible_commands']\n",
    "            if random.random() <= self.epsilon:\n",
    "                actions.append(random.randrange(len(commands)))\n",
    "            else:\n",
    "                obs_t = self.preprocessor.encode_sequences([state['obs']]).to(self.device)\n",
    "                commands_t = self.preprocessor.encode_commands(commands).to(self.device)\n",
    "                q_vals = self.net.q_values(obs_t, commands_t)\n",
    "                actions.append(np.argmax(q_vals))\n",
    "        return actions, agent_states"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "jmZVNWs6Ucdt"
   },
   "source": [
    "## Loss calculation (practice)\n",
    "\n",
    "Agent class allows us to start gathering training samples from environment. Experience buffering and sampling are already implemented in ptan, so, we'll just use existing classes. The only piece we're missing is **loss calculation**.\n",
    "\n",
    "In DQN we're using MSE loss between net prediction and estimation given by Bellman equation. In our baseline we'll use the simplest 1-step Bellman: \n",
    "\n",
    "$Q_{s,a} = r_{s,a} + \\gamma \\cdot max_{a'}Q_{s',a'}$\n",
    "\n",
    "Below one function is given to you: `unpack_batch`, which takes list of environment steps and convert them into the form, suitable for MSE calculation. Your task is to implement this MSE, which, in fact, just 5 lines of code.\n",
    "\n",
    "**NB** Function `unpack_batch` is not very efficient, as it calculates encoded sequences and commands for every state in the batch. You can make it much more efficient, by combining encoders applications. This is left as an excercise :)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "collapsed": true,
    "id": "Umh9TwX7YIsZ"
   },
   "outputs": [],
   "source": [
    "@torch.no_grad()\n",
    "def unpack_batch(batch: List[ptan.experience.Experience], preprocessor: Preprocessor,\n",
    "                 net: DQNModel, device=\"cpu\"):\n",
    "    \"\"\"\n",
    "    Convert batch to data needed for Bellman step\n",
    "    :param batch: list of ptan.Experience objects\n",
    "    :param preprocessor: emb.Preprocessor instance\n",
    "    :param net: network to be used for next state approximation\n",
    "    :param device: torch device\n",
    "    :return: tuple (list of observations, list of taken commands, list of rewards, list of best Qs for the next state)\n",
    "    \"\"\"\n",
    "    # calculate Qs for next states\n",
    "    observations, taken_commands, rewards, best_q = [], [], [], []\n",
    "    for exp in batch:\n",
    "        observations.append(exp.state['obs'])\n",
    "        taken_commands.append(exp.state['admissible_commands'][exp.action])\n",
    "        rewards.append(exp.reward)\n",
    "\n",
    "        # calculate best Q value for the next state\n",
    "        if exp.last_state is None:\n",
    "            # final state in the episode, Q=0\n",
    "            best_q.append(0.0)\n",
    "        else:\n",
    "            obs_t = preprocessor.encode_sequences([exp.last_state['obs']]).to(device)\n",
    "            commands_t = preprocessor.encode_commands(exp.last_state['admissible_commands']).to(device)\n",
    "            q_vals = net.q_values(obs_t, commands_t)\n",
    "            best_q.append(max(q_vals))\n",
    "    return observations, taken_commands, rewards, best_q\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "collapsed": true,
    "id": "0INdyoknYMdO"
   },
   "outputs": [],
   "source": [
    "def calc_loss_dqn(batch, preprocessor, tgt_preprocessor, net, tgt_net, gamma, device=\"cpu\"):\n",
    "    observations, taken_commands, rewards, next_best_qs = unpack_batch(batch, tgt_preprocessor, tgt_net, device)\n",
    "\n",
    "    # Here you need:\n",
    "    # 1. encode observations and commands,\n",
    "    # 2. convert them into q-values predicted by network\n",
    "    # 3. calculate the Bellman approximation using rewards and next_best_qs\n",
    "    # 4. calc MSE between predicted Qs and approximation\n",
    "    # Should be fairly straightforward, it took me 6 lines of code\n",
    "    \n",
    "    raise NotImplementedError"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "M0TiaajAZkUx"
   },
   "source": [
    "### Tests"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "collapsed": true,
    "id": "DZz4ojusZnGU"
   },
   "outputs": [],
   "source": [
    "# this code cell is also useful to understand how we are going to use our classes\n",
    "env = make_env()\n",
    "env = TextWorldPreproc(env)\n",
    "\n",
    "ENC_SIZE = 5\n",
    "prep = Preprocessor(dict_size=env.observation_space.vocab_size, emb_size=20, \n",
    "                    num_sequences=env.num_fields, enc_output_size=ENC_SIZE)\n",
    "\n",
    "net = DQNModel(obs_size=env.num_fields*ENC_SIZE, cmd_size=ENC_SIZE)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 258
    },
    "colab_type": "code",
    "collapsed": false,
    "id": "rGjNl0cq9y5R",
    "outputId": "a31f8693-6298-4b21-b5c1-dcbb8dd08800"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "ExperienceFirstLast(state={'obs': [array([   3,  919,    1,  584,  948, 1144,  558,   35,  919,    2,    3,\n",
       "       1239,  126, 1102, 1052, 1012,  746,    1,  523,    2,    3,  525,\n",
       "       1177, 1102,  925,   35,  860,  647,  776, 1081,   35,  860,    2,\n",
       "          3, 1078,  860,  577,  770,    2,    3, 1135, 1081,    1,   35,\n",
       "       1084,  755,  579,    2,    3, 1081,  577,   35,  235,  452,  613,\n",
       "        352,    2,    3, 1081,  577,   35,  235,  333,  613,  986,    2,\n",
       "          3,    2]), array([   3, 1239,   78,    1,   35,  589,   35,  599,    2])], 'admissible_commands': [array([  3, 341, 589,   2]), array([  3, 341, 599,   2]), array([  3, 382, 333,   2]), array([  3, 382, 452,   2]), array([  3, 382, 589,   2]), array([  3, 382, 599,   2]), array([  3, 382, 860,   2]), array([  3, 574,   2]), array([  3, 647,   2]), array([  3, 762, 333,   2]), array([  3, 854, 589, 755, 860,   2]), array([  3, 854, 599, 755, 860,   2]), array([   3, 1137,  452, 1211,  589,    2])]}, action=12, reward=1.0, last_state={'obs': [array([   3,  919,    1,  584,  948, 1144,  558,   35,  919,    2,    3,\n",
       "       1239,  126, 1102, 1052, 1012,  746,    1,  523,    2,    3,  525,\n",
       "       1177, 1102,  925,   35,  860,  647,  776, 1081,   35,  860,    2,\n",
       "          3, 1078,  860,  577,  770,    2,    3, 1135, 1081,    1,   35,\n",
       "       1084,  755,  579,    2,    3, 1081,  577,   35,  235,  452,  613,\n",
       "        352,    2,    3, 1081,  577,   35,  235,  333,  613,  986,    2,\n",
       "          3,    2]), array([   3, 1239,   78,    1,   35,  589,   35,  599,    2])], 'admissible_commands': [array([  3, 341, 589,   2]), array([  3, 341, 599,   2]), array([  3, 382, 333,   2]), array([  3, 382, 452,   2]), array([  3, 382, 589,   2]), array([  3, 382, 599,   2]), array([  3, 382, 860,   2]), array([  3, 574,   2]), array([   3,  638,  452, 1211,  589,    2]), array([  3, 647,   2]), array([  3, 762, 333,   2]), array([  3, 762, 452,   2]), array([  3, 854, 589, 755, 860,   2]), array([  3, 854, 599, 755, 860,   2])]})"
      ]
     },
     "execution_count": 86,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# do one step in the right direction (that's good to know where to go, right?)\n",
    "s = env.reset()\n",
    "orig_info = env.last_extra_info\n",
    "action = env.last_admissible_commands.index('unlock gate with keycard')\n",
    "ss, r, is_done, _ = env.step(action)\n",
    "info = env.last_extra_info\n",
    "exp = ptan.experience.ExperienceFirstLast(state=s, action=action, reward=float(r), last_state=ss)\n",
    "exp"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "collapsed": true,
    "id": "4_mrF4-l_5Sh"
   },
   "outputs": [],
   "source": [
    "l = calc_loss_dqn([exp, exp], prep, prep, net, net, 0.9)\n",
    "assert isinstance(l, torch.Tensor) and l.size() == ()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "XwNvArlkZoWY"
   },
   "source": [
    "## Loss calculation (solution)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "collapsed": true,
    "id": "cQF5tBouZqH0"
   },
   "outputs": [],
   "source": [
    "def calc_loss_dqn(batch, preprocessor, tgt_preprocessor, net, tgt_net, gamma, device=\"cpu\"):\n",
    "    observations, taken_commands, rewards, next_best_qs = unpack_batch(batch, tgt_preprocessor, tgt_net, device)\n",
    "\n",
    "    obs_t = preprocessor.encode_sequences(observations).to(device)\n",
    "    cmds_t = preprocessor.encode_commands(taken_commands).to(device)\n",
    "    q_values_t = net(obs_t, cmds_t)\n",
    "    tgt_q_t = torch.tensor(rewards) + gamma * torch.tensor(next_best_qs)\n",
    "    tgt_q_t = tgt_q_t.to(device)\n",
    "    return F.mse_loss(q_values_t.squeeze(-1), tgt_q_t)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "o4MlZBeJZLWm"
   },
   "source": [
    "## Fast batch unpack (solution)\n",
    "\n",
    "About 2x faster than naive version"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "collapsed": true,
    "id": "YvDDvGxvZKfw"
   },
   "outputs": [],
   "source": [
    "@torch.no_grad()\n",
    "def unpack_batch(batch: List[ptan.experience.Experience], preprocessor: Preprocessor,\n",
    "                      net: DQNModel, device=\"cpu\"):\n",
    "    \"\"\"\n",
    "    Convert batch to data needed for Bellman step\n",
    "    :param batch: list of ptan.Experience objects\n",
    "    :param preprocessor: emb.Preprocessor instance\n",
    "    :param net: network to be used for next state approximation\n",
    "    :param device: torch device\n",
    "    :return: tuple (list of observations, list of taken commands, list of rewards, list of best Qs for the next state)\n",
    "    \"\"\"\n",
    "    # calculate Qs for next states\n",
    "    observations, taken_commands, rewards, best_q = [], [], [], []\n",
    "    last_obs, last_commands, last_offsets = [], [], []\n",
    "    for exp in batch:\n",
    "        observations.append(exp.state['obs'])\n",
    "        taken_commands.append(exp.state['admissible_commands'][exp.action])\n",
    "        rewards.append(exp.reward)\n",
    "\n",
    "        # calculate best Q value for the next state\n",
    "        if exp.last_state is None:\n",
    "            # final state in the episode, Q=0\n",
    "            last_offsets.append(len(last_commands))\n",
    "        else:\n",
    "            last_obs.append(exp.last_state['obs'])\n",
    "            last_commands.extend(exp.last_state['admissible_commands'])\n",
    "            last_offsets.append(len(last_commands))\n",
    "\n",
    "    obs_t = preprocessor.encode_sequences(last_obs).to(device)\n",
    "    commands_t = preprocessor.encode_commands(last_commands).to(device)\n",
    "\n",
    "    prev_ofs = 0\n",
    "    obs_ofs = 0\n",
    "    for ofs in last_offsets:\n",
    "        if prev_ofs == ofs:\n",
    "            best_q.append(0.0)\n",
    "        else:\n",
    "            q_vals = net.q_values(obs_t[obs_ofs:obs_ofs+1], commands_t[prev_ofs:ofs])\n",
    "            best_q.append(max(q_vals))\n",
    "            obs_ofs += 1\n",
    "        prev_ofs = ofs\n",
    "    return observations, taken_commands, rewards, best_q\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "JUhqwAvwEX5j"
   },
   "source": [
    "## Training loop\n",
    "\n",
    "Ok, we're now fully prepared for training our agent, whee!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "collapsed": true,
    "id": "bIFbvnn1TKK0"
   },
   "outputs": [],
   "source": [
    "import time\n",
    "import datetime\n",
    "import itertools\n",
    "import torch.optim as optim\n",
    "import matplotlib.pylab as plt\n",
    "\n",
    "ENC_SIZE = 20\n",
    "EMB_SIZE = 20\n",
    "REPLAY_SIZE = 10000\n",
    "REPLAY_INITIAL = 100\n",
    "GAMMA = 0.9\n",
    "LEARNING_RATE = 5e-5\n",
    "SYNC_NETS = 100\n",
    "BATCH_SIZE = 64\n",
    "\n",
    "INITIAL_EPSILON = 1.0\n",
    "FINAL_EPSILON = 0.2\n",
    "STEPS_EPSILON = 1000\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "collapsed": true,
    "id": "g9HVFupZA6QF"
   },
   "outputs": [],
   "source": [
    "device = torch.device(\"cuda\")  # you should use GPU, CPU is way too slow\n",
    "env = make_env()\n",
    "env = TextWorldPreproc(env)\n",
    "\n",
    "prep = Preprocessor(dict_size=env.observation_space.vocab_size, emb_size=EMB_SIZE, \n",
    "                    num_sequences=env.num_fields, enc_output_size=ENC_SIZE)\n",
    "prep = prep.to(device)\n",
    "# we'll use target network to disentangle target predictions in Bellman\n",
    "tgt_prep = ptan.agent.TargetNet(prep)\n",
    "\n",
    "net = DQNModel(obs_size=env.num_fields*ENC_SIZE, cmd_size=ENC_SIZE)\n",
    "net = net.to(device)\n",
    "tgt_net = ptan.agent.TargetNet(net)\n",
    "\n",
    "agent = DQNAgent(net, prep, epsilon=INITIAL_EPSILON, device=device)\n",
    "exp_source = ptan.experience.ExperienceSourceFirstLast(env, agent, gamma=GAMMA, steps_count=1)\n",
    "buffer = ptan.experience.ExperienceReplayBuffer(exp_source, REPLAY_SIZE)\n",
    "\n",
    "optimizer = optim.RMSprop(itertools.chain(net.parameters(), prep.parameters()),\n",
    "                          lr=LEARNING_RATE, eps=1e-5)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 629
    },
    "colab_type": "code",
    "collapsed": false,
    "id": "mbmmqabCTV6s",
    "outputId": "efe946dc-d15e-4173-fab4-467d489d15d6"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "52: Done 1 episodes: reward = 4.00, steps = 50, speed = 86.28 steps/sec, epsilon = 1.00\n",
      "100: sync nets\n",
      "103: Done 2 episodes: reward = -1.00, steps = 50, speed = 42.70 steps/sec, epsilon = 0.90\n",
      "154: Done 3 episodes: reward = 2.00, steps = 50, speed = 4.79 steps/sec, epsilon = 0.85\n",
      "200: sync nets\n",
      "205: Done 4 episodes: reward = 3.00, steps = 50, speed = 4.66 steps/sec, epsilon = 0.80\n",
      "256: Done 5 episodes: reward = -1.00, steps = 50, speed = 4.54 steps/sec, epsilon = 0.74\n",
      "294: Done 6 episodes: reward = 6.00, steps = 37, speed = 4.63 steps/sec, epsilon = 0.71\n",
      "300: sync nets\n",
      "345: Done 7 episodes: reward = 3.00, steps = 50, speed = 4.61 steps/sec, epsilon = 0.66\n",
      "379: Done 8 episodes: reward = 6.00, steps = 33, speed = 4.52 steps/sec, epsilon = 0.62\n",
      "399: Done 9 episodes: reward = 6.00, steps = 19, speed = 4.35 steps/sec, epsilon = 0.60\n",
      "400: sync nets\n",
      "450: Done 10 episodes: reward = 1.00, steps = 50, speed = 4.40 steps/sec, epsilon = 0.55\n",
      "472: Done 11 episodes: reward = 6.00, steps = 21, speed = 4.35 steps/sec, epsilon = 0.53\n",
      "495: Done 12 episodes: reward = 6.00, steps = 22, speed = 4.38 steps/sec, epsilon = 0.51\n",
      "500: sync nets\n",
      "506: Done 13 episodes: reward = 6.00, steps = 10, speed = 4.05 steps/sec, epsilon = 0.49\n",
      "535: Done 14 episodes: reward = 6.00, steps = 28, speed = 4.37 steps/sec, epsilon = 0.47\n",
      "586: Done 15 episodes: reward = 0.00, steps = 50, speed = 4.29 steps/sec, epsilon = 0.42\n",
      "598: Done 16 episodes: reward = 6.00, steps = 11, speed = 4.15 steps/sec, epsilon = 0.40\n",
      "600: sync nets\n",
      "611: Done 17 episodes: reward = 6.00, steps = 12, speed = 4.15 steps/sec, epsilon = 0.39\n",
      "662: Done 18 episodes: reward = 0.00, steps = 50, speed = 4.29 steps/sec, epsilon = 0.34\n",
      "688: Done 19 episodes: reward = 6.00, steps = 25, speed = 4.23 steps/sec, epsilon = 0.31\n",
      "699: Done 20 episodes: reward = 6.00, steps = 10, speed = 4.26 steps/sec, epsilon = 0.30\n",
      "700: sync nets\n",
      "709: Done 21 episodes: reward = 6.00, steps = 9, speed = 4.10 steps/sec, epsilon = 0.29\n",
      "716: Done 22 episodes: reward = 6.00, steps = 6, speed = 4.01 steps/sec, epsilon = 0.29\n",
      "726: Done 23 episodes: reward = 6.00, steps = 9, speed = 4.19 steps/sec, epsilon = 0.28\n",
      "740: Done 24 episodes: reward = 6.00, steps = 13, speed = 4.10 steps/sec, epsilon = 0.26\n",
      "747: Done 25 episodes: reward = 6.00, steps = 6, speed = 4.04 steps/sec, epsilon = 0.25\n",
      "756: Done 26 episodes: reward = 6.00, steps = 8, speed = 3.91 steps/sec, epsilon = 0.24\n",
      "763: Done 27 episodes: reward = 6.00, steps = 6, speed = 3.98 steps/sec, epsilon = 0.24\n",
      "771: Done 28 episodes: reward = 6.00, steps = 7, speed = 3.98 steps/sec, epsilon = 0.23\n",
      "Environment has been solved in 0:02:33.585697, congrats!\n"
     ]
    }
   ],
   "source": [
    "steps_done = 0\n",
    "episodes_done = 0\n",
    "losses = []\n",
    "rewards = []\n",
    "prev_steps = 0\n",
    "start_ts = prev_ts = time.time()\n",
    "\n",
    "for _ in range(10000):\n",
    "    steps_done += 1\n",
    "    buffer.populate(1)\n",
    "    rewards_steps = exp_source.pop_rewards_steps()\n",
    "    if rewards_steps:\n",
    "        speed = (steps_done - prev_steps) / (time.time() - prev_ts)\n",
    "        prev_steps = steps_done\n",
    "        prev_ts = time.time()\n",
    "        for rw, steps in rewards_steps:\n",
    "            episodes_done += 1\n",
    "            print(\"%d: Done %d episodes: reward = %.2f, steps = %d, speed = %.2f steps/sec, epsilon = %.2f\" % (\n",
    "                steps_done, episodes_done, rw, steps, speed, agent.epsilon))\n",
    "            rewards.append(rw)\n",
    "        if rewards and np.mean(rewards[-10:]) == 6.0:\n",
    "            print(\"Environment has been solved in %s, congrats!\" % datetime.timedelta(seconds=time.time() - start_ts))\n",
    "            break\n",
    "    if len(buffer) < REPLAY_INITIAL:\n",
    "        continue\n",
    "        \n",
    "    batch = buffer.sample(BATCH_SIZE)\n",
    "    optimizer.zero_grad()\n",
    "    loss_t = calc_loss_dqn(batch, prep, tgt_prep.target_model,\n",
    "                           net, tgt_net.target_model, GAMMA, device=device)\n",
    "    loss_t.backward()\n",
    "    optimizer.step()\n",
    "    losses.append(loss_t.item())\n",
    "    \n",
    "    if steps_done % SYNC_NETS == 0:\n",
    "        tgt_prep.sync()\n",
    "        tgt_net.sync()\n",
    "        print(\"%d: sync nets\" % steps_done)\n",
    "        \n",
    "    agent.epsilon = max(FINAL_EPSILON, INITIAL_EPSILON - steps_done / STEPS_EPSILON)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 269
    },
    "colab_type": "code",
    "collapsed": false,
    "id": "maL3A_VlUJgh",
    "outputId": "f11ea368-655b-41ea-85a4-abfd169951fc"
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYIAAAD8CAYAAAB6paOMAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4zLCBo\ndHRwOi8vbWF0cGxvdGxpYi5vcmcvnQurowAAIABJREFUeJzsnXm8HUWZ93/POffe7BskbFlIgIAk\nrOESNkFElgAKKqKgMsgoqCMuwzgzcRnUMAiiIjqDCwriOwqIIhplCTsiS0hYQ4CQlZCQkJCE7Hc5\n5zzvH93VXV1d1V3d55x7zr23vp8P5J5eq7d66lnqeYiZ4XA4HI7+S6HRDXA4HA5HY3GCwOFwOPo5\nThA4HA5HP8cJAofD4ejnOEHgcDgc/RwnCBwOh6Of4wSBw+Fw9HOcIHA4HI5+jhMEDofD0c9psdmI\niGYA+DGAIoBfMfPVhu3OAfBHAEcy83wimgjgFQCL/E2eYubPJZ1r9OjRPHHiRKvGOxwOh8PjmWee\neZuZx+TZN1UQEFERwPUATgGwCsA8IprNzC8r2w0D8GUAc5VDLGXmw2wbNHHiRMyfP992c4fD4XAA\nIKLX8+5rYxqaDmAJMy9j5i4AtwE4W7PdFQC+B6Ajb2McDofD0fPYCIKxAN6Qfq/ylwUQ0TQA45n5\nLs3+k4joOSJ6lIiOz99Uh8PhcNQDKx9BEkRUAHAtgE9pVq8BMIGZNxDREQD+TERTmXmLcoxLAFwC\nABMmTKi2SQ6Hw+HIgI1GsBrAeOn3OH+ZYBiAgwA8QkQrABwNYDYRtTNzJzNvAABmfgbAUgD7qydg\n5huYuZ2Z28eMyeXrcDgcDkdObATBPACTiWgSEbUBOA/AbLGSmTcz82hmnsjMEwE8BeAsP2pojO9s\nBhHtA2AygGU1vwqHw+Fw5CbVNMTMJSK6FMAceOGjNzHzQiKaBWA+M89O2P0EALOIqBtABcDnmHlj\nLRrucDgcjtpAzVahrL29nV34qMPhcGSDiJ5h5vY8+7qZxQ5HP2P91k7MWbi20c1wNBFOEDgc/YwL\nbpyLz/7fM+joLje6KY4mwQkCh6OfsWLDdgBApcnMwo7G4QSBw9FPIVCjm+BoEpwgcDgcjn6OEwQO\nRz/DWYQcKk4QOBwORz/HCQKHo48w+4U3sWjtVuvtGU41cHhUnXTO4XA0B1+69TkAwIqrz7Ta3pmI\nHAKnETgc/RQnBxwCJwgcjn6GEADNll7G0TicIHA4+huu/3coOEHgcPRTnDxwCJwgcDj6Kc4y5BA4\nQeBw9FecIHD4OEHgcPRT3DwCh8AJAoejn+JMQw6BlSAgohlEtIiIlhDRzITtziEiJqJ2adnX/P0W\nEdFptWi0w+HIj9MEHCqpgsAvPn89gNMBTAFwPhFN0Ww3DMCXAcyVlk2BV+x+KoAZAH4qitk7HI5k\nNu/sxtfvXICdXbUtICM0AScOHAIbjWA6gCXMvIyZuwDcBuBszXZXAPgegA5p2dkAbmPmTmZeDmCJ\nfzyHw5HCjx9YjFvmrsRt81YGy6574DX8YM6imhzfTShzCGwEwVgAb0i/V/nLAohoGoDxzHxX1n39\n/S8hovlENH/9+vVWDXc4+jqiglhF6q+ve2Ax/vfhJTU5vhMDDkHVzmIiKgC4FsC/5T0GM9/AzO3M\n3D5mzJhqm+RwOCxwCoFDYJN9dDWA8dLvcf4ywTAABwF4hIgAYA8As4noLIt9HQ5HD+P6f4eKjUYw\nD8BkIppERG3wnL+zxUpm3szMo5l5IjNPBPAUgLOYeb6/3XlENICIJgGYDODpml+Fw+HIjIsecghS\nNQJmLhHRpQDmACgCuImZFxLRLADzmXl2wr4Lieh2AC8DKAH4AjPXNgTC4ejj1LrEPLuwIYeCVWEa\nZr4bwN3KsssN256o/L4SwJU52+dwOOqEkwMOgZtZ7HD0U5rJWfz08o3oKlUa3Yx+ixMEDkcfIM+c\ngGbxEbz85hZ89BdP4up7Xm10U/otThA4HH2ALHKgObr/kA3bOwEAi97a0uCW9F+cIHA4+gB5Ovdm\nMg0BANXcLe6wxQkCh6MPkM801Fw0i6mqP+IEgcPRB8jShQbRo1WqBBu3d+Frf1qAjm4XEd7bcYLA\n4egD5OnTqzUNXXPvq7j16ZX4y/PpyQIefnUdFr65uboTOuqG1TwCh8PR3DTCrFLJIEkuunkeAGDF\n1WfG1jnfQONxGoHD0QdohOO31udsNud1f8IJAoejD/L6hu04+dpH8fa2TuM2tep43Yi+9+MEgaOp\n2byzG/e+tLbRzWh61E79l48tw5J123D3gjXmfVyUjsPHCQJHU/PFW5/D5377DN58Z2ejm9LUqJ26\nEAx+anj9Pk0iBxKa6OghnCBwNDUrN2wHAHS6PDSJqJ26+JnUxzaJHHA0AU4QOBx9ALVTDzWChH2a\nRSVwNBwnCByOjKzZvBMTZ96Fucs2NLopAfFO3ftdT0durcWIk0uNwwkChyMjc5dtBADc8vTKBrck\nJJdGUKuTVylrnIug8VgJAiKaQUSLiGgJEc3UrP8cES0goueJ6B9ENMVfPpGIdvrLnyein9f6Ahz9\nA2fGSEa9PWKyV6KPwN1Sh0/qzGIiKgK4HsApAFYBmEdEs5n5ZWmzW5j55/72ZwG4FsAMf91SZj6s\nts12OBpPU3WkqrPYQiNw7mKHwEYjmA5gCTMvY+YuALcBOFvegJnlROJD4N4wR41JCoPsaRrVlGXr\ntxnXxcJH/X+TfARNJcgcDcVGEIwF8Ib0e5W/LAIRfYGIlgK4BsCXpFWTiOg5InqUiI6vqrUOR414\nY+MOTJx5F+5/+a1GN8Wak374qHb59s4SrpmzKLKMLeJHq5UDTpD0HWrmLGbm65l5XwD/CeCb/uI1\nACYw8+EALgNwCxENV/clokuIaD4RzV+/fn2tmuRwGHlxlZcJ80/PrrLavlKJ93rN0g9e98BruGVu\n1HHNSPcR1Irm0dXy8dunXrfKoNqXsREEqwGMl36P85eZuA3ABwGAmTuZeYP/9zMAlgLYX92BmW9g\n5nZmbh8zZoxt2x2OHuHJpRuwz9fvxrMrNzW6KVq2dWrqAfhSqtALZhYLGpXy4pt/fglfvu35hpy7\nWbARBPMATCaiSUTUBuA8ALPlDYhosvTzTACL/eVjfGcziGgfAJMBLKtFwx2OnuLviz0t9cmlzTNv\nQKZUjs+6DqKGEk1D1XW8teq4m0we9UtSo4aYuURElwKYA6AI4CZmXkhEswDMZ+bZAC4lopMBdAPY\nBOBCf/cTAMwiom4AFQCfY+aN9bgQR9+mkeGjzW76KCWYrZJnFtenPVnJUtfAUR+sCtMw890A7laW\nXS79/WXDfncAuKOaBjocAKDp63ocVRg1y9wGrSAQ4aM5o4bWbu7AoNYiRgxuNW5Tq1nLTXIb+zVu\nZrGjl1Db3iJLCKi6bTOFsgJAuRI3DdloBEkcfdWDmP7dBxK3qZVpSGgETiA0DicIHL2CZtAImpVS\nWacRCB9BgkaQ0pHbZnytVjC6R9t4nCBw9ArqNVrMclxTqudGo5qGmFmaUGam6ntaoxvQLCa2/owT\nBI5eQSMdisIWbtO5NoJuJWqIGUFjq7Vibenoru4AFjg50HicIHD0CurZWaR1ds3gEkhKL1FWNIIy\nszShrLp5BJt31F8QiOY7edA4nCBw9ApqrRGI7vHehWtxyLfvS+xoVe6wnI1cS0zpJYC4aahcYcs0\n1On3tEszR6EWvLOjK2xHk6oE67d2YntnKXW7zTu70dEdndSnm9vRzDhB4OgV1LuvWLZ+u3Gd6EuZ\ngeVvb8cji/w0KE3Sf6kaAVA7wdll4TBOUph+88QKXHlXmKh44sy78KVbn8Nhs+7HfQvXAmjeQIAj\nr3wAZ/7ksdTtDv3OffjIz58Ifv/p2VXY7xv3YOWGHfVsXk1xgsDRK2hU+gEAwbCawbGRny3MjIkz\n78L357xay5YBiI8+KxxqBEnyQKz7vydX4OFX12m3sREESXxr9kL88rHlkWWzX3gTADB3uZhb2qSS\nAMAKy878pdVhAua/+te3eN3WurSpHjhB4OgV1HvUaHv4vANtsd/1Dy/Nd4AEupXwUebwepI0A7Hm\nv/6yEBfdPE+7TZJpqFaPpFk1gryIy0nK89RsOEHgqDm3Pb0Sx1z1YE2PaWNHXrVpB+59aU1Nz6si\nd6xZtJR6Rj2ppqGIRpCwHzNHbPU6Orvrb+vuaykmxOPIIgf+58HFuODGufVpkAVWKSYcjizM/NMC\nAF5HU6tZuDajxjN+/Bi2dJSw4uozU7fN0yzm/BpBuY6dXUmZWczy/1M0guVve74R0/3oKptNYbUa\n79b61ry0ejOWv70dHzh0r9oe2BKbyXwqP7z/tXo1xwonCBx1g7mWoZfpvcWWjvQIjzwEzmLkH71W\n09nN+tvLietjE8oqMPoI7n1pbaRNG7Z7GsHooQO0x07yEdTONFTb+NH3/88/AKCBgsD7t9B7LEPO\nNOSoH7VU+RtpR5aFWd5rqqf5Q00xwZLRSjVf/fKxaBZ4kafI1GnZpJloVlP4C2+8AwDYuL2raqd3\nFsSzLhDh0dfW41ePxTPv7+wqB6GpqzY1PrrICQJH3ahl562rENZZKmNrD8x8lZFbkaVvr7Ugu+DG\nuUEEk2oaqnBonlDz0UX7bEZauHuiRlAj4VYvIXn29Y8DAKZdcT/+5XfP1OUcOuRaEBfe9DT++65X\nYtu0//f9mPqtOQCAU3/09x5rmwknCBx1o5YfuO5IH/nZkzj42/fV5vg2bWXO3fnVurN7bPHbQblN\n1VnMzMbZuvLonTkuRFRsJpRVqxH0hK/4gVf04bH1IHAWJ3hRtneFvpcdXflCkmuJEwSOulGLD1w4\n3HQd6YLVm6s/gU0bpFxDeUf2XAfLRNG356g+gooUPprmLNZNRpOxMamIU7yzoyuToBSbhkKr9hJB\np0nWHecjcDhCajEK5hyORLvOyP4rjfgIpI4lm2ko+71IG2m3CEGg8xEYcvzLo1TmUBCYRq+2qajf\n2LgDh826Hzf+Y3n6xgr1TDHRnaLxVIuu7aFpqPdIAitBQEQziGgRES0hopma9Z8jogVE9DwR/YOI\npkjrvubvt4iITqtl4x3NTTWC4KFX38Kfn1stHct+33qaGvIOMOthBw81Ak32UfG3KkEjpiE2agRi\nNGvrZF2xwQtDfXhRdhNMPZ+XrlaDzNrNHdrltpqE7v6FE8qsDtEUpIaP+sXnrwdwCoBVAOYR0Wxm\nlmPabmHmn/vbnwXgWgAzfIFwHoCpAPYC8AAR7c/MjTeK9TNee2srJu82tEdHKdVo5f9883wAwMRd\nBwPIPnmrkDHK3ebozPlHr7p5BJUKgyj/yLGlaNAIOG52Eahn0pW5BLyIlwqztWlIHKdYyG5kqGf6\nkDRBcLRh4qOt4NbdPtlZ3FuweWrTASxh5mXM3AXgNgBnyxsw8xbp5xCE39XZAG5j5k5mXg5giX88\nRw/y99fW49Qf/R1/eKZns2bWUuXPpBHkOH5SW8N5BPm7LN3h9/n63fjcb/NHsxTJ5CMIW5qYawjR\nTuvbsxdi4sy7gnWAXYoJBlD2O9zWHMPgeprxTaahrlIF7/7eQ8b9bCcA6gRGNu21gXHREjaCYCyA\nN6Tfq/xlEYjoC0S0FMA1AL6Ucd9LiGg+Ec1fv369bdsdliz1Uywv7CHnqqAWH3iSs9gEM3DL3JX4\n24tvWu+TFBxjmkdQixQTcxa+ZX0MFdPZGdKEMmWrWNRQWfgIgJufWBGsE+210wg4EEYFXxCUK4xT\nf2ROnR3dP/y7XGG8+c5Oq/1sMGkE67Z2YNUm83mSXAuvrNkS1LDQmoY0/hmTqalJ5EDtnMXMfD0z\n7wvgPwF8M+O+NzBzOzO3jxkzplZNcvQgcxauDeLaRWdTU7t4Jo2A8fU7F+DSW56z3sdmBMhcjY8g\n335JmJpckeoRJJ2XofcRcCR7abh+7eYObaEaOfpIOLC37OzGa28l13gQQkouXv/D+xbh2KsfMtru\ns6JWbwvOnXBf5ixci9vmrTSuP/3Hj+ETv/TyAuneG929N71fzZJnyUYQrAYwXvo9zl9m4jYAH8y5\nr6MOyCkS6sHzb7yDz/7fM/jOX6OpEGo7s9h8rHgcvc0RFXNKQo8p2/BzzyyugyRIziwa9OSR5ZHo\nIA47KPka2dCBHX3VgzhOY07xNAKvwy3mMA3JLXx8ydsAgLVbaiMIdD6QTdu78Jfnzd2Q7l1WEaHL\nuucaCrZwnUkzaZbMqzaCYB6AyUQ0iYja4Dl/Z8sbENFk6eeZABb7f88GcB4RDSCiSQAmA3i6+mY7\nsiA+8noNPrb5OX5WbvQTmPnLa3E+UxikzLv+657IR2c5NyxCWjw9IEwu2c6TZ1tbTIJAzj4acxZH\n5UBqURt1QL1NU7FLDkP924trcMGNczMNOuR7OqClCADozFn3QUVXKexLtz2HH9yXL8mbatPXvTa6\ne2+auNdrNAJmLgG4FMAcAK8AuJ2ZFxLRLD9CCAAuJaKFRPQ8gMsAXOjvuxDA7QBeBnAvgC+4iKGe\nR3z8Jpt2pcL41K+fxpNLN+Q6vhgEqu96T2kE3WWOODVtbPfqB2ztHEwxmb+1pQPfnr1QWyym1pii\nmOSoIXW9GsmiG6nK9yZJk5GneMgj78cWv40P+InfbJDP0dbidUmdpQre2LgDr64N41CuuvsVfPFW\ne3MfAHRo0mivqcLsFKsPnaQRSO+hzUCjkVj5CJj5bmben5n3ZeYr/WWXM/Ns/+8vM/NUZj6Mmd/r\nCwCx75X+fgcw8z31uQxHEmkj9E07uvDIovX4wi3P5ju+37uonWkt3/20Q8kTn6w0ggymIfm4aQLj\na39agJufWIHHFaGatt9zKzdh4sy7Ih2fTXvUojSAEjWkrItNKEuYECX/3Vkyj99kjUCwOoPDV95z\ngC8IukoVvOf7D2PGdY+BmbGts4Rf/H1ZUP3Llp0azaKa+H71NdHdl1AIh8t0z8k7XnMICDezuB8Q\nRt7o11f7KooPS83DXku7eFqYnVxAxeasMdNQUvioYWaxDvFhq9ultf/+l73oofszRBFVmLXOUM+E\nJbbx/i2VK7FOy3MWe/vLHbk8uhf35ZCEnE4MNs5HsEHedUBrqBGI5a9v2IEdXflSjOsFQX5JIHfc\nTy/fiHd/72HjNmo0lI7Hl+TTwmuNq0fQDwjfe/3LKF7YvJ9HUQoZ1B1XZltnCaVyBSMHt1kdO02I\nCeRawjax2epIzM4EwhGBodtD3MPY8ZWN1RHzkAHep7g9QwIyhj4qhllKQ+2346z/fRwvr9mC4yeP\nlrYLO335usrS6FXcl6RUExUGyhbJ6UzIz6utKARBeB9WbNiOfccMzXXsnV1hJNtLqzcHtQryIr/j\nJlOq2EJ+B0w+gov/3/yq2lMrnEbQDwiSphk1AjGSz3l8pbM2dYYAMP3KB3DYrPsznyOtb4+YhjIf\n3TzD1ju3va23YBBc6r047upo9M1QIQg0zlgTS9dtw/HXxEeksu9AnPblNXGTEyPs6OXrkidhWQ30\nuTqNQPY1CGexLBBLZY4IegD4wu+exc2PL089tthP1AawoSXBdiQLTDGzWyX0EUj79QUfQW/lmdc3\nptZk7Qn+78kVmDjzLmNMc70JnMWp72I+SaCahgQ6QZA35W6aLVUeQaZl+ly5YQf++kK0tnHShxqs\n4vR2mCbApTmZh+QQBD9+cDG2aqqyRbKPpojFkkYQyA5kq/kVyNfR6Sa9CWexXKylVKnETDx3LViD\nb6eEeALAV37/PAC7N/uKv72MSoUxYlCrcRtZczSGygZmOUm4pqS6aDR9WhCc87Mncf4vG1cQWvD9\nOYsAZPvIa0naR1Ctv0p1FocTyqLbbdyeXyirTVQ7no5u+6ihM3/yGB54JWqLT4zJj9h69csFBYPQ\nTRMgrf7oUheeacI065ch1SNQThudL8BBxyZ3cPKAJdFkFhwnWaNKQ+z6zOub8PfF3qh9y87wPpQq\nHHm+eQZUXk6n5G1u/MdyLHpra7IgkC6zaDigrvSm0wgahBidvqJRiXsakw29pzF1kEEq4pymodBB\nqpxP6oW2dnRj2hXZTUK6YwHxTjCiEaTc5q2azjapb5FV/TRnMRm0o7Q2iWewPYNT1DiPoBKeMCnp\nHEPvI5A7Wqv5FWzOYmqDfBmvb/A0gZ3SfVBNQ5tyDCiIyMpJvLWjhKEDza7TsoVGYPIRLFu/LVfb\ne4I+LAga3YKQRguCNNNQkHgs5/GDUWWCg7TawvJq2+OCoDofQbJGEI6a00wltj4CFfFubOu0N52Z\no8DkCmXm817063nBeeURvfy3zSsrC5Q86O7Nn58Pw0S7y5WIIHh7m9eZthps9DoIdu/3lp2aFBqa\ncFrAwkcgXVapzDjph4/ifdfa5V/qafps1FATyYGgc+hulCCQKmzpqLZ2h2qG8M7H0QRtVUpmtbPY\notQq7ujKFjWkYuMjqLC6XXyfgsFHkCZAxHE7u8vYvLMb67d2WrRaf0w5rj9uGtKfVxasEdOQVQQW\ngjDUelCqMDqk9j27chMAYJRl5BlgZxoCvPdKN6ARfb78/E0ahrgVUY3A+7sa82g96bMaQbNM1ABC\njaDbssiHiR1dJfzPg4u10+aT0I1QZFTbflbE8eMTytKfwapNO/DM65sszhH9vVkZucnmnjxPPkkQ\nyMnR0p3F+jbIwkmX50bcu+5yBefd8BROthg5GqPAWJ/vBoiPinXXLTuLSxXGtfctSmlHlfMIUvYt\nKRrBN//8EgBgeIItX6VAlFhDWLBlZ3dsYFSq6AWjKbooiNiSltlaA/IU9qkFfVYQ1FIOrNvSgavu\neSW3aScQBFVGDV1732v44f2vRdRmG8rBi5niI8hpHAp8BMqEBPkZmJ7Hu7/3MM752ROp51A7NFWF\nlx3xeQYBSfuEGkG6LVyMEpNy0nz5tudj+4njdpc58GulXYZNrqEkZzGgN+nIHd/zKzfhJw8t0Z4n\nElZbRVRM2p7dZdbmHkoK81SpMFsNdDbu0GgEFf3fJh+BuKXL1m8PltkO3n78wOL0jepAnxUEtdQI\n/uOOF/GLR5dh7vJ8swDFC5NU5MMGEVudNN1fhymCRBDOCM7XLvFxZI2UyYJ6KFUj2Cb7IHKcNtk0\n5K3r6C5j+dvbjdsBUt4lZmzY1qmNytEhRuGyiSbtfUmaKV4OhDPwr78PBY/a8etMVrJGULDobKuP\nGkq5N5WKNmdQhRnrlCylSWZBG2fxTx5cHGuPLBhl7dnkIxADru/d+6p0DLv706jyln3WR1BLxMeZ\n1wwqwswaFUtcCezFBo2gRvZ7tTONJC+r8hyqNhMTBAmmoXVbO7BpezcO2GOY8fhJ90Csun3+Ku1y\nGdHZvL21C0f89wMAgPOnj8dZh8bqMUUICsFInf/OlDkXJuFSkcJCGYw7pdrP6shUN5KXO63WYvpY\n0VTXwIYl67bhppSC96UKa2cuv/bWNkz/7oNYftUZqTPQdx0ywHqgo15LZF5FoD0n+Ag0bbC9P40q\neN9nNYJamoaqfTaFGpmG8hI4DlPW544aUkxDupnF1UZMZfIRKNsee9VDOO26vycf33JmcSr+xW+V\nnNm3Pv1G6jFE59tVqgSTqtLy65hGmbKP4DrF1KDuoztG1veUOf9EwY/8/InUiLKSkl1WJZonSb9d\nucLW77f6ruqc5+TXdNahe9a297RRGkGfFQRN5SwWGkGVzuK8pDmLA9N+Tokn9k9KtKZ+XLbRE6Z6\nBOqMWtk0pGoPorNbt6UDl//lJe15kucRWDUVQDhKVO9l2jGC6J1yBQMDQZDcuZo6PWZzmKtNeuyI\ns9iiA6twPIrLFt3M6Hh7KokDiW7NiF1l7ZYOq5nI6vGAaGiyPNjR3X5mDsJbZWzTW+T101VLnxUE\nzSMGQo2gWh9BXtJG49WO1sX+Qa4hzcxitWOynVwm9orbbaO/tydoBIJv/3Uh/t+Tr2vXJTuLDSM/\n5fe2zlLQDrWTtp1H0F2uYECrl28nzTRkMjUyzM7bpNGuQG67qbJWR3c5ksZC1dBsYGbj7FyZnzy0\nBH9+zlxRTP6uqvFV6I4HAJ+8McxOIN8/nbB98BV91M/8FemRcY2kz/oI6qURbOnoxrABLZlGzy2F\n/D6Cjdu7sPDNzTh+cv5azmHUUPL6vMSihsRy+aOxEEa6KIwwHt6sbQBR01DijFsDpUoFTy/fiBVv\nb8dHjxwfWWd7ew7+9pxgW7UDTbvHpeA6w/clSyZSmUrFHM6pLtcJgi6p7brBy7otHZj+3QeD38z6\niVg22H5GbyYUk5G1llpo3epkRTHbGQjfISJohdP6bfr5H4ve2mp38mY2DRHRDCJaRERLiGimZv1l\nRPQyEb1IRA8S0d7SujIRPe//N1vdt17UQw6s29qBQ759H376yNJM+9n4CN40FPK46NdP44Ibn45l\nX8xCmrNYrJc/ypUbdljnvZHDK4FQvY1oBGlRM/4XdsvclZHMnEEYZOyc0SUR05DhVIWEt71cAT76\niyfxH3e8GFtn6yOQN1OFftoxZKEpri1vDn6GWRiqAko3OJE7Vl366XWayW7vSEXt9xwx0Kqd67d1\nJqa3tkW+hloEZOi+0yBLazDfg/GEJg31qMH2cxuaiVRBQERFANcDOB3AFADnE9EUZbPnALQz8yEA\n/gjgGmndTr9y2WHMfBZ6iGpnssqIjk2UuLv3pbWZ9hdRZiZB8KdnV+HYqx/CvBUbY+sWr9sGQIzk\n8l2TOK1pb9WkAwAnfP9hnH/DU1bHD1IwsH6514YUQeB/wF+/c0EkV7/o0DZs64qEbqqHM+XokduQ\nFD4od8S/n7cyKBSjO5cNMdNQSn8nj9RFk6vJ1GrWCKIN0SVCNJmDBGokEXPUNGRbO+DuBdm+IxPy\nd1WLgAzdMbo1BXx0dFUpiBqkEFhpBNMBLGHmZczcBeA2AGfLGzDzw8ws9KenAIyrbTOzUw+NIG+8\nfTCPwDD6eXq5JwCW+J2+TFhmkqVl2RoQmIZM4aOGl3vB6s2Zjp+Ua8hWEKiIY/74wcV47w8eMR7P\npBHI6YuNaYMRNd385x0LIgVD8pgZ1c4k3UcQfzd25M1Wy/Y+Ap2zNs3Orl5bVymaJtoUX18v5Pbo\nTFlZI3F0WoXJRKmim/imotMaAO96AAAgAElEQVTuDx47wrJ19cFGEIwF8Ib0e5W/zMSnAci1iQcS\n0XwieoqIPpijjbmoh4dAmThrTSFlHoH48HSOM+GLkPe0Kc4uoys+ElmvmHSyEpiGFBNTWvio/FGZ\nw/7050zyEcj3Z61kW07TCFLSy2dCfta7DRtgETUU/h2Ej+Y0B1bY3Jmr7+Dmnd0Y0laMLDM9C4Ha\n2arhny1JNrg6sGjtVkyceRdefnOLdjRvWw0vie4y47oHXsOvH1+RuJ1NQMi//zFufrSZuFdPavrE\niOiTANoBfF9avDcztwP4OIDriGhfzX6X+MJi/vr1dmFWadTDWRwcMaNKkJZiQnSSuhFrqBHIv7Ih\nRrupgiDnu2gyDUUEgeZ5yKPIzTu7tTZxo19DWdxlKF7/5jt2gqDMbExiZmpDd7lifKbCzj5sQAuG\nDGhJHUnKGoF9ISE9aklNGbW92zpL2HXoAGWbFPNHKdm8lCUraC342wKvyNBfXliN7lK87YePH1n1\nOcoVxnUPLMZdC9YkbmfS+mX++kI8RUwP37IYNoJgNQA5jGKcvywCEZ0M4BsAzmLmwJvEzKv9f5cB\neATA4eq+zHwDM7czc/uYMfmjY6LHzL7PW1s6jE5bIOzYsgrvNEEgRm86lToYXddgCr9plFj9ZC9x\nfO/6osLLfI4pl88J/j7ph49Gfgf7GQWBuc3yutXvhBEfSZNkyxU25qE3DZAfW/w2TvuRfqKayDTb\n2lLwk9WZz62mt642lrzC5meqG7GOHhoVgGlzB3TCRKbFYjZyLQmDIfTXd1ANzC62uYLyOr9FH9Gg\nicVWgmAegMlENImI2gCcByAS/UNEhwP4BTwhsE5aPoqIBvh/jwZwHAC7WR1VksdZfNR3H8SxSi1Z\nIB4XT/A+hudW2sUGi4dseknEaFCrERjSGsePwcYKaPamoXyIb0S189diZrE5jYJ5H3mVrMon+Qgq\nbJ55mnTvlxlyD4mOo6XgzUBNCh8tVaIpGqrtDJKS4+lGrKpGkNVHcI8SPNHTGkFZer91g62kimMm\nTjxgDCbuOjj4nZZCfpg/iLDRCHTY5EGqJ6mCgJlLAC4FMAfAKwBuZ+aFRDSLiEQU0PcBDAXwByVM\n9EAA84noBQAPA7iamXtGENTloMKEQvjePa/iQz99AovWpscHi4ds+jjFcl02RfF+pMWhX/6XlzD1\nW3O0IxdTR62uD/wRGYWoSeOoRa4h0/eXnBIi/FvEbxcoxTSUdLzkJmoR5pXWYgGVSvI97S5XIs+m\n2i4hKROobjCiagRpkTddGvOLTFsPawThhEa9IGhtyd6eSaOH4JaLjw7PkXBPzz1iXPDt5hUEgUbQ\noLghqwllzHw3gLuVZZdLf59s2O8JAAdX08C81CNqSNYIXnrTi6jZsL0TgDmZmdeW5BG56ASKhQI6\nustoKxYC55HOzKLjD35CtDJz7KGGpRbtNIKsg/dYznuNFpMWkmgiKdWySltLwf8Qw3VB+gtOiRqy\nKFWZBdEhtLV4zzTpGBu2dVU9qU8mydmrewdHqxpBmo8gRVD0dNRQGBWnF2I2s5dVCkSB0x4Iw0d1\ntBQp+GayZgaWz9dI+myKifo4i0Onqnz4bZ0lfPrmeVizWe9fUCdcqcgj6Xf9172Y9bdQaaIUbSLW\nRs1magqIWPtiI/msGoGpLSxtk+95mGLpdeccOqDFP290uRihmtJLiPaZWpin6V2KaSgpEGftlo7I\n8632zc0ay66aTtI0grTJjT0dNSSbhnTaSp6AnAIhIgiSvr9iIUxAl9s01At8BL2SepiGwvDR6NP6\n6wtv4sFX1+G6+/VFJUxpmk3rb5u3Mliny+SpJSHSJHUegRIXm9WeH+nEJFt7dB5BpkMaSZolLQSB\n2vy2FNNAS4HQWSpHZsdGzmkhCW57emXkt+gQWosFlCvJx1izeWc0h02VzvusFeyGDIjqkLqooX1G\nDwn+ThMEjfIRVFifpTRPaGahQBETV5JwbCkUwAx88dbn8JsnX89UMEfQG6KGeiXVRNmYCI6oPDTx\nIppeuGDEkhLSpxsNhFFDyW1LEhjpzuLoMbKOgOVOudNQ+zYtNt2WMjN++sgSPPhqPLmX6NDufzmb\n83JQaxGPL9lgTJxmcz9m/mlB5Le49taWAt7e1olZCZkv12zuqLEgSN5f9QkMVuYR6Ca3ye9muiCw\n71YuOWEfPD7zJEzda7j1PiqyxqvLNZTbNFS00wiE1ifCQgfk8EkUe1iLUumzgsCWrlIFb2zckb4h\n9NE1hFAtNL3/wYua4izWp1y2ixrSTeKKnb+GpqFN27uw2R9By7t3dleCGySXZKyVqa5cYVxz7yJt\npz3MFwQ/uO+1yPI0jWCg0hGqVOMjaPUHB1sTZgk/sXRDxDxYbQbNJHs2EPeVDGqNXr8uQka2Ye/s\nih9fPkYWQUAEjB05CNMn7aJdbzO4nu/XvGaDszjJN2SiQNGBXdLcCtk0BOQLIe1h/3qMPisIbD/e\nr9+5AMdf83CkkIjMPQvW4LHFbwMIIwcKRNrC1KaRh2jKLx9bjnVb41kUQxtnfN9o1FD6Nen6ENsU\nE7ahqgBw+BX349BZ98W232kYLeZ1FqskjcxM8wDSOia1I1SpxkeQdu537TEM81dsjCadq7NGoL6n\ng9SZxdo0DeE+HRqH6KUn7Rf8ncU0JNpi+nZsOvG0qKFcpiG/PVd+6CAAyeY2r0hN+DuPIM8jrGpJ\nnxUEth/v3/2CEds79R3Y53/3bPC3eMBPLtsQydGSahqSGvPlW82Fy3XmE9tcQ2KZrrOvpGgEZUXT\nyfoey52zyWxQK40g6SNTbd2CtHDGga3J67Om9ABCE0Va6OKIQa3Y0VWuaU79NGevOmJV749OkMjv\ntu4Zyx1ZlgllosM1dYRZomnKFb2jPE8fKwZFIoFeUqSUFzxS3TMLCxpVdZj852/MaeuP7WMRD8DG\nhi1/YK+s2RL8HZiGDE9R7ijf0Zg0StKIBoi2PZyDEC774zNvYNoV90dGjrpiMGH7ou1UUZPpqSPS\nNJuwfNiOUlkbCV0rZ3GiRmAQBHk1gt88sQId3WYnchKBRpDSCwn7vDw7N0+RF5m0FBEblOpw6gBG\nVzpSvoVaQSC9+1mcpWJT0yAqy0jZaBrK0buKfYR2k5QJtkDeNzZ25CAAwK/+qT3z+Vz4aJ2wHYGm\nzfqV0b1kRGEnZ3ppo/H0uglfUdNQV6mC/b5+d3B89RjPrnwHG7d3aW3BWh+BIReQ4D/viDo65WM8\n/8Y7eNd/3YuHNc5Z3fZqVa1HX1uPiTPvwooN+hm4WUkSBMMMpqE0H8EAgyD41uyFmHbF/dq882l0\nluxMQ4PbvDbnLeyiI2vUUIGAfzpm7+C3Lgw66iNI1gjS7nfkuCkdfZYOssKsdRYTET57wj7Wx/HO\n6/0rnLhJg6ECEZgZrUXCWYfuhaP33TXTubzzNHZCWZ8VBElyYP3WThw26z4sfHNzUKwkrSwgkJ4q\nOS1qCNCr/aVAEMTNA+KIaZEkSVFDomNIU191BWVEKur7X3lLt0ts+47uSqR625+e9Sa6zV0er7WQ\nh3waQXrUkIm8NQHkqKEkhEZgU7vX+twZTUsFInzg0L2C32s11cAiPoLuZIdsFmdxWkefxaxTZlNV\nN8Zlp+5vfyCE37LQbpIEgfAR7OwuY1BrMVdX7jSCOpHU6T2yaB3e2dGNG/+xPFABbapxmT6wNGex\n3HnptIpkZ7HswI0eX3eJumXC/2EbeSTfO5GiOCk3vnxc1ZaaVoshK0kmPDUMUpDWMWXpuGwJ5xEk\nf+ATdvHy2dRSEGTXCCjSEammIyDa0euKAEV8BAm9927DorOYw5GwniymoUqF8awm/xdz9pG2uB3i\n3UgaKIom7ugqY1BbMdapi5xFrUXClD31YbLiFXQ+giopVxhrNu8MOvSkLk/O7SMkv5yw7R9+lJCK\n7gP777teTnUWp6VaEJ1bUpoBve1f9hGYI35E6KLtQFFuh3DAbjM404GoT0GNQRfCsZrKUXLce1JE\nzEDDyD7NVGEjB7JOEhK3JMlR/eHDx+KwCV6K5J3d5ZpNxMp6r4sFSh15y+t1QstGI7jl4qPwRSm6\nCAg7PlMHmEUQdJYqWLBqcyRZnCCrw1h1Yu/UaEHqtkIQyNdyxN6j8P5DPG3riydNNj5jFzVUIzbt\n6MIxVz2EO31TRNLoV5hd5ix8C8vWe7ZrWSP45I1ztfvpPrCXVm8JClabNIJoaJmcN59xy9yV2LDN\nG4GVtaFv3r/6wi7J5xJs6xDx/rpjaKKMpEWiczVlNlW3L5U58iGIv6vRCHYbFtbATYreMJl40qKG\nbD7CvFqDnHdHHQ0fPG5E4CMAapeaIeu9JgrvwaDWIu758vE4Z1q0yOCuQ8K2b9GEWkecxYbOjkAR\ns6G8n2nEnsVk0lkqY2d3WZttVD2vjK46mOosNoVFe230/i1X2DMNUXxd8LsGTvF60GcEgXhwopNP\nsoKITlWOzrAyDZmydwZJ4/T7VSKmofDvF1dtxtfvXBDYoXX+AzksdKlSyjKiEWjOJRDXplsXTX9M\nse1sCqnbpJuuRiMYKRUE70wYmanx8II0jcCms8mbSE0IkGkTRuL2zx4TWccMDBkgT8SqTWeQNerI\n0wi8cw8ZUMSBew7HDz96KL7w3rCG1GWn7o8jJ44CAGzZmU8jIIrf63Qfgf09EYMVnfM/qZ/Vp3+P\nrjPNM/K2Dfcf3FaMCDVVAMma5YcODws95olsqiV9RxD4H9GLqzZj4sy78Npb5vTQug53R4LpI9wv\nubCMzTwC2bykhpLqRuzi/Xhs8dt4Wilur+tzdQJQ1PMNwkgrXtm99Vs7lYIo8WMIoZAkKCOmr0o0\nr79YVStB0FU2PyfTfIC00bzNaCxNq5Bz8ej2a2spaM8zuDXUCPJqHTd9KhquuCljuKvqIxD8+2nv\nCv4e1FrERcdNAqDvFKOCwDy6V29BLU1DwhemmggZyRqBzuxXCDQC75kklaiU791Rk3aNXGOBpGSV\nyrZXffhg7DrEM3smta8n6DOCQDzMO5/ziqc98Io53FGXSyUtVh4wawTCXGHlLJb+Vh1QWmex/+/i\ndRrBxvENy8y4/uEluP7hJcG5RSSF6LCfXrER1z2wGN+4c4F2BK8b4ZuiZ555fRP+56Elwe+fPLg4\nIqDEn3krNwHR7JhJGoHJtJLqI7D4CNPyx5gmgQlNoq2lqO3Udhs+ILZtFkYNbsVJ79o9suydHXFn\nbxIFosAEadKk5dG87lnm1QjSnMVZrGVisDJQeVZpkeS6AVwYPpr+TGTt4eBxIyKduul6zz5sr4jA\ncqahGpF1BqKKbtq8iikao7NbdLT6/WRTi3wMVfjok30J05DmuDrTEDO+P2cRvj9nEYBohAezZ2L6\nrz+/5LW7VIl0YOIWrtsaVBoNzrFeWiZzzs+eiPxevG6b1jRRjY9guCwIEo5jegVqohGkCQLDuyHO\n3Vqg2HkY3uj13CM8e3weH8Ftl3jmJvnYaRrB7z5zVOR3gdKFYVFpv1rwPho1ZBAEiD+jQopKkMVk\nIt51U9CAjJzkTncOcT02cyLEpQtNKKoREI6f7JXfPXa/XYPjqqUpw9+NEQh9RhBkierQfbS6RFqx\n/Qw9veicTA5q2fwiH0J1QCVpBNqsopqoIdX5u02K8KgwY9OObiz2fQ1e9ay4IPjoL56MtalU4VzT\n6MUu1WgEA1rCDztREBjGlWmj+Vo4i02hxWK/QoFi76i4nyIlQx4fwQF7eEWR5GvctKMrMW3G8IFR\nZ2qxEDpxTU+4QBTxgakOWXkgJl/H37747uBvrY8gVSPQrznj4D1iy4SPIC13FAB84NC9cOg4z0ms\nrxXuLRvall67SzUjyZ05EXD0PrtiyZWn44i9dwkFQUZfSb2xEgRENIOIFhHREiKaqVl/GRG9TEQv\nEtGDRLS3tO5CIlrs/3dhLRsvk0W10nXoqkbwu7mvx7YxjWpFVSJzdlF9O2KmIV1H61+Wds6A5phq\nE4RJZ0hbERWO1jVua6HUvDayX2SdQSuoN3KahqQKUGaNwPxutBbJKinZgJR8RCYnuTh3kcg4uUxs\nkyRsfv7JI3Dpe/czrpcFwdaOEkYMasWJB4zBx9rHx7ZVvxWvk0++B15kUXiO4YogkIWcfB1y3DxJ\nJqjw3Imn1XaQx+yza+B7ad97VLBcmG7VoAFdrii5vbprF+e1eTdExy8fM1R0/HV+e4Mke8pxmz77\nKBEVAVwP4HQAUwCcT0RTlM2eA9DOzIcA+COAa/x9dwHwLQBHAZgO4FtENAp1gDSOKBO60bVqpvnG\nnS/FtknTCEzzAEwj6bhGYE46lz6PIL4MCK9r8IAWVDgaA65qBDpfuHy8t7dlFwS6zvnH5x2W6Rhy\nErNam4ZemTXDyvyQ5izWZr2URsBFpdCJjOgUkpK1zThoD+zu+xOGa1JpqCaMIW0tuPmi6fjuh+OV\nYmOCQJpHYHpXC0SR+6Sm84gmnZNs5HLnCI3NXISPmpzFhhXiOLuPGBhbpwpt3SWlTYDLYrIX2pfG\nZWcMH1UFTBhG2xhs5NB0AEuYeRkzdwG4DcDZ8gbM/DAzi6T+TwEQQcinAbifmTcy8yYA9wOYUZum\nx5Ftk0k3VNeh65yQ6gjfFPkiptwbNQLDx9VpYRoSL7zuA9X6DSryeg6EzZC2Ipg5EvHRWixE7oW+\nloH+2LbojnlMxlws8oea5GswmYaS7LwtRX00j0qaaUg30U0OyywUyHiM1oymofdL6SAEsvlMnFv8\ne8vFRynrovvKAsukIBY0o3nd+YAkZ3F8HkGQddM0j8D0bIRtXSMoBrakm4bk9uq0jiymGmGKkr//\n4LkbBJ8qfPKkyq4lNoJgLIA3pN+r/GUmPg3gnpz7VoX8oqrv89rNHfjWX15CqVzRqvG6qKGv/vGF\nyG/TrNadvpPK1OHrJ4NxbLbu76USlQLTSB+Iju51voSd3eXA/DS4zdMI5DBQr4xiuL1esMihofaS\nQLzoukirAcX0DzVyLKmDrIezuNp5BOu2dGgnHMkzdosUH4mL+y2W24aP6lqi+kHka4pHrkS3LVK6\neayoaARqumebFBOewIkuS7v1ulsi+xoKBPz10ndj9NAw+krVCMbvEp9pbNJggrZm6JgDQSDdEnFd\n6mGKgUNZrND7DHqamlqmiOiTANoBfD/jfpcQ0Xwimr9+/frc50+KuvjGnQvwmydfxz+WvK3t0HVR\nQ396dnXkt0kj2NGljxpasm4bFr+1VaspTPra3bjp8eWRZWq0h1f/19cINOc9+dpHY8vkznxbRynU\nCAYUUa5wRHgUKCqkdIIs6ui2dxaLj0vVeoDwY7A/lmwaSvARGJanO4vT25BUSnD6dx/U7yPZ3nUd\niyikI3wgtgEPuj5D1XpUk4zarujxwk7eZBqiQrTzVN9pWdiYo4bi8xXStDGTkA5ML37I5vGTRwfr\nRMd84gFj8MBlJ+Cw8SNj+8v3QPdsM5mG2oQgiAdvmGZS68xzjcRGEKwGIHucxvnLIhDRyQC+AeAs\nZu7Msi8z38DM7czcPmbMGNu2x0h6qeQC8fp5BPmjhsSoW/04Tr72UZzyo78n5hBKYv22TizyJ8aZ\nVPawloBQ7cMNZ7/wZugjaGtBRTENdSnho0llLr2/vX9feOOd1La3+h+XLiVE1tGP7Cy+5t5Fxu1M\noXepGoHFR5inIHlBisZRr3nfMUPwUd+RKzoiW3OE7nXaU7GVJ86kLcY747TI1YKiNZzbPi4yMzbq\nF9AfI2lmsbxYnkBouicF5b7K374IHyUA++02TLt/uo8gu0YQmaUfHEd/XnP4qPVpa4qNIJgHYDIR\nTSKiNgDnAZgtb0BEhwP4BTwhIM/kmgPgVCIa5TuJT/WX1QX54aojG/GxlSus7dBt0lBv1GRlBEKn\nr7E4vNIX2hYn/9ffh9XMTCM11VQid+b3vrQ2uK6hA1rAHK2d21kKzWQtBQrmGZiOV64wXlmzBWdf\n/3hq20ONwJw/yRaTgJfV71suPsrY+dmYPdKwkQNqO1sKFPvQBZ86bpLkJLbsBBJCPH947mH4t1PC\nVMvy+UyjUoHsIzCHj0Y7zJGD2/Cjjx0mrZdH2OYLMTlP5cWff0+Y2mLUYF3eoHgHKrdNRBKdP32C\nsR3FQlhuNinFBAA8cNl7jMcB9IJAfJcmwReLGmp20xAzlwBcCq8DfwXA7cy8kIhmEdFZ/mbfBzAU\nwB+I6Hkimu3vuxHAFfCEyTwAs/xldUG+uar5RwwKPY3AzjRkS1hhTL9eHWnr0vjqEMnodMcQiARg\nFGyHILpkt+EDAiE12A8f3dpRQkuBsP/uQ9ElCYLWYsGv+Ro9T1nRGEwTy1SEOacmGoFhRC8+qvcd\nuDuO3Xd05siTWqM6e72i5t7fqjCSf7ZaCoKk1SMGt+Kfjp0YbptwMJ1ZIu0eqWko1AioSDim4Vht\nLQVNxyjaG91O8PUzDsTFx0+KHUs1vcj3d/wug7Hi6jNx6tT4XIOgjSntldfvt9tQ43GAMFxV942q\nkXbiPsXNc/6/iWeqH1ZjM2a+m5n3Z+Z9mflKf9nlzCw6/JOZeXdmPsz/7yxp35uYeT//v1/X5zI8\nItElhrz4ZU1nBySnLrDFZkIZYJ97Xj6eybq0TTkWcyjoKhVIPgLPNLSto4RhA1swoKWIzlI51AiK\nhApzzCEsC4J7XlpjrboKc44a5XP+9AnGEeMVZ0/Fo/9+Ymy5yVEbP4x+uzQfgN3zSL9wVWAVCxSY\nC9VokegI2p+IVGU3IN+PJDeMbh5B0J6EqKFoFbK40DMdXzBsYEtMGywqHbp37HCjIQNacOl7J8eO\nFWoE3r/ivhKl+4S87cNtdBpjWriwjM5ZHLRTeaam8FGXa6iGyB+X6tgVH9vOrjIefS3ukE5KM2uL\nyeSjduKiLOFlpyRXTVJH4zpEJxZGF0kVz5jR0VX2Po7Wgj+PoBvDBrairaWArnIlEFJtxQIqlXiU\nj3ze3z61Eis27IANgUYgCYJpE0biqg8fbHzpxwwbqLXNmpyP7z1gN5x16F749llTAZhH1PIx1VTQ\ngL6OdB7UDqhIFBYtUmzD8oiwVn5CIn1nrN6XmN2aCJTqI4ges02J/NKZhg5VnLRDB7TEnr3uXZAr\nzRWI0KoIHc/pHLZdPufAlqJVp1oshKJd5yMwTf674YIjAiFx+ISROGz8yMSUFrrJbKZzAi7FRE2Q\nR47qSFSsWv72du3EKJukc2nYOoVFTdhdpYIrOmS5YjI7qVlBK5JGIOYRDGotokjkzyMoYeiAFgxo\nKaCzuxI4zluLBW3xb9Wyo4sC0hH4CCSTW9KEKcD7OHSjSVN8/aC2In5y/uFB0XDTJyRGmON3GYQz\nD9kztt6UtjnNJBBvZzxyJyhjqmgE8vcu/qy2D5BvXVKHoo5S5U3NPoKoaUh9JuoksidmnoRblfkL\nQ9parKKGzjw4fEZE8YEAI4ymU2f1JqXWiJ63IP1trxGcOnWPQGu++VPT8ecvHGesjAeYA0zUZHuN\n1Qf6mCCQH+iqTdEC3EIVU5cLaiEImL18PJfd/jyeTqjR+883zwcQnwSkEql1bAhdDaOAvOurSD6Q\ncoWxcXs3hg9sRYE8e/XWTs80JDQC4UvxTENx34oaYWUbTdGm0QjSJkzJE7DU5TrUbU2dnxixHbD7\nMO1IbLMhW+fgtiKu8mfm2ly2KghaCrJG4LdZYxqq1ShQ7uATbf6qRlBIN0p5HbLefAMoGgER9ho5\nKFJ0B4jOYA73C48PAJ99zz6RAQNB/94EGkGgaXn72CSc87YPB1dajUB5luceMQ4n7O9FNAb7FYXw\nMZ/TZCXQRUsBjRMI6RmVehHyy7/6nWiH3xIIAr1pQ/e89hk9BMve3m59/nKF0dFdwZ+eXY27XlyT\nun1aZkP5JXp2pT5kc/PObvxj8dta01CFvfTV++02FAVC4CweO3IQigWvk5ZNQ12lUqpGYJvTKdQI\nwgOkZdcsSlE2AHDygbvjux86KEiSp5Jm8tDsoY0ZVydHhVtLHZW/7PjJo/GYoZSp2mEVNM7iwEcg\nNaNW1gD5OJHjJ2wHeJ2SeBdPP0jvYKWYjyDBWZwYNUTa3yZRpJuNLJbL5xL9tk3COW/7QhAFuOvQ\nuLlQ9Ut9/9xDY9sIYVEsEI6cOAoXHDMxto06sFJ9G+ryRtFnNQLTulWbdoLIbjp/1kke5QrQLZla\n0khzatlM4PrZI0vxyRvnBtE8FebAQVmuMJas24b9dhsKIi881PMRtKCtpYjNO7tx0z+WB+2VhUhw\nTUobTPdEXSw6/SwagYhcEuw1ciB2Gz7QaE+NRV4YOhNxSCK94/h/P344Tj5wt9jybZ2lSMf12n+f\njpsvmh78luPoAa/mgNo+cT2hLTs+ZyA0DVXXG0QEQaJpKEqBPO10/jdP1uYmCraL+AjijnGrc6tC\nSH22yisvby9SR3slL6PnEvdVV51MR5EIb23pAOC9Zyo2zmL5ff7D547FWZrUH6pGII4rglM+8+59\nAACjBiebietNnxIEpuiSUrkSvDCbdnRhUGsx1Swj9stChTno+GqR495GEKjOWy/yx9tvR1cJO7rK\n2H146ITdvNMXBMUC1mzuCAr4tLUUUKpUYteszuQ1xuorX3hrDo1gjxEDtZqZybcQNw2ZjuxPuoN+\nFum+Y4bim2eqeRSBpeu3x0bB8u/dh0c7kDZN+GgpKGMaHblG2p7QcQ5pK2LeN06OXk1CZI/u7zSE\nABo9dEDiAEYWvOLdFaHK8ruaNPkuLXxUvTSx/Yqrz8R/zggrpgmhL94zcU71GSQhvpO9fB+TTNJ9\nEE5wG8GtRuGJPFuvb/S+24tP2Acrrj7TWGa1p+hTgsBkF+0sVSK5b1o0tkod3WW2CkUTVJjx04eX\nArCbiZp2bBs5pJ5GHoGIENrWYphLvqO7jIGtxVg+lhGDWrG9sxSLGupQJtr98u/LDO2INkR0FHIY\nb1oFrj1HDMQYSU0XfVQeCKoAACAASURBVIsxd43lo5E1AtOxJo4egis+eFBk2UXHTUyc8akeShs+\nGnMWJ2gEmnaNHTUIY/xIJ7UNE3eN5tBpLRYwfdIuwbmD46t26Jyah3y/xfP96qkHAABGDWmTtose\n/4oPHoQrP+TdW1MSNuHwHz8q2imntVSNxkoLSBDIgkvNpAoka6+//fR03PevJ1idR9UITj5wd3zl\n5Mn48vuiIbENtgz1MR+B4SPv6C5HXs4WxQRholSpYGBr0bqoytxlG7HWVzdNbbnj88cGVb3SBIFN\nIZhigVCROm/Zxt9dCkej4uPvLrNnE1Y+mJGDW9Fd5lgUklqi0hQ+WigAkDYd1Kr7uJKvVzjdZp09\nFZf/ZWEQemeeWWynEYi7Q0jOu3/B0XvjmntexdbOEn5w7qH4yBHj8NcX3jRubxJ+ghaNIFA1A7nd\nuvabchw9+u8nYqTGnHDouBF4evnGTKYhW+SBlniW57aPx7mamgcyFxy9d/C3evvFM3z/IXti5OBW\nHLfvaGV9/HhEYUcu2+m933ZXV2bGHZ8/Bq+9tU1rUkx6V4cNbMWwgfEZzzpUU2uxQPjKyfGw8XCC\nnNVha06f0ghMpofOUiUaUWERJQF4jp4sGoEQAl5b9GeQj5caNWQpCGRkx6cYjbco0TgtBYppBCK/\ni5pGY4dlNJXa8QwZEL82Uyd82tTdI6NbsZWanXOsosLHOhXlqZ4yZXfc+S/HBuF9o4e1WTu7tSYc\nhTSNoFAIC/8IbWilbxKQHZRJb6PpPdp71yGxKmHinGrb1Hbl7Wzk78umhKOOWLqLQOPySjomTbSS\nvwZ1fkZLIAjs2lUuM47YexecP32C1jqQ9/pi57FMJ5On+l8t6VOCwGQq6CxVIhpA0RCJoNJdrliH\no6mYVFQ5zjk1asgQzSKjmsNkG7/wV6hhe8UCYYCqEQzyRpeblFBKmxxMunboYqtNo7VfXNCOR/79\nveEC/1hqLhj1mLGcOcrtPGfaOBw+YRTes/8YXPXhg/GNM6akm+wUJ2QwWtd01uo7FDMNUVwjEFky\np00I6zOF54ijF1zm9yJIYSDtd/DYEfjmmQeG58upE0RMQzlLapnCR02YPlMxSFKv11oQaDKFytge\nx8TVvtM9rQJgnMaoBH1KEJg0go7uckQy244KSxW2DkdTMb3A8ozMVB9BjlFCxDSUqBFEr8ukEazd\n3AEb1OtVY8i984bX++79RsfWB8dSfovHpd6PmGlInc5P4XbnT5+AQW3F9A+cw32i54pvqr5HsZnF\nBcIQ/z6IlNM3XNCO5/7rlKgN33A9gFL+0KKT0EYlEeEzx++TeC02RGcW5xUE5hG/DtNaERnXomgE\nSZO7dPt7bYqvz1M/WqZ94i6x8yTRWH2gjwkCUwffqaRbbinam4aGahxJeTh47AgcPHZExCSTpm3Y\nqJUdiv+iS+MvKBYKkY+/WNT5CHyNQBEEL6/ZYtVxFAuEo/fZJfit+yDl5/O/Hz889ZisdMqqXEyL\nGtJ1MtYzT3P0lrqkc5ecsA9mnv4unH+kZ0cf1FaMOFa9dia0I2MIc5502UlcfPyk4FlGopJynkd9\nZmmRefL2svkkzJHlP09/Ozk9RRL7jAlnjevMf9VqBOL7yqoROB9BDUhyFsszZNWJSzraWgrorlS0\nEQU2qJ3WRcdNxF+/+O7IqHFIyktroxCowkJnGtJpBKpZSqT7VYvjAHajvwIRbrvkGBzoFyvXaQQy\niSkQglVRs4rq4I/7CNQ2xY8tC1/t+xKYhvwWJDwD9d7rooYGthbxuffsmxjNEqRL0KzTRVoltUlc\nU5YiQkl848wpeHnWjMixq0F97GowQtr2AtU0JKoE2giCZ755Mg7YI6xToDtHtYJA5Eey9xFUdbqq\n6ZOC4NwjxkWWe3n3w982o6bucgXMsI4OUFGTTYnOTHYQD07TCKo2DYUOtaiPoBAzYwwfJARBPN2C\nzSjlcN/mLUZtOo1g3zFDgr+THkFQlU0JH02r8xrXCOLHlu9/0qjfxjQUmyykMQ3ZcPQ+Xmz5R5T3\n1juGlG7BUjMDkjuWvJ1OLdJ5q/dEjVJT0Y3WicKsrkLAiiy8aYMrID6TOEtaE1taA42g+qzGPUGf\nCh8NIgeUD1LVCNIm2xCFH0tejUBFdFpyZ5GmXtuOJmS6DVFDMkWKd1rClv3Qq+ugklQwHgB+f8nR\nOHjciMgynSA4fnJYfc5m9ql4BuNGDcIXT9oP50wbhxN/8EjCMdLtz7JpSFuQJDi2sXkBqtqvcxbb\nMGFXL3/+Zo02ltXUI7Y3Zb0E8psfalFOUX5mk0YPSfQVqUSihvwfwhwn6n/bmoZk6mGOEe9Cnm+4\nEfQpjSDobJUPUucjSPLOt0qjsOE5NQIVU63SWqPrtIsFghyAVCwWYqGrIzSVoARp73L7xF1ipiD5\n9+fesy8en3kSJo6WNYIEQeD/KzozIsK/nXoAJo4egk9JxVdU7T3NVARETUNJE9zCil1iVnJ8WzUh\nX1LdYBt0qaCzvi82GsHA1iJ+95mjzBvUEflyHv7qiTF/iYrpNako4aM7uuw1gnibav9N5vYR1Lwl\ndlgJAiKaQUSLiGgJEc3UrD+BiJ4lohIRfURZV/arlgWVy+pFGEscvZ0d3eVo+KjiPI0dR9rfRiO4\n+0vH44i9R0WWqR9ilSZHa9SkcYCfWVQWhBofwfAqNJ9oiVDvX1kjGDm4NT4HIOF+JD2bb581FZ9+\n9yQAXu756H6K6UirEUiCQKcR+PuI9gXPUdOmuEZQ3Wes9RHk1giSOS7DSLyWZJ3VrDUNQZpH4K+f\n5n9/B48dEds+zzmqRbwL9ma4Jp9HQERFANcDOB3AFADnE5GamGUlgE8BuEVziJ26ymX1QKhjqore\nWapEsgCmfVyySi+rmjde2K7dfspew2OmkLQIl3pRUkZK4twlJXxW9RGkOXdtESNoOXeKbpRmcz9M\nH5HQetSoq5ihSOsslk1D8ddf3BebDiueUKy6fDG1sFUXLDSCRiIuZ5jlyF2++v38SJ8ZB+0R+M/E\n9Z57xDg8/fX3xUyUVueow6cpntsYTSGknm6LDTbj1OkAljDzMmbuAnAbgLPlDZh5BTO/CKChnhHR\nGauj3VK5Ep1HQKGi/9VT94+YG4CoSi93mO87cPfUcwvUqA3TB33cfrvW1FwkOkl5dNpSKMSSguV1\nbKYhTiPfN91Hn2waSh7VCq1HnR2tHlKrEUhahG4ELzRAG0EVixpqqe4eyue85pxDAOTXCOo5wtwl\nxZyThLhnaUWZBLJAHr/LYLx6xQycd+T4WAlQIsJuw+NZRO3OEf79wGXvwW/+ebp5Y+tjEq7/+DTc\n+S/HWm3faMFtIwjGAnhD+r3KX2bLQCKaT0RPEdEHM7UuI2KEqGoEpTLHRsTi4Y8a0hYbWSYV4Eg7\n92S/qpWuQLiO333maCz97hlW5wC8UnkTdhlsXN8dJJqLOkVljaioEQQqFx6zd+L6NOTJYzoHXmL/\npjiLVQKNQDUNqc5izb5p4aMieioIH01opinFsCBr9lq5QxLzV9Js6CpCy6mXj/KOzx+Le79yfO79\nRQCDCDNOQ31EA1u9UpRCI6jFAEYWwPvtNhTv2X9Mwtb2nHnInhg3yvyt6qi2bnVeesJyvTcztwP4\nOIDriGhfdQMiusQXFvPXr4/XE7ZFjELVYhDdlWiKiUhZPaK40zGHIBAjk3Pbx+GA3YfFUjPIL9vU\nvYbjnGnxUEEbTp26B370sXiRDIG4drlTailSbB5Fa0rqzu+cfRCu/ah3HvGvDcIMJN83XYrd5FKK\nHqbIly5LjUD3TUVSfGgcNyI4QPW16Fqr+gjUd8U2WWFwDukkM6bugW+eeSD+47R3xbZLGj0GPoI6\nDTGP2HsUdhumH3l/6aT98H5NKVCZKXsOxzUfOURb6EWH6T0RQrjWkUyNotGWPBtD3WoAcnrBcf4y\nK5h5tf/vMiJ6BMDhAJYq29wA4AYAaG9vz31PhCBQc+iXyhwbEQd/E8XC/CJ51y29vOKFHNTWgkKB\nsEGZoSsf864v5R9ReZhfXHHt6si3pDiL01JCA8CHp43Dh32BddntL1i17KefmIY/PrMK++8eztzM\nat5Is88LjUD1c6i7qfUCgOh9+aXG5yM0gq1+XHpSh5o2oSyzIJCea6EQTQvhrfdICg0V76GNRnDK\nlN3xwhv6ynd5uMxPSZ0EEeGjKdlKbVAL/lRDnQP5MtEomWQjCOYBmExEk+AJgPPgje5TIaJRAHYw\ncycRjQZwHIBr8jY2DfGRqx+g6iNoKZBU/Do+qlCLkVxzziEYPSxZRRcvpFeAPb7eNne+4Nh9d8UT\nSzdo1yW9uDu7hSCQNIICKbmWCjVPRSDYa+QgfEnJtW6bI14QtMzQmYnRujoCVwXIpF2HQEUIjyFt\nRew7Jl6cXkRPbfEL2h/kR6GcflB8pJsmCNLmX6jkTcAmk+W5/vKf9MEPvYFQI6j+WI0yx+holK8g\nVRAwc4mILgUwB0ARwE3MvJCIZgGYz8yziehIAHcCGAXgA0T0HWaeCuBAAL8gogo8M9TVzPxyvS6m\nzaQRVDgyS1et5BTTCBRB8NEj00cw4oWsMAfHG+QXgHlnR3fmkUvSFPckVXanXwJPNsfoNYKem0KS\nFFa554j4qF1cnilNgjANqVE68llOPnA3rdmAiHDFBw/CMVJeJJnpk3bB7+auxDjfD7PvmKFYcuXp\n2vsVFwTR86nvYRq1MFEE8wiqPlJzIyydtdAIdPM3+htWMVzMfDeAu5Vll0t/z4NnMlL3ewKAuQhq\njQlMQ92KRlDheNI5//0pFJI1gqQModMn7YIf+rbOIB9OhYPj7T58QDCFPqtTK0kQJL37Hb5GMEgx\nDanZV+ulEegwZYW99eKjse9u8VF7MLPYcDxRcEfteMV+uw0bgF9deKSxPXKhFJWzDxuLg8aOiGgL\nJqGppg9QO/KsGkEtzAL19hE0C/VyFjeaZjYN9RrG+x56Naqmu1xRnKUFKZVAvGpVNN2uOTb8fe/a\nDeP9cxUl26wYpYwY3BaYarI6tdoSQhGTXtzVm3YCiE+cUsNHq02qlQWTRiDqt6qouYZUOkVklEFI\nV/sx6UxGOlSNQH0u2Z3FyQ3fdYgXk66rsStImln84rdPzdSeZqa2zuKqD1E1jZbbfUopOna/0bjl\nM0fh8ydGA5NizmLJL0Aa05D8Yqh26Ce/dhIO84tXy+aXY/zEYVP3Gh50CCMHtWpr1NqgdtR3fP4Y\n/PyT0/w2m/db/Y4nCKIaQSEWPmvjLK4VmX0EKU0TI22TI7+nbL4inv7ajx6KGy9sjxVizyoI0njf\ngbvh5588Ape+dz/jNqFpKN6zDB/YWrOUKY1GnVlcDU4j6GMaAeAJAyCaOK5UqWB7V5jlsMJhJ1Ik\nio0q5BdDFQR7jhiEA/ccjuffeCeiOZx+8J6Y/82TMXrogMBfMGJQa9DhVusjOGLv0KZt09HJE9xa\nCoQhis8gLXy0luQ1Q6VOKFOejVykvp5844wD8dwbm3D5B6biiL1H4UOHjwUR4d6X1gDwEvht6yxl\nNg2lQUSYcdAeidsEOZL6tmUoVqqyt5MUCdYT9CmNQEZ0vG3FgleUvSMUBN3lStDRFsjTECL7pswj\nEKvVEL3Rfnpbsf+IQa1BO7L2u0mvt82xVGfxZaccEPktawRPfe19kX1/++naJiTLa4Yy2blFuUc1\nnbAwf9W7a7j4hH3w008cgaEDWvCxIydIhcfD5w6E/pqeRFx7reoRNCtqCdBqaCqNoA9PKGsIYpTf\nWiSUyhVs7SgFeT92dJWDdACk8RFEZhZrOrFw1KX/2ApShyCOnXXk0pUwK9XmxVV9BIPaipjiz+Ys\nFqLXvIcfuXPH54/Br/6pHe+eHE9I9sWT4uaIj7WPxyNfPTG1LVnNUEFFMsP6b5xxIB6feVIs1YFa\n0aynEc9ltP+effGkyUmb1wfhaO/bciAoLDN8UPVGjWZQKhr9vPqcaUggRuKtLQVs6yyjVGGMHTkI\n67d2YkdXKejgC5p5BGkziwONwDBrR8wqlgVB1voUauSTjM17m5RKoUB6Z7FsflL5xFF7438eWhJZ\nduSkXSKppU1knlAm/jB8HC3FQiybaTMg2j1iUCtWXH1mg9rQP8JHr/7wIfj49AmZUzjoaCaNoFFT\nGvquRuDf0LZiIai6JTqP7Z1lyTSUPLNYFz4qqpa1tegjirb7gmDYwJbcpQM7EmLQbUa8cucrHNby\nbtmzWmraYblvXmdxVrtpT/kITOzoDp97owiuvY9LgkFtRRy1jz7qLCvNJAcaRZ/VCAqSSeaZ1zcB\nAPYa6ZlAtneVguyHafMIdKahf3nvvmgpkra0IBA6Mwe1FYMOOWuBikSNgMJ2miogRQuNZzq1Fp2z\n2/a42TWCfF+maOKglBKg9WJrhzcbuZraDtWiFtRxpNMoU2Iz0WcFgejM12zuCJbpNIJKJd7JRXL5\nazqxwW0t+MrJ+xvPHUa1FINjqdWs0kialSo+9gIBpq2ixeqrlwT6AiF2H1B+Z3G27ceNGoSvnDw5\nd0K/atmy0wtIaGSIZjgru2FNcOSg0Y+rz5qGdCNYMRFnR1cpmOTUXa4kTijLQ3dQOKWADxy6V+Tc\ntiTFoAfx6gkdsdz3qteTxzGlE4i2o86s9zMwDWVsJxHhKyfvH0zy62mE5rOHJm1GTxGkz26099GR\ni0bpJn1WEEzVlKwTUSZT9hwejFK7ypVE01AeuvzJawNbi/jUsROx8DunYc8RyYLglCnRojeffU8s\nW3dAIAASmkkRjUBESCU2IRHdLeku16ezCc3cvaszu/DYifjWB6YkprCoP/3DWdzXaLTg7rOC4Kef\nmIbbP3tMZFlrsYA/f+E43HjhkUE0UFepkugszkOXlAqaiKwKasuZIJ+//BScklANjVLkwMFjR2jN\nW1d/+BCceMCYIPQuC/LxPjzNq0tk8k9US16NoNG0tRRw0XGTejShn4nedu96gtmXHodbLq7tHJla\n07DQ54actQcYOqAF0ydFwyGLBcJh40dixODWwAncXebEwjR56A40gny3Vze3QSatfXK6A5mDxo7A\nzRdNty62Ezmn9IIKZ2xWB7g9znmXF5Fn65xpWYoINh+fPWEf7D48W73fNA4ZNxLH7hufI9MMHDrO\nS1vzgZTCPvWizzqLdehmDHeXKzFHaLUagXAWq6UUbSlQcqRNWutai4XgmmqVZbSgMTWVM5ZizEp/\nHNRe//Fp2G83u6R3OsYMG4Cl3z2jKSZJVcPXzjgQXzvjwEY3o8eYOHpIw+aeAP1YELQWQ0FQa2ex\nGCmrtZBtKWjyH6nrAbPNv61FEgQ1Si4XqermRyHVSyMQYb5TLOva9iXOrMGIsK/k33H0HP1KEMij\n2i+dNBnv7OjGedMnYN7yjdHtavQh5TcNJa9Pa15rsRB0BrUIHVXPGUZc1UcQHDJuJP566bsxZa/+\nJwgcjkbQrwSBPFIaMbgVP/SLsseihmo0oMqqEbT4lcRSp7ynCgIKOu5aaQSyEyvv3IgsHDwuHvXl\ncDjqg9VwkYhmENEiIlpCRDM1608gomeJqEREH1HWXUhEi/3/LqxVw2259eKjg79Ntv/4hLLajKKT\nqpvpELOd0yI+AtOQQSKQZFqqlUYgI6Ji6ucsdjgcPUmqRkBERQDXAzgFwCoA84hotlJ7eCWATwH4\nqrLvLgC+BaAdnu/vGX/fTbVpfjoHjQ3NC0XD6FjtK2sV/Zc1FOz2zx6Dh15dF0khrcMmSVaYGiO5\ns86TjiFIm1En05DD4ehZbExD0wEsYeZlAEBEtwE4G0AgCJh5hb9OtRWcBuB+Zt7or78fwAwAt1bd\ncksiTs4e0giu/NBBuOOZVZn323vXIbjouEmp2yWJAXG5IvFZUnGUV2bNyDXJTJib0jSC319yNDbv\n7M5+AofD0aPYCIKxAN6Qfq8CYDsrQ7dvjwY42yRfi0cNVXfOTxy1Nz5xVP1mlyZFDS27ygtBGzrA\ny3eTVNcgTfNQmT5pF3xk2jhM23sUrrl3Ed6fEuFSq+yQDoejvjSFs5iILgFwCQBMmDChpse20Qji\nzuLmDr8jX1AltXLIAK+Tr2W5RHmmdiNjnh0OR22xGfuuBjBe+j3OX2aD1b7MfAMztzNz+5gxYywP\nbUdRE+2StA1Qu/DRemHTOmEacv5ch8ORho0gmAdgMhFNIqI2AOcBmG15/DkATiWiUUQ0CsCp/rIe\no1CwEAS9TCMQJDmjhWnI4XDYkXfeT18g1TTEzCUiuhReB14EcBMzLySiWQDmM/NsIjoSwJ0ARgH4\nABF9h5mnMvNGIroCnjABgFnCcdwITIJAjcJpzZGLp54cNn5k4nqieMjp0AYWR3E4ehsPXHYCRgxq\nS9+wj2LVWzDz3QDuVpZdLv09D57ZR7fvTQBuqqKNNcMUdimcyGNHDsLJB+6GC47eGz97ZGkPtkzP\nHZ8/Bl0ljgkCuc9/edZpAIApl0cVrcENqtLlcPRG9tste0bevkS/Gjam5WAZOqAF3zn7IGze0Rwh\nj6Zi8oNbixjSVsTlH5iCwW3eI/zhuYdiqjRnQpjETptqTmftcDgcQH8TBJa2/zpMxq0pLcUCFs6a\nEVl2jqZ+8sLvnJZ5drPD4eh/9CtBYIoGUu3rfSV7o01BHIfD4XDDRQmhMNikcHA4HI6+ghMEGvqK\nRuBwOBw2OEEAjWnI1whOPKC2k9scDoejGXFGZADjdxkEAPj8ifsC8HwJj3z1ROw+fGAjm+VwOBw9\nghMEAIYNbI3lzpk4ekiDWuNwOBw9izMNORwORz/HCQKHw+Ho5zhB4HA4HP0cJwgcDoejn+MEgcPh\ncPRznCBwOByOfo4TBA6Hw9HPcYLA4XA4+jlWgoCIZhDRIiJaQkQzNesHENHv/fVziWiiv3wiEe0k\nouf9/35e2+Y7HA6Ho1pSZxYTURHA9QBOAbAKwDwims3ML0ubfRrAJmbej4jOA/A9AB/z1y1l5sNq\n3G6Hw+Fw1AgbjWA6gCXMvIyZuwDcBuBsZZuzAfzG//uPAN5HSZXVHQ6Hw9E02AiCsQDekH6v8pdp\nt2HmEoDNAHb1100ioueI6FEiOr7K9jocDoejxtQ76dwaABOYeQMRHQHgz0Q0lZm3yBsR0SUALgGA\nCRMm1LwRN32qHV2lSs2P63A4HH0BG41gNYDx0u9x/jLtNkTUAmAEgA3M3MnMGwCAmZ8BsBTA/uoJ\nmPkGZm5n5vYxY2pfA+Ckd+2OGQftWfPjOhwOR1/ARhDMAzCZiCYRURuA8wDMVraZDeBC/++PAHiI\nmZmIxvjOZhDRPgAmA1hWm6Y7HA6HoxakmoaYuURElwKYA6AI4CZmXkhEswDMZ+bZAG4E8H9EtATA\nRnjCAgBOADCLiLoBVAB8jpk31uNCHA6Hw5EPYrVOY4Npb2/n+fPnN7oZDofD0asgomeYuT3Pvm5m\nscPhcPRznCBwOByOfo4TBA6Hw9HPcYLA4XA4+jlOEDgcDkc/p+mihohoPYDXqzjEaABv16g5PUVv\nbDPQO9vt2txz9MZ298Y2A167hzBzrhm5TScIqoWI5ucNoWoUvbHNQO9st2tzz9Eb290b2wxU325n\nGnI4HI5+jhMEDofD0c/pi4LghkY3IAe9sc1A72y3a3PP0Rvb3RvbDFTZ7j7nI3A4HA5HNvqiRuBw\nOByODPQZQUBEM4hoEREtIaKZjW6PDBHdRETriOgladkuRHQ/ES32/x3lLyci+ol/HS8S0bQGtXk8\nET1MRC8T0UIi+nKzt5uIBhLR00T0gt/m7/jLJxHRXL9tv/fTqYOIBvi/l/jrJ/Z0m6W2F/1Kfn/r\nRW1eQUQLiOh5IprvL2va90Nq90gi+iMRvUpErxDRMc3cbiI6wL/H4r8tRPSVmraZmXv9f/DSYy8F\nsA+ANgAvAJjS6HZJ7TsBwDQAL0nLrgEw0/97JoDv+X+fAeAeAATgaABzG9TmPQFM8/8eBuA1AFOa\nud3+uYf6f7cCmOu35XYA5/nLfw7g8/7f/wLg5/7f5wH4fQPfkcsA3ALgb/7v3tDmFQBGK8ua9v2Q\n2vgbAJ/x/24DMLI3tNtvTxHAWgB717LNDbugGt+cYwDMkX5/DcDXGt0upY0TFUGwCMCe/t97Aljk\n//0LAOfrtmtw+/8C4JTe0m4AgwE8C+AoeBOEWtR3BV6NjWP8v1v87agBbR0H4EEAJwH4m/8BN3Wb\n/fPrBEFTvx/wqicuV+9Zs7dbOv+pAB6vdZv7imloLIA3pN+r/GXNzO7MvMb/ey2A3f2/m+5afPPD\n4fBG2E3dbt/E8jyAdQDuh6cpvsPMJU27gjb76zcD2LVnWwwAuA7Af8Ar3gS/Dc3eZgBgAPcR0TPk\n1R0Hmvz9ADAJwHoAv/ZNcb8ioiFo/nYLzgNwq/93zdrcVwRBr4Y9sd2U4VtENBTAHQC+wsxb5HXN\n2G5mLjPzYfBG2dMBvKvBTUqEiN4PYB17Nb17G+9m5mkATgfwBSI6QV7ZjO8HPC1qGoCfMfPhALbD\nM6sENGm74fuJzgLwB3VdtW3uK4JgNYDx0u9x/rJm5i0i2hMA/H/X+cub5lqIqBWeEPgdM//JX9z0\n7QYA5v/f3r2r1BFFYRz/ryZRJHgBO4sghHQhRYqAKQQ7CyubENDCpwiBPEIgRZ4hFoEEay+1F7wn\nFioIMeQCgVinWCnWmjgIIsjxzMb5fjCcmT2n+Ab2Yc3eew7jf4AVYlplwMyq17LWc/3PnOf7gd9d\njjoGTJnZCTBPTA+9pezMALj7t/z8BXwkCm/p/eMUOHX31Tz+QBSG0nNDFNxNd/+Zxx3LfFsKwTrw\nIJ+0uEMMnxYaznSVBWA292eJOfiqfSZX/p8CZ7XhX9eYmRHvoj5w9ze1U8XmNrNhMxvI/V5iTeOA\nKAjTl2SurmUaWM47q65x95fuPuLu94l+u+zuLyg4M4CZ9ZnZvWqfmLvep+D+AeDuP4CvZvYwmyaA\nLxSeOz3nfFoIppjtnAAAALtJREFUOpm5qUWPG1hEmSSebDkGXjWd50K298B34C9xRzJHzOsuAYfA\nIjCU3zXgXV7HHvCkoczPiKHmLrCd22TJuYFHwFZm3gdeZ/sosAYcEcPqu9nek8dHeX604X4yzvlT\nQ0Vnznw7uX2ufnMl949a9sfARvaTT8Bg6bmBPmLk119r61hm/bNYRKTlbsvUkIiIXJMKgYhIy6kQ\niIi0nAqBiEjLqRCIiLScCoGISMupEIiItJwKgYhIy/0DkNMtJd4oGTwAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "tags": []
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.plot(losses);"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 269
    },
    "colab_type": "code",
    "collapsed": false,
    "id": "14pHKgfPXNga",
    "outputId": "26d8a395-9024-41f4-95e0-c354ea17e838"
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXYAAAD8CAYAAABjAo9vAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4zLCBo\ndHRwOi8vbWF0cGxvdGxpYi5vcmcvnQurowAAIABJREFUeJztnXuM7Gd537/P3C87szuz3l3sc876\nGEIgrsHYOTihoZSLgiBQiHuJME0Vmiqnf5CISI0aSkSbVopUVTQKlaJIbkOTtGCUAgYEKYGmUIpU\njI+NwcbmYpC9e47tc9Y7e5nLzv3tH795Z387O5ff5f1d3nefj2R5z+6c37xzZuc7zzzv9/0+JIQA\nwzAMYw6JqBfAMAzDqIWFnWEYxjBY2BmGYQyDhZ1hGMYwWNgZhmEMg4WdYRjGMFjYGYZhDIOFnWEY\nxjBY2BmGYQwjFcWd3nTTTeLixYtR3DXDMIy2PPLIIy8KIdYW3S4SYb948SKuXLkSxV0zDMNoCxE9\n6+R23IphGIYxDBZ2hmEYw2BhZxiGMQwWdoZhGMNgYWcYhjEMJcJORCtE9Cki+j4RPUVEr1NxXYZh\nGMY9quyOHwXwJSHEPySiDICCousyDMMwLvEt7ES0DOANAN4HAEKILoCu3+ueRQZDgU89so177zqP\nTEptl+zxqwf4ypMvOL79WjmHX/25TRCR0nV89Qc38O1n95Re0wuJBOG+ezaxUc4pve5z+0f4yyvb\nGA6djZzMZ1J439++iHwmqXQdT1w7wJe/5/z5ZsLj3rvP47abioHeh4qK/TYAOwD+KxHdCeARAB8Q\nQjTtNyKiywAuA8Dm5qaCuzWPx7b38LuffhzL+QzedsdLlF77I1/+Af7PD3fgRKflGNw3v3Id51by\nStfx4c8+gat7R47WESRCAOlkAu9/008pve4nH97Gf/qbH7n6d37FS5bw5lduKF3HR//mR/jKk9cj\n/3dmTnP3rRUthD0F4G4AvyWEeIiIPgrggwA+bL+REOJ+APcDwKVLl3iC9hT2Wz0AwFatueCW7tmq\ntfCOV92MP/7Hdy+87VeevI7f+Isr2G10lAv7i40OfuPv3Ibfe8ftSq/rlp/58Jew31L/wXK/1cVK\nIY3H/vVbF972mRebeONHvoa9Zi+Qdbzupat44PLPK782E39UfN6/CuCqEOKh0Z8/BUvoGZc0On0A\nlgirZDAUuLrXwoWqs62PajENAKg11QrfUXeAdm+ISjGj9LpeWM6ncXCkXlAPjnpYzqcdr0H+nSjX\nwZiHb2EXQrwAYJuIXjH61lsAPOn3umeRw7YU9iOl133hsI3eQGDTsbBnAQB7iiva3WYHALDKwg4A\nKLOwMwGhyhXzWwA+PnLE/ATAP1V03TNFvW29wLcVV+xbu9b1HAt7wRLe3YZaYZcth0qBhR0AkglC\nKZsKbh0FFvazihJhF0I8BuCSimudZRqjiv3qXguDoUAyoWbnS75ROBX2Ui6FZIKUV+y10fWqMajY\ny/k0ru2r/WQEWIJ6i4t9iXI+jUPFwt7pWy0vrtjPLnzyNEbUR8LeGwi8cNhWdt2tWgvJBOHmFWfW\nvkSCUClklPfYa6NWTByEfTkAQQWAQ5ctkCA+OcjrlVnYzyws7DFCbp4Cx+0TFWzVWrhlJYd00vnT\nXS2mAxD23uja8RB21YIqhHDd2w5iHfINiyv2swsLe4yot3so5azumMo++1at5bgNI6kWg6nYkwlC\nORe94Czn02h0+ugNhsquedQboDcQkQv7AQv7mYeFPUYctvv46Y0SUglSanncjo2w91AppJFQtHfg\nh+W89Qaqsh3jRVBZ2JkgYGGPEY12Hyv5NM5V8sqEvdHpY7fZdexhlwRVscehDQNg7BhRKaqehL3A\nws6oh4U9RtQ7Vitms1pQJuxuHTGSaiGD/aMeBg4zT5yw1+zFwuoIBHM4SJ4cdluxd/pDtHuDSNfB\nmAULe4xotPtYyqVwoVpQ1mOXbxC3Vt1lU1SLGQihVvh2mx2sLpkr7F4q5SAOKY1dMblIZtUzMYCF\nPSYIIVBv91HKpbFZLWC32T3hkvGK14pdHvuXFkUV7LXMrti99tiDWMdSNoWUCxcUYxb8zMeEdm+I\n/lCMWzGAGsvjVq2Fci7l+hRidSzsagRnMBTYb3Vj02OXlbLKzdNDD/7xoISd2zBnGxb2mFDvWC/s\nUtYm7AraMc/utrC56n7uSVVxxX5w1MNQxMPDDgQnqETWc+h6HS21bzB8OOlsw8IeE+Sp01IuPXaw\nqOize7E6AuordumwiYuwZ1NJ5NIJ5cJezrmzcwZXsXN//SzDwh4TZE7MUjaF5Xway/m074rdius9\ncm11BI6DulRV7HETdkC9h9xLC4RbMUwQsLDHhOOK3aq0VFgerx+20R0MPVXsuXQSxUxSecUel81T\nIB7CLp0rUa+DMQsW9pggI3tLo+P2mwosj1seHTGS6lJGWcKjFPa42B2BeAh7KpnAkuLoXhZ2hoU9\nJtQ7Jyv2C9UCru4d+Tog5FvYCxnsKjp9Kt8g4lex+7eUSrwKqsqkSY7sZQAW9tgwrRXTHQxx3Ud8\n73athQTBVT64nUoxgz1Fwl5rdlHIJJFLJ5VcTwWqs9C9ulHKCj85cJwAA7Cwxwa5eVrMHgs74M/y\naMX15l3F9dpRmRdTa8bHwy5R2YrxEtl7vA51rRgvXnrGPFjYY0K93UM+nRyLsCph99qGAaxWjOnC\n3uj00VcQ3eslste+Dq7YGZWwsMcEK07g2Ht8y0oOyQT52kD16mGXVJcyOOoNcNT1H1AVV2EHjoeI\n+8GPoLKwM6pRIuxE9AwRPU5EjxHRFRXXPGs0OlYAmCSVTODcivf43manjxcb7uN67cih1jUFzpha\nszu+XlxQ6SFnYWfihMrjaW8SQryo8HpnisN2b2x1lPjxsm/v+XPEAMeHifaaXZzzuAEriXPFrkTY\nfUTl2qN7/W4u+1kHYw7ciokJjU7/VMbIhWrBcxCY/Hu3esiJkUgh9mt5POoOcNQbjBMj48KKwmEb\n8horLsPWAGB59ElGhUNH2jd58/Rso0rYBYAvE9EjRHR52g2I6DIRXSGiKzs7O4ru1hwme+wAfMX3\n+vWwAycrdj/IVs5qzIQ9Tq0YlesoZpKenVCMGah69l8vhLgbwNsBvJ+I3jB5AyHE/UKIS0KIS2tr\na4ru1hzsg6wlmz7CwLZrLZRyKV8fyVVV7PKNIW4Vu8ohFwc+bIaqhZ3bMIwSYRdCXBv9/waABwHc\no+K6Z4lGu4+l7OkeO+DN8iitjkTeB0eXc2kkE+S/Yo9hABhgc8UoENRDD5G9k+tQJezchmF8CzsR\nFYmoJL8G8FYAT/i97lliMBRodgdKK3a/HnYASCQIlULad8UeV2FXGd3rJbJXolLYD7liZ6CmYt8A\n8A0i+g6AbwH4ohDiSwque2ZoTOTESJYLaZRzKdcV+3AosL135FvYASvbRVnFHjO7IzCyGioYcuGn\nBcKtGEY1vu2OQoifALhTwVrOLMfJjqefjs1V95bH6/U2uv2hLw+7REWsQK3ZRYLiacFT5SH3I6gq\no3sPjnqenDmMWfDWeQw4rthPvyC9eNml1VFFxV4tZnwfUKq1uqgUMp7aFEETB2FXGd3LFTsDsLDH\ngrptetIkF6oFXK0dYegivleF1VFSVZDwWGvE73CSJA7Crmod3f4QR70BCzvDwh4H5rZiZHxv3Xl8\nr9+4XjvVojVsw80byyS1Vjd2VkeJqsjcg6O+LzeKighhjhNgJCzsMWAyi93O2PLo4gTqVq2Fm5fz\nyKT8P73VYgZD4a//W2t2Y3c4SaKiUhZC+HajqIju9eOlZ8yChT0GHAv79B474M7LrsLqKFFxSGmv\nGd+KXUV0b7s3RHfgb2qRijcYrtgZCQt7DJhldwSsdorb+N6tmhqrI3A8ys7r7NPhUGCvFb9kR4mK\n6F4VgqpC2A9Z2JkRLOwxoN7uIZkg5Kck+6WTCdyyknNcsbe6fbzY6GDTR/iXnXHF3vAm7AdHPQxF\n/A4nSVR4yOMi7FyxMxIW9hhQb/exlE3NPP6/WS3gWYfCvl07AgAlHnbAFgTmsWKXVkkW9sXraPeG\n6PS9DzVhYWckLOwxoDES9llsVguOWzGysr9VsbB7PaQU1zgBSZyEXdU6ePOUYWGPAYdTInvtXKgW\n8GKji6aD+F6VHnYAyKWTKGSSLOxzUCHsUoz9WB45speR8G9ADGh0rACpWYzDwPYWV+3btRZK2ZTS\nY+V+DimdBWHfH7WbVFTs+z5ya/ZbfOqUsWBhjwH19sl5p5O48bJv1Vq44DOud5JqMePZ7iiFvRJT\nV4yKSnkc2TvnOVyEqk8O3IZhABb2WDBtepIdN152lR52SaWQ8b552uwin04in/E3yzMocukksil/\n0b0HRz2UsilfWTgqhJ0jexkJC3sMaHTmb54u59Mo5VILN1CHQ4HtWkuZ1VGyWsx4tjvuxXCI9SR+\no3sPjnpY9tn6UlWxs7AzAAt75AghRmPxZr8gichRyuNOo4OOorheO5Wij4q9pYmwRyyoKsb0sbAz\nEhb2iOn0h+gNxML+rBNhV+2IkVSLGbS6A7R77j3WNV0q9ogFNZ1MoJhJRr4OxgxY2CNmXgCYnc1q\nAdt78+N7n1WYw27Hj5edhT2cdXBkL2OHhT1i5kX22rlQLaDbH+JGvTPzNlu1FoiAcwrieu2wsM/n\n4KivRFD9RPeOvfQ8PYkBC3vkyACwpez8F6QTZ8x2rYVbFMX12vEq7O3eAK3uIPbC7kdQZWSvCpuh\nnzcYjhNg7ChTACJKEtG3iegLqq55FnDTigHmC7vlYVdbrQPe82LifjhJspxPo97pY+BhmIiKyF77\nOvwKO/vYGUBtxf4BAE8pvN6ZwKmw37KSR4IWC7vq/jqAceSuW8tj3A8nSZZ9HFJSWSn7EXaO7GXs\nKBF2IjoP4B0A/ouK681CCOEoL0Unxj32Ba2YTCqBW1by2NptTv35UXeAnXonEGFfzqeRILMrdsCb\n1TAuws6tGMaOqor9jwD8SwDex9A44EMPPoE3fuRrQd5F6Dit2IH5lkeZI7O5WlS3uBGJBKFScB8r\nsBfzyF5JnITda3QvCztjx7ewE9E7AdwQQjyy4HaXiegKEV3Z2dnxdF8rhTT2ml0I4X2wctwYb546\nFvajqT/bCsjqKKl4CALTpmIvxETYY7IORn9UVOy/AOBdRPQMgE8CeDMR/ffJGwkh7hdCXBJCXFpb\nW/N0R6vFDPpD4WuMWdyot3vIpROOolat+N4OWt3Tjz+ow0mSajHj2hVTa3aRoPiLzYqCin0l7//N\ny2+vv8CRvcwI378FQoh/JYQ4L4S4COA9AP63EOJXfa9sCuP5mz4GK8eNRqc/N07Azji+d0rVvlVr\nYSmbQiUgH3O14E3YVwoZJH2EY4VBnFoxftYR9zdQJjy0enuvLo381B5zS+LIYbuP0pwAMDvzLI/b\nAcT12qkuuc+L0eFwEuAvp+VAQWSvhIWdUYX/30YbQoivAfiaymvakba7msekwTiyKLLXzjxh36q1\n8NI19Runkmohg71WD8OhcBxPW2t2x89ZnJHRvV5aIIcKInslfoWdPeyMRK+KvWhexd5o9xxtnALW\n5nEpezq+VwgRmIddUi1mMBgKHLadi44uFTvg3WqoIrLXvgYAniKEOYudsaOnsBvUY6+3+ws97BIi\nwoUplseduhXXG7SwA3BledxrdVHRSNi9jKVT2QI5bgm5NwdwK4axo5WwFzJJZFIJAzdPnXfEpnnZ\n5Z9V57DbkQLt9N9+OBTYa/VQLeohNr4qdkWC6ie6l4WdsaOVsBORNc3HIGFfNO90ks3VArZrrRPx\nvUFbHQHLago4/7R02O5hMBSoFrOBrUklcRB2r+voDYZodTmylzlGK2EHRvM3DRH2wVC4sjsCVlXe\n6Q+x0ziO7x3H9VbUB4BJKi6F/fhwkh5iExdhL3tYBx9OYibRTthXl8yp2Jujg0ZO7Y7AdGfM1m4L\nN5dzyKaCGxg9diQ53Lg+FnY9Knav0b2q3SjLHtbBws5Mop2wVwre52/GDTc5MZKxsO/ahH3kYQ+S\nfCaJfDrp2Go6FnYN7I6At+jedm+Abl9NZK99HVyxM37RTti9HG2PK42xsDt/QZ4bxfc+Wzsp7EH2\n1yXVYsZ9xb6kj7AD7o7zByGofoSdfeyMREthr7f76PYDDZIMBRnZ62bzNJNK4Obl/NjLftQd4EZA\ncb2TuHlTlW8AOlXsgLvDQXERdvlmtMJj8ZgR2gm73MTbN6Ad46UVA5y0PF4dx/UGL+xuEh5rjS5y\n6QTymeD6/iqJk7AfjVo8Ua6D0RvthH3Vw0GZuFLvuN88BU4KexhWR4kbq2mt1cWqJhungLfIXHlC\nVKmwx2QdjN5oJ+wmJTyOpye56LEDVnW+U+/gqDsIVdjdWE1rzS4qmlgdgXhV7F7WwZG9jB3tfhNW\nl8yp2BseWzHSAbO918JWrYViJhlKJsvqUgbN7gDt3uIJP3vNrjZWRyA+wu4laZJPnTKTaCfs44rd\nkB57gqyoBDfYLY9Bx/XacfNvX2t1UdVoM8+PsLv9xOVkHW7dOSzsjB0Nhd36Bd41ILq33u5hKZty\nLcr2Q0phWR0BdyFstYZeFXsubeUQuRXUUi6ldJCIFOj9I+e/3/sc2ctMoJ2wp5IJLOfTZlTsLuME\nJJVCGkvZVGyFvd0boNkdaBMnIHFrNQyiUvYS3cuRvcwk2gk7YM4hJTdDNuzI+N5Ht/bQ7g1DsToC\nzoVdvunqVLEDMRN2F9G93IphJmFhj5CGR2EHgM1qHo9fOwAQbFyvHafCrlsAmCQOwp5OJlBwGd3L\nws5MoqWwVzwMVo4j9Y7VY/fCZrUAIY6/DoPlfBpEi62m8rmpaHLqVBIHYXe7Do7sZabhW9iJKEdE\n3yKi7xDR94jo36pY2DxWDanYrVaMtxekFHMiKz8mDJIJst5UF+xvyOdmVZOcGImOws6nTplpqBhm\n3QHwZiFEg4jSAL5BRP9TCPFNBdeeSqVoJTwKIUKx+QVFw+WQDTuy/fKScg65dHjH9iuF9OIeO1fs\nvnATIczCzkzDd8UuLBqjP6ZH/znPPvXAajGD3kCMj+TritfNU+C4Yg+rvy5ZLWYd9diJgBXNhL2c\nT6PedhbdKyN7g7AZcsXO+EVJj52IkkT0GIAbAL4ihHhIxXVn4Xb+pmru//qP8d+++ayva3T6A3QH\nQ5Q9tmLOVfIgCq+/LqkUF1fstVYXK/m0Un93GEhxlFEP8whSUL0IO/vYGTtKhF0IMRBCvAbAeQD3\nENEdk7chostEdIWIruzs7Pi6P+m2iCJWYK/ZxUe+/EP8jyvbvq4jkx29bp5mU0n8zltfgfe89oKv\ndbilWsyi1pwvOrVmN5SIA9W4OX0aF2E/5IqdmYJSV4wQYh/AVwG8bcrP7hdCXBJCXFpbW/N1P9If\nHUXF/plvX0O3P8T1w7av63iN7LXz/jf9FC5drPpah1uqRetw2HBOu4KF3f86nEb3ciuGmYYKV8wa\nEa2Mvs4D+EUA3/d73XmM52+GLOxCCDzwrS0AwIuNrqsxapM0fFbsUVEtZjEYivEb0zTOhLAHGJUb\nl3Uw+qKiYr8ZwFeJ6LsAHobVY/+CguvORI5bC1vYH35mD0/faOBV55YxGArsNjuer+U1sjdqZBts\nnuWx1uxpKewrLrLQ5W2CmFrkdh35Uc4Nw0hUuGK+K4S4SwjxaiHEHUKIf6diYfMoZpLIJBOO52+q\n4oFvbaGUTeHXX38RAHDj0Iewd/y3YqKgMv60NP2xD4cCe62udlZHID6tGDfRvXzqlJmGlm/zRGTF\nCoSY8Ljf6uKLjz+Pe+8+h1tXiwCAG3XvfXYVPfYokFORZm2gSrugjhX7OFnRQQBXEJG9k+tw4mVn\nYWemoZeq2JCHlMLiM49am6bvee0mynnrn+26n4pd01aMnIo0q2IfD7HWUNjdRPcGEdkrcfvJgYWd\nmUTLih1wN3/TL3LT9DUXVnD7LWWslayq1U8rRtfN00UVuxR8HYUdcG41DDIq162ws4edmURbYa8U\nnc/f9MuVZ/fwoxsNvPeeTQCWh7xazOC6n1ZMp49sKqHdplc+k0QunZhdsY8E33RhD7JSdiPsnMXO\nTEMvVbFRLaRDq9gfeMjaNH3nnTePv7deyvrbPPURABY1q3MOKZ2Vij1IYXcT3cutGGYa+gp7MYt6\nu4/eYPEhDj/st7r4wuPP45fvOodC5rhtsl7O+dw87Wm3cSqxYgW4Yg9SUJ2sozcYosmRvcwUNBZ2\n65c56A3UB0cnTe8btWEkfiv2Rsd7AFjUWNG9syv2bCqBfIiJkyrRSdiP4wT0/D1igkNjYZebeMEJ\nu9w0vXO0aWpno5zFTqPj+fRpvd3XbuNUsjpnf6PW7GG1mNE2Tjkuwl52sI6xlz6AQ1KM3mgr7Me2\nu+CE/ZFn9/DD6w28957TQVvrpRwGQ+H5/vVuxcwedFJrdsbpmzriJLq33RugE1Bkr2TZQSY758Qw\ns9BW2FdDqNg/8a0tLGVTeOerbzn1s42ydf9ew8AaWm+eZtDo9NHpD079rNbSM05A4iS6N4xERSef\nHFjYmVloK+yyYg/K8njQ6uGL330e737NLShOaZmslXIAvJ8+1bkVc5yHf1p49jQNAJM4sRqGIags\n7Iwf9BX2URZJUJbHB799FZ3+EO/9uc2pP5cVu5cN1OFQoNHto6xpK2a1ODuETddkR0mchL3VHcx1\nfclPDnxAiZlEW2FPJxMo51KBVOzWpuk27jy/jL91y/LU28jTp15iBZrdPoSA53mnUVOZEZvc6Q/Q\n6PTHsco6Eidhj8M6GD3RVtgByysdRMX+6NYefnC9fsriaCebSqJSSHtqxRwHgOn5glyVsckTVlPZ\nmpGxyjoSF0F1uo5cOoFsSk9rKRMc2gt7ED72Tzy0jWImib935+lNUzsb5Zynir2haWSvRFbsk5+W\nZAXPFXt46+BqnZmG9sK+aP6mWw5aPXzhu8/h3Xedm7ppametlMWOp4rdWrOum6crhQyITu9vSGHX\n2e7oRtiD7G07yWRnYWdmYYCwez/9OY3PPnbN2jSd04aReK3YdW/FJBOElXz6dMU++vS0qrGw59IJ\nZJKJhYJaygYT2StxksnOws7MQmthtxIeexDC++xRO/Kk6avPL+OOc9M3Te2sl6zTp/MGO09D1yEb\ndqYdUqo1OuOf6QoRobzgcFAYUbnOPjn0WdiZqWgt7KvFDLqD4bhn7ZdHt/bx/Rfmb5ra2SjnRrNP\n3fX5TRD21WnC3uqBCFjRXGyW86n5gtoKvlJ2Ms3poNXFcl7fN1EmOLQW9uNNPDV99ge+teVo01Sy\nLgduuOyzNzp6Tk+yUymcFva9ZhfL+TRSSa1/rRYeDgqjBZIZBalFvQ5GT3y/AonoAhF9lYieJKLv\nEdEHVCzMCfIgzK6CPvvBkbVp+q7XnHO8qbleHp0+ddlnr7f7IAIKmiYgApblcdLuqPvhJEkchH3R\nOjiyl5mHitKqD+BfCCFuB/DzAN5PRLcruO5CpIiosDx+7rFraPecbZpK1kve8mJknEAiwM23oKkU\nrIRH+/5GrdnV2uoo0UHYObKXmYdvYRdCPC+EeHT0dR3AUwDO+b2uE6rjo+3+WjFCCHzioS3cca6M\nV51fvGkqWZexAnX3FXtJU6ujpFrMoD8UOGwf728YVbHP620f9UKJyp0n7BzZy8xDaTOUiC4CuAvA\nQ1N+dpmIrhDRlZ2dHSX3dyzs/loxu80uvv9CHe9y2FuXyNOn7iv2ntb9dcD2acnWZ6+1zBH2eqc/\n1e0kI3vDqNjnuXM4ToCZhzJhJ6IlAJ8G8NtCiMPJnwsh7hdCXBJCXFpbW1Nyn0vZFNJJ8l2xb9Va\nAICXrS25/rvrpZzril3n6UmSSvFkCJsQAnvNrtZWR0k5n4YQx+4lO2EGbzmq2FnYmSkoEXYiSsMS\n9Y8LIT6j4poO71fJIaXtkbBvVguu/+56OYsbXnrsmgv76kTFftjuoz8UWh9OkszzkIcpqCzsjFdU\nuGIIwJ8CeEoI8Yf+l+QOy3bns2LftYT9fMWDsHuu2PV+QU4mPI7jBAzZPAXiIeyzons5speZh4qK\n/RcA/BMAbyaix0b//ZKC6zpidcl/xb5Va2G9lEU+495+uFHOYqfu7vRpvd3TNidGMpnwOA4A0zjZ\nUTI+HHR02m0VrrCnTtxnVOtg9MO3ugghvgEgMt9epZDBc/unWvqu2Kq1cOuq+2odsCyP/aFArdXF\nTUtZR3/nsK3vkA1JPp1ENpUYC/qeAcmOEuk0iVpQ7euY/N3iyF5mHnofEcQok73hv8d+wUN/HbBi\nBQDnXvZOf4Buf6j95ikRnYgVGFfsBvTYV0bH9OcJexixCYvWwdU6MwsjhP2w3Z87Qmwenf4Azx+2\nPW2cAu697I2R00L3VgwgQ9hGwt4yR9id9NjD6G3Pi+5lYWfmYYSwA/PDkuZxde8IQnhzxADW5ikA\nx86Y4yEb+r8o7ROsas0uMqkECh72KeLGvOjeMCJ7JfOie1nYmXkYI+zTBis7YcuH1RE4nn3qNC9G\neqN1tzsCJydY1ZpdrBYzsExSejMvujeMyF7J/E8OHNnLzEZ/YZ8xWNkpfjzsAJBLJ7FSSOO6w4TH\nw7ZMdtRf2CuFDGqNY2E3weoomRXdexhipTwW9imfRg9DfINh9EN/YV/yWbHvtpBNJcaVtxc2Ss4n\nKckee9mAVsxqMYN6p49uf2hV7AZYHSWzDgeF2QKZF93LrRhmHvoLe+Gkn9otW7UWNqsFXy2E9XLW\n8eZp3bDNU8BK1zSvYo9e2Getoz8aLsPCzsxCe2FfkcLe8Cfsflgv5Txsnuov7Ku2/Y09Q5IdJXEW\ndpmoycLOzEJ7Yc+kEijlUp4y2YUQvjzsknUXp0/rox67CZunsmK/fthGvdM3T9in9LbDiuw9sY4J\nYedTp8witBd24KTtzg21ZhfN7sB3xb5hO326iHq7j0zKjBODsmL/8U4TgBkedsm06N5Of4B2L5zI\nXkmZhZ3xgDHCvudB2P1aHSVuRuTVO/rHCUhkxf70jQYAs4R9WnRvmIeTJMtTbJcs7MwizBD2KYOV\nnTAWdo85MZKN0elTJ5ZHORYv8qQqAAAOyklEQVTPBOSx+h+PhN20zVPgpIf8MAJB5VYM4wUzhL3o\nTdilh/2Ch7heO/L06Y6Dir1hwPQkSSqZwEohjad3LGE3ze4InBT2KAR1OZ9GcyK6l4WdWYQ5wt46\nOVjZCVu1FtY8xvXaWXMx1Nqkih04+WnJ9Io9GmG3flcOp3xy4ANKzCyMEfZuf4hmd+Dq723VWrjV\nZ38dOD596sTLXm/rPxbPjr2vXjFosPK06N5IhH3GOrKpBHJp/TfgmWAwQtgrE2PanLJdO/K9cSpZ\nL2UdVewmTE+yI//tl/NppJJG/DoBmFGxt6JpxUxbB7dhmHkY8UqUp0/dWB67/SGeOzjy7WGXbJRz\nuO6gYj9s94yq2KXl0YRZp3amt2JkHER4z9+slhALOzMPM4R9yX3Ffm3fX1zvJGulLHYWVOxCiFHF\nbo6wy4q9Ypiw59NJpJN0SlCXsqlQP5mwsDNeMEPYPVTsqqyOko2yNdR63unTZncAIczIiZHISt0k\nDztgRfdOWg2jEFS5QXrIws64QImwE9HHiOgGET2h4npu8VKxqzqcJJGzT+dFG9THkb3mvCilE8aE\nWaeTTGayh5nFLuGKnfGCqor9zwC8TdG1XFPKppBOkquEx63dphXX63AA9SKOZ5/O7rPLyF6TWjFV\nQ1sxwOnDQVYWe7jPXTaVRC6dOLUOtjoy81Ai7EKIrwOoqbiWF4joxNAHJ2yNwr8SikacrctJSnNO\nnx4aND1JUjV08xQ4LexRVcr2dfQHQ9Q5spdZQGg9diK6TERXiOjKzs6O8uvLQ0pO2VJodQSOK/Z5\neTEysteUrBgAOFfJI5NM4LabilEvRTmTwr5/1I1M2OVMX47sZZwQmrALIe4XQlwSQlxaW1tTfv2K\ni7wYGderUtjXHFTs48jerDkvypuWsvjmh96Ct/zMetRLUU4cK3aOE2CcYIQrBrA2UJ1unu61emh0\n+so87IB1+nQ5n57bY68b2GMHrE9LJgyxnmQ5n8Zhu4fhUEQS2WtfBws74wZzhL3gPJNdtSNGslHO\nzq3YTdw8NZllW3RvlIJqd+eM12FQfAOjHlV2xwcA/D8AryCiq0T0z1Rc1w3VYgYHRz30bSl4swhK\n2NcXDLWut3sgAooZFnYdKNushlEGb02r2Fe4YmfmoERhhBD3qbiOH6rjwcq9cb97FuO43mpe6RrW\ny1n85MeNmT+vd/pYyqSUOXGYYLF7yLuDwYnvhb0OGd3LrRjGCea0YsbCvrgds7VrxfUWFFfO66Uc\ndhqzT5/W232jrI6mYxf2KAVV3udhxJ8cGH0wTtidOGO2FDtiJBvlLHqD2adP64YFgJlO3IRdroMj\ne5lFsLArRE5SmpXLblpkr+ms2LLQZWTvSgTRCZPr4DYMs4gzJ+zd/hDPK4zrtTOefToj5dG06Umm\nc7JSju5w2WTFzsLOLMIYYZdVzSJhf27/CEOFcb12Flbshk1PMh17dG8Ukb0SFnbGLcYIezaVRCmb\nWijsQVkdAcsVAwA3ZlTshyzsWmGP7o1SUO3RvSzsjBOMEXbAShiMUtjl6dNZFbu1ecovSp2Qh4Oi\niOyVcMXOuMWo8rFazCy0O27XWsikEuM0RtXMmn3a7Q/R6Q9R4h67VsiKvdsfhh7ZK7FH93JkL+ME\noyr2ajGD3QXRvVu1Fi5U8oEdEpKTlCaRyY7sY9eLOLRi5DpqzR5H9jKOME7YF1XsQVkdJeul7NTo\n3uOcGH5R6kSchP3qXmv8NcPMwzhhrzW7EGL6yU8hBLZ2Axb2cg436u1TazgcR/Zyxa4TcRJ2GYXB\nws4swihhrxQy6PSHaHUHU3++37I+ygbhYZesl+Tp096J78vIXpOGbJwFZHTvUW8QubA/P9q7YWFn\nFmGUsK8uOKQUpCNGcjz79OQGquyxcytGL2R0r/w6Ksr2dXBkL7MAo4S94lTYV4NsxchJSif77OPp\nSVyxa4XdgRKlG8X+psIVO7MIo4R9HCswYwNVCvuFSoAVe2lRxc7CrhNxEdS4rIPRAzOFfYblcbvW\nwk1LWRQD3MCUFfvOqYp9ZHfkzVOtiIugxmUdjB4YKeyzLI+W1VHtcI1JcukkyrnUqYq93u4jk+S4\nVd2Ii6DK+85wZC/jAKOEvZxLIZWgmbNPg/awSzbKuVNeds5i15O4CTtX64wTjBJ2IkKlmMHeFGHv\nDYZ4bv8oFGFfL2dxvX66YueNU/1YjtnmKQs74wRVw6zfRkQ/IKKnieiDKq7plWphehCYjOsN0sMu\n2SidrtitIRss7LpRyCSRShCKmSTSEUT2SljYGTf4/k0loiSAPwbwdgC3A7iPiG73e12vVIrpqcIe\nhoddslbOYqfeOXH6tN7u8caphsjo3qgFlYWdcYOKEuQeAE8LIX4ihOgC+CSAdyu4ridWi9mpdscw\nPOySjVIO3cEQ+7bTp/U2j8XTleV8OvJExTILO+MCFcJ+DsC27c9XR9+LhHkVeyaZGPvMg0RaHu19\n9joP2dCWm5ayuGkpmJhnp+TS1iAZebqaYeYRmtIQ0WUAlwFgc3MzsPupFrM4OOqhPxieGGO2XWvh\nfDW4uF47x7ECHbzyJdb36u0eZ7Fryh/ce0covzeL+LNff20oe0SM/qio2K8BuGD78/nR904ghLhf\nCHFJCHFpbW1Nwd1Op1qwMjX2j06GcIVldQQwHuIhR+QJIUabp/wxWkdevlHCy9aWol4GfvbW6niu\nLsPMQ4WwPwzg5UR0GxFlALwHwOcVXNcT1dFH5knLY9BxvXYmh1q3ugMMBefEMAwTDr6VRgjRJ6Lf\nBPDXAJIAPiaE+J7vlXmkWjgdBHbQ6uGw3Q9N2PMZ6/SprNg5J4ZhmDBRojRCiL8C8FcqruWXStFq\nd9iFfRz+FWJ/cr2cw/WRl10mO3IrhmGYMDDq5Clg2R2BkwmPz9aaAMLxsEs2ylncGLliDuVYPN48\nZRgmBIwT9nHF3oi4Yi8dV+zH805Z2BmGCR7jhD2bSmIpmzpRsVtxvZlQT36u206fjiN7WdgZhgkB\n44QdOH1IaavWCt3/u247fdrocI+dYZjwMFLYq8XsKWEPs78OWD12wLI81rkVwzBMiJgp7IXjit2K\n622HLuzrthF5cvO0mGFhZxgmeMwU9mJ2fEDp+f02BkMReitGVuzXD9totPtYyqaQjMGxdIZhzMdQ\nYU+PN0/DjOu1Yz99ypG9DMOEiZHCXilm0O4N0er2IxP2fCaJ0uj0KQ/ZYBgmTIwUdhltWmt2j+N6\ny+GHJ22Uc+PNUxZ2hmHCwki1qdjyYrZrLZyv5CPpb6+Xsrh+aPX4lwuco80wTDiYWbEvnazYo8qw\nHlfs3IphGCZEjBR2e8UehYddsl7K4sZhB4dHfc6JYRgmNIwUdhkE9syLTRwc9aIT9rJ1+nS32eGK\nnWGY0DBS2Es5yzP+2NUDAOGGf9mRk5SE4DgBhmHCw0hhTyQIlUIa39neBxC+1VFid+Kwj51hmLAw\nUtgBoFrM4GA093RzNdqKHeCcGIZhwsNYYZcbqKvFcON67ayXWdgZhgkfY4VdWh6j6q8DQCGTGrth\nuMfOMExY+BJ2IvpHRPQ9IhoS0SVVi1KBrNij6q9LZNXOFTvDMGHht2J/AsDfB/B1BWtRiowViFrY\n5QYqb54yDBMWvtRGCPEUABDFL462EhNhlxuo3IphGCYsjO2xV4vR99iB44qdWzEMw4TFQrUhov8F\n4CVTfvR7QojPOb0jIroM4DIAbG5uOl6gV/7uT6/h8hteirtvXQn8vubxD372PCrFDHLpZKTrYBjm\n7EBCCP8XIfoagN8RQlxxcvtLly6JK1cc3ZRhGIYZQUSPCCEWGlWMbcUwDMOcVfzaHe8loqsAXgfg\ni0T012qWxTAMw3jFryvmQQAPKloLwzAMowBuxTAMwxgGCzvDMIxhsLAzDMMYBgs7wzCMYbCwMwzD\nGIaSA0qu75RoB8CzHv/6TQBeVLicOGL6Y+THpz+mP8a4Pr5bhRBri24UibD7gYiuODl5pTOmP0Z+\nfPpj+mPU/fFxK4ZhGMYwWNgZhmEMQ0dhvz/qBYSA6Y+RH5/+mP4YtX582vXYGYZhmPnoWLEzDMMw\nc9BK2InobUT0AyJ6mog+GPV6VENEzxDR40T0GBEZEVhPRB8johtE9ITte1Ui+goR/Wj0/0qUa/TD\njMf3+0R0bfQ8PkZEvxTlGv1ARBeI6KtE9ORocP0HRt834jmc8/i0fg61acUQURLADwH8IoCrAB4G\ncJ8Q4slIF6YQInoGwCUhRBz9s54gojcAaAD4CyHEHaPv/QcANSHEvx+9QVeEEL8b5Tq9MuPx/T6A\nhhDiI1GuTQVEdDOAm4UQjxJRCcAjAH4ZwPtgwHM45/H9CjR+DnWq2O8B8LQQ4idCiC6ATwJ4d8Rr\nYhYghPg6gNrEt98N4M9HX/85rBeSlsx4fMYghHheCPHo6Os6gKcAnIMhz+Gcx6c1Ogn7OQDbtj9f\nhQFPwAQCwJeJ6JHRjFhT2RBCPD/6+gUAG1EuJiB+k4i+O2rVaNmmmISILgK4C8BDMPA5nHh8gMbP\noU7CfhZ4vRDibgBvB/D+0cd8oxFWL1CPfqBz/gTAywC8BsDzAP5jtMvxDxEtAfg0gN8WQhzaf2bC\nczjl8Wn9HOok7NcAXLD9+fzoe8YghLg2+v8NWJOp7ol2RYFxfdTblD3OGxGvRylCiOtCiIEQYgjg\nP0Pz55GI0rBE7+NCiM+Mvm3Mczjt8en+HOok7A8DeDkR3UZEGQDvAfD5iNekDCIqjjZvQERFAG8F\n8MT8v6Utnwfwa6Ovfw3A5yJci3Kk4I24Fxo/j0REAP4UwFNCiD+0/ciI53DW49P9OdTGFQMAI8vR\nHwFIAviYEOIPIl6SMojopTieH5sC8AkTHh8RPQDgjbDS8q4D+DcAPgvgLwFswkr5/BUhhJYbkDMe\n3xthfYQXAJ4B8M9t/WitIKLXA/i/AB4HMBx9+0Ow+tDaP4dzHt990Pg51ErYGYZhmMXo1IphGIZh\nHMDCzjAMYxgs7AzDMIbBws4wDGMYLOwMwzCGwcLOMAxjGCzsDMMwhsHCzjAMYxj/H3tbTEBm7tAP\nAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "tags": []
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.plot(rewards);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "g3_rGfRhcMFy"
   },
   "source": [
    "## Check the agent in action\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "collapsed": true,
    "id": "bVYJjt_yctdJ"
   },
   "outputs": [],
   "source": [
    "test_env = make_env()\n",
    "test_env = TextWorldPreproc(test_env)\n",
    "test_agent = DQNAgent(net, prep, epsilon=0, device=device)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "collapsed": true,
    "id": "itphK_4tdu_w"
   },
   "outputs": [],
   "source": [
    "total_reward = 0\n",
    "step_idx = 0\n",
    "s = test_env.reset()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 119
    },
    "colab_type": "code",
    "collapsed": false,
    "id": "HTsk2hBFdzj5",
    "outputId": "3c420a19-ca85-46ac-907e-2b545b2233b9"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1: unlock gate with keycard -> reward=1, total_reward=1\n",
      "2: open gate -> reward=1, total_reward=2\n",
      "3: go east -> reward=1, total_reward=3\n",
      "4: take cookie from counter -> reward=1, total_reward=4\n",
      "5: take shadfly from counter -> reward=0, total_reward=4\n",
      "6: eat cookie -> reward=2, total_reward=6\n"
     ]
    }
   ],
   "source": [
    "while True:\n",
    "    step_idx += 1\n",
    "    actions, _ = test_agent([s])\n",
    "    action = actions[0]\n",
    "    action_text = test_env.last_admissible_commands[action]\n",
    "    s, r, is_done, _ = test_env.step(action)\n",
    "    total_reward += r\n",
    "    print(\"%d: %s -> reward=%s, total_reward=%s\" % (step_idx, action_text, r, total_reward))\n",
    "    if is_done:\n",
    "        break"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "mngpKw3WfKrh"
   },
   "source": [
    "# Further steps to explore\n",
    "\n",
    "Lots of!\n",
    "\n",
    "*  We're feeding one command at a time in our DQN, maybe, [Pointer networks (1506.03134)](https://arxiv.org/abs/1506.03134) will be more suitable model for this. (Pointer Networks is a tweak to attention-based model to produce variable-sized outputs)\n",
    "*   We've simplified our life a lot with `intermediate_rewards`, can we solve the game without it? (very likely more advanced exploration will be needed)\n",
    "* Lots of opportunities to check different exploration strategies (curiosity-driven exploration, for example)\n",
    "* Solve one specific game is boring, by tweaking `TextWorldPreproc` we can combine several games together to sample from different games.\n",
    "* What about transfer learning? Can we train on one set of games to solve another games?\n",
    "* We're using `admissible_commands`, which looks like a cheating. Can you train proper command generator?\n",
    "\n",
    "And, finally, could you train agent to solve zork?\n"
   ]
  }
 ],
 "metadata": {
  "accelerator": "GPU",
  "colab": {
   "collapsed_sections": [
    "eyqqc-mhkxuL",
    "zThuJ03sDIOE",
    "XwNvArlkZoWY",
    "o4MlZBeJZLWm"
   ],
   "name": "RLSS19-TextWorld.ipynb",
   "provenance": [],
   "toc_visible": true,
   "version": "0.3.2"
  },
  "kernelspec": {
   "display_name": "Python [default]",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}