Lab24_distribute_Concurrency3Lab.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Crawlers\n",
    "\n",
    "Code for this lab is almost entirely taken and modified from Brent Slatkin's Pycon 2014 talk, since it provides a beautiful illustration of the entire process."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Synchronous Blocking Crawler\n",
    "\n",
    "This code, taken from Brent's talk, is provided to you as an example of a synxhronous, single-threaded crawler you will make async"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from urllib.parse import urljoin\n",
    "from urllib.parse import urlparse\n",
    "from urllib.parse import urlunparse\n",
    "import re\n",
    "import requests\n",
    "URL_EXPR = re.compile(\n",
    "    '([a-zA-Z]+\\s*=\\s*[\"\\'])'   # Tag attribute: href=\"\n",
    "    '(?P<url>'\n",
    "        '((http(s?):)?'         # Optional scheme\n",
    "        '//[^\"\\'\\s\\\\\\\\</]+)?'   # Optional domain\n",
    "        '/[^\"\\'\\s\\\\\\\\<]*'       # Required path\n",
    "    ')')\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def canonicalize(url):\n",
    "    parts = list(urlparse(url))\n",
    "    if parts[2] == '':\n",
    "        parts[2] = '/'  # Empty path equals root path\n",
    "    parts[5] = ''       # Erase fragment\n",
    "    return urlunparse(parts)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Notice the quick and dirty use of assert's here to throw exceptions if something goes wrong. The calling code should catch generic exceptions."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def fetch(url):\n",
    "    print(\"Doing\", url)\n",
    "    response = requests.get(url)\n",
    "    assert response.status_code == 200\n",
    "    data = response.content#get as bytes\n",
    "    assert data\n",
    "    return data.decode('utf-8')\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Doing http://www.xkcd.com/353\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "'<!DOCTYPE html>\\n<html>\\n<head>\\n<link rel=\"stylesheet\" type=\"text/css\" href=\"/s/b0dcca.css\" title=\"Default\"/>\\n<title>xkcd: Python</title>\\n<meta http-equiv=\"X-UA-Compatible\" content=\"IE=edge\"/>\\n<link rel=\"shortcut icon\" href=\"/s/919f27.ico\" type=\"image/x-icon\"/>\\n<link rel=\"icon\" href=\"/s/919f27.ico\" type=\"image/x-icon\"/>\\n<link rel=\"alternate\" type=\"application/atom+xml\" title=\"Atom 1.0\" href=\"/atom.xml\"/>\\n<link rel=\"alternate\" type=\"application/rss+xml\" title=\"RSS 2.0\" href=\"/rss.xml\"/>\\n<script>\\n(function(i,s,o,g,r,a,m){i[\\'GoogleAnalyticsObject\\']=r;i[r]=i[r]||function(){\\n(i[r].q=i[r].q||[]).push(arguments)},i[r].l=1*new Date();a=s.createElement(o),\\nm=s.getElementsByTagName(o)[0];a.async=1;a.src=g;m.parentNode.insertBefore(a,m)\\n})(window,document,\\'script\\',\\'//www.google-analytics.com/analytics.js\\',\\'ga\\');\\n\\nga(\\'create\\', \\'UA-25700708-7\\', \\'auto\\');\\nga(\\'send\\', \\'pageview\\');\\n</script>\\n<script type=\"text/javascript\" src=\"//xkcd.com/1350/jquery.min.js\"></script>\\n<script type=\"text/javascript\" src=\"//xkcd.com/static/json2.js\"></script>\\n\\n</head>\\n<body>\\n<div id=\"topContainer\">\\n<div id=\"topLeft\">\\n<ul>\\n<li><a href=\"/archive\">Archive</a></li>\\n<li><a href=\"http://what-if.xkcd.com\">What If?</a></li>\\n<li><a href=\"http://blag.xkcd.com\">Blag</a></li>\\n<li><a href=\"http://store.xkcd.com/\">Store</a></li>\\n<li><a rel=\"author\" href=\"/about\">About</a></li>\\n</ul>\\n</div>\\n<div id=\"topRight\">\\n<div id=\"masthead\">\\n<span><a href=\"/\"><img src=\"//imgs.xkcd.com/static/terrible_small_logo.png\" alt=\"xkcd.com logo\" height=\"83\" width=\"185\"/></a></span>\\n<span id=\"slogan\">A webcomic of romance,<br/> sarcasm, math, and language.</span>\\n</div>\\n<div id=\"news\">\\n<b>Protip:</b> If you don\\'t like how your garden is growing, you can click to prune it.<br />\\nYou can copy the URL to share your garden. From other browsers, it will be view-only.\\n\\n\\n\\n</div>\\n</div>\\n<div id=\"bgLeft\" class=\"bg box\"></div>\\n<div id=\"bgRight\" class=\"bg box\"></div>\\n</div>\\n<div id=\"middleContainer\" class=\"box\">\\n\\n<div id=\"ctitle\">Python</div>\\n<ul class=\"comicNav\">\\n<li><a href=\"/1/\">|&lt;</a></li>\\n<li><a rel=\"prev\" href=\"/352/\" accesskey=\"p\">&lt; Prev</a></li>\\n<li><a href=\"//c.xkcd.com/random/comic/\">Random</a></li>\\n<li><a rel=\"next\" href=\"/354/\" accesskey=\"n\">Next &gt;</a></li>\\n<li><a href=\"/\">&gt;|</a></li>\\n</ul>\\n<div id=\"comic\">\\n<img src=\"//imgs.xkcd.com/comics/python.png\" title=\"I wrote 20 short programs in Python yesterday.  It was wonderful.  Perl, I&#39;m leaving you.\" alt=\"Python\" />\\n</div>\\n<ul class=\"comicNav\">\\n<li><a href=\"/1/\">|&lt;</a></li>\\n<li><a rel=\"prev\" href=\"/352/\" accesskey=\"p\">&lt; Prev</a></li>\\n<li><a href=\"//c.xkcd.com/random/comic/\">Random</a></li>\\n<li><a rel=\"next\" href=\"/354/\" accesskey=\"n\">Next &gt;</a></li>\\n<li><a href=\"/\">&gt;|</a></li>\\n</ul>\\n<br />\\nPermanent link to this comic: http://xkcd.com/353/<br />\\nImage URL (for hotlinking/embedding): http://imgs.xkcd.com/comics/python.png\\n<div id=\"transcript\" style=\"display: none\">[[ Guy 1 is talking to Guy 2, who is floating in the sky ]]\\nGuy 1: You&#39;re flying! How?\\nGuy 2: Python!\\nGuy 2: I learned it last night! Everything is so simple!\\nGuy 2: Hello world is just &#39;print &quot;Hello, World!&quot; &#39;\\nGuy 1: I dunno... Dynamic typing? Whitespace?\\nGuy 2: Come join us! Programming is fun again! It&#39;s a whole new world up here!\\nGuy 1: But how are you flying?\\nGuy 2: I just typed &#39;import antigravity&#39;\\nGuy 1: That&#39;s it?\\nGuy 2: ...I also sampled everything in the medicine cabinet for comparison.\\nGuy 2: But i think this is the python.\\n{{ I wrote 20 short programs in Python yesterday.  It was wonderful.  Perl, I&#39;m leaving you. }}</div>\\n<br />\\n<hr width=\"80%\" />\\n<br />\\n<a href=\"http://amzn.to/1GCXMJ5\"><img border=0 src=\"//imgs.xkcd.com/store/te-pages-sb.png\"></a><br />\\n</div>\\n<div id=\"bottom\" class=\"box\">\\n<img src=\"//imgs.xkcd.com/s/a899e84.jpg\" width=\"520\" height=\"100\" alt=\"Selected Comics\" usemap=\"#comicmap\"/>\\n<map id=\"comicmap\" name=\"comicmap\">\\n<area shape=\"rect\" coords=\"0,0,100,100\" href=\"/150/\" alt=\"Grownups\"/>\\n<area shape=\"rect\" coords=\"104,0,204,100\" href=\"/730/\" alt=\"Circuit Diagram\"/>\\n<area shape=\"rect\" coords=\"208,0,308,100\" href=\"/162/\" alt=\"Angular Momentum\"/>\\n<area shape=\"rect\" coords=\"312,0,412,100\" href=\"/688/\" alt=\"Self-Description\"/>\\n<area shape=\"rect\" coords=\"416,0,520,100\" href=\"/556/\" alt=\"Alternative Energy Revolution\"/>\\n</map>\\n<div>\\n<!--\\nSearch comic titles and transcripts:\\n<script type=\"text/javascript\" src=\"//www.google.com/jsapi\"></script>\\n<script type=\"text/javascript\">google.load(\\'search\\', \\'1\\');google.setOnLoadCallback(function() {google.search.CustomSearchControl.attachAutoCompletion(\\'012652707207066138651:zudjtuwe28q\\',document.getElementById(\\'q\\'),\\'cse-search-box\\');});</script>\\n<form action=\"//www.google.com/cse\" id=\"cse-search-box\">\\n<div>\\n<input type=\"hidden\" name=\"cx\" value=\"012652707207066138651:zudjtuwe28q\"/>\\n<input type=\"hidden\" name=\"ie\" value=\"UTF-8\"/>\\n<input type=\"text\" name=\"q\" id=\"q\" size=\"31\"/>\\n<input type=\"submit\" name=\"sa\" value=\"Search\"/>\\n</div>\\n</form>\\n<script type=\"text/javascript\" src=\"//www.google.com/cse/brand?form=cse-search-box&amp;lang=en\"></script>\\n-->\\n<a href=\"/rss.xml\">RSS Feed</a> - <a href=\"/atom.xml\">Atom Feed</a>\\n</div>\\n<br />\\n<div id=\"comicLinks\">\\nComics I enjoy:<br/>\\n        <a href=\"http://threewordphrase.com/\">Three Word Phrase</a>,\\n        <a href=\"http://www.smbc-comics.com/\">SMBC</a>,\\n        <a href=\"http://www.qwantz.com\">Dinosaur Comics</a>,\\n        <a href=\"http://oglaf.com/\">Oglaf</a> (nsfw),\\n        <a href=\"http://www.asofterworld.com\">A Softer World</a>,\\n        <a href=\"http://buttersafe.com/\">Buttersafe</a>,\\n        <a href=\"http://pbfcomics.com/\">Perry Bible Fellowship</a>,\\n        <a href=\"http://questionablecontent.net/\">Questionable Content</a>,\\n        <a href=\"http://www.buttercupfestival.com/\">Buttercup Festival</a>,\\n        <a href=\"http://www.mspaintadventures.com/?s=6&p=001901\">Homestuck</a>,\\n\\t<a href=\"http://www.jspowerhour.com/\">Junior Scientist Power Hour</a>\\n</div>\\n<p>Warning: this comic occasionally contains strong language (which may be unsuitable for children), unusual humor (which may be unsuitable for adults), and advanced mathematics (which may be unsuitable for liberal-arts majors).</p>\\n<div id=\"footnote\">BTC 1FhCLQK2ZXtCUQDtG98p6fVH7S6mxAsEey<br />We did not invent the algorithm. The algorithm consistently finds Jesus. The algorithm killed Jeeves. <br/>The algorithm is banned in China. The algorithm is from Jersey. The algorithm constantly finds Jesus.<br/>This is not the algorithm. This is close.</div>\\n<div id=\"licenseText\">\\n<p>\\nThis work is licensed under a\\n<a href=\"http://creativecommons.org/licenses/by-nc/2.5/\">Creative Commons Attribution-NonCommercial 2.5 License</a>.\\n</p><p>\\nThis means you\\'re free to copy and share these comics (but not to sell them). <a rel=\"license\" href=\"/license.html\">More details</a>.</p>\\n</div>\\n</div>\\n</body>\\n<!-- Layout by Ian Clasbey, davean, and chromakode -->\\n</html>\\n\\n'"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "fetch(\"http://www.xkcd.com/353\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For simplicity, we keep to the same site for now. You can pass over this code, it just extracts urls on the same domain from the page using regular expressions."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def same_domain(a, b):\n",
    "    parsed_a = urlparse(a)\n",
    "    parsed_b = urlparse(b)\n",
    "    if parsed_a.netloc == parsed_b.netloc:\n",
    "        return True\n",
    "    if (parsed_a.netloc == '') ^ (parsed_b.netloc == ''):  # Relative paths\n",
    "        return True\n",
    "    return False"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def extract(url):\n",
    "    data = fetch(url)\n",
    "    found_urls = set()\n",
    "    for match in URL_EXPR.finditer(data):\n",
    "        found = canonicalize(match.group('url'))\n",
    "        if same_domain(url, found):\n",
    "            found_urls.add(urljoin(url, found))\n",
    "    return url, len(data), sorted(found_urls)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Doing http://www.xkcd.com/353\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "['http://www.xkcd.com/',\n",
       " 'http://www.xkcd.com/1/',\n",
       " 'http://www.xkcd.com/150/',\n",
       " 'http://www.xkcd.com/162/',\n",
       " 'http://www.xkcd.com/352/',\n",
       " 'http://www.xkcd.com/354/',\n",
       " 'http://www.xkcd.com/556/',\n",
       " 'http://www.xkcd.com/688/',\n",
       " 'http://www.xkcd.com/730/',\n",
       " 'http://www.xkcd.com/about',\n",
       " 'http://www.xkcd.com/archive',\n",
       " 'http://www.xkcd.com/atom.xml',\n",
       " 'http://www.xkcd.com/license.html',\n",
       " 'http://www.xkcd.com/rss.xml',\n",
       " 'http://www.xkcd.com/s/919f27.ico',\n",
       " 'http://www.xkcd.com/s/b0dcca.css']"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "extract(\"http://www.xkcd.com/353\")[2]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "def extract_multi(to_fetch, seen_urls):\n",
    "    results = []\n",
    "    for url in to_fetch:\n",
    "        if url in seen_urls: \n",
    "            continue\n",
    "        seen_urls.add(url)\n",
    "        try:\n",
    "            results.append(extract(url))\n",
    "        except Exception:\n",
    "            continue\n",
    "    return results\n",
    "\n",
    "\n",
    "def crawl(start_url, max_depth=1):\n",
    "    seen_urls = set()\n",
    "    to_fetch = [canonicalize(start_url)]\n",
    "    results = []\n",
    "    for depth in range(max_depth + 1):\n",
    "        batch = extract_multi(to_fetch, seen_urls)\n",
    "        to_fetch = []\n",
    "        for url, datalen, found_urls in batch:\n",
    "            results.append((depth, url, datalen))\n",
    "            to_fetch.extend(found_urls)\n",
    "\n",
    "    return results"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Doing http://www.xkcd.com/353\n",
      "Doing http://www.xkcd.com/\n",
      "Doing http://www.xkcd.com/1/\n",
      "Doing http://www.xkcd.com/150/\n",
      "Doing http://www.xkcd.com/162/\n",
      "Doing http://www.xkcd.com/352/\n",
      "Doing http://www.xkcd.com/354/\n",
      "Doing http://www.xkcd.com/556/\n",
      "Doing http://www.xkcd.com/688/\n",
      "Doing http://www.xkcd.com/730/\n",
      "Doing http://www.xkcd.com/about\n",
      "Doing http://www.xkcd.com/archive\n",
      "Doing http://www.xkcd.com/atom.xml\n",
      "Doing http://www.xkcd.com/license.html\n",
      "Doing http://www.xkcd.com/rss.xml\n",
      "Doing http://www.xkcd.com/s/919f27.ico\n",
      "Doing http://www.xkcd.com/s/b0dcca.css\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "[(0, 'http://www.xkcd.com/353', 6924),\n",
       " (1, 'http://www.xkcd.com/', 6219),\n",
       " (1, 'http://www.xkcd.com/1/', 6399),\n",
       " (1, 'http://www.xkcd.com/150/', 6904),\n",
       " (1, 'http://www.xkcd.com/162/', 6914),\n",
       " (1, 'http://www.xkcd.com/352/', 6540),\n",
       " (1, 'http://www.xkcd.com/354/', 6400),\n",
       " (1, 'http://www.xkcd.com/556/', 7977),\n",
       " (1, 'http://www.xkcd.com/688/', 7319),\n",
       " (1, 'http://www.xkcd.com/730/', 11800),\n",
       " (1, 'http://www.xkcd.com/about', 7649),\n",
       " (1, 'http://www.xkcd.com/archive', 103648),\n",
       " (1, 'http://www.xkcd.com/atom.xml', 2092),\n",
       " (1, 'http://www.xkcd.com/license.html', 2558),\n",
       " (1, 'http://www.xkcd.com/rss.xml', 2022),\n",
       " (1, 'http://www.xkcd.com/s/b0dcca.css', 3487)]"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cr = crawl(\"http://www.xkcd.com/353\")\n",
    "cr"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1. Synchronous crawler, async style\n",
    "\n",
    "(using yield from)\n",
    "\n",
    "Just like in the lecture, let us slowly bring in the async technology, still keeping a synchronous crawler going. This means that we'll have one `yield from` after another.\n",
    "\n",
    "We write the fetcher async now:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "import asyncio, aiohttp\n",
    "\n",
    "@asyncio.coroutine\n",
    "def fetch_async(url):\n",
    "    print(\"Doing\", url)\n",
    "    response = yield from aiohttp.request('GET', url)\n",
    "    try:\n",
    "        assert response.status == 200\n",
    "        data = yield from response.read()\n",
    "        assert data\n",
    "        return data.decode('utf-8')\n",
    "    finally:\n",
    "        response.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Write the extractor"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "@asyncio.coroutine\n",
    "def extract_async(url):\n",
    "    #your code here\n",
    "    print('Extracting ' + url)\n",
    "    found_urls=set()\n",
    "    data=yield from fetch_async(url)\n",
    "    for match in URL_EXPR.finditer(data):\n",
    "        found=canonicalize(match.group('url'))\n",
    "        if same_domain(url, found):\n",
    "            found_urls.add(urljoin(url, found))\n",
    "    return url, len(data), sorted(found_urls)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We wrap the top level coroutine in a task. Since a task is a future, we can also get its result in this form."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Extracting http://www.xkcd.com/353\n",
      "Doing http://www.xkcd.com/353\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "('http://www.xkcd.com/353',\n",
       " 6924,\n",
       " ['http://www.xkcd.com/',\n",
       "  'http://www.xkcd.com/1/',\n",
       "  'http://www.xkcd.com/150/',\n",
       "  'http://www.xkcd.com/162/',\n",
       "  'http://www.xkcd.com/352/',\n",
       "  'http://www.xkcd.com/354/',\n",
       "  'http://www.xkcd.com/556/',\n",
       "  'http://www.xkcd.com/688/',\n",
       "  'http://www.xkcd.com/730/',\n",
       "  'http://www.xkcd.com/about',\n",
       "  'http://www.xkcd.com/archive',\n",
       "  'http://www.xkcd.com/atom.xml',\n",
       "  'http://www.xkcd.com/license.html',\n",
       "  'http://www.xkcd.com/rss.xml',\n",
       "  'http://www.xkcd.com/s/919f27.ico',\n",
       "  'http://www.xkcd.com/s/b0dcca.css'])"
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "future = asyncio.Task(extract_async('http://www.xkcd.com/353'))\n",
    "#future = extract_async('http://www.xkcd.com/353')\n",
    "#you could do the above but could not access the result as \n",
    "#future.result()\n",
    "\n",
    "loop = asyncio.get_event_loop()\n",
    "loop.run_until_complete(future)\n",
    "#loop.close() ONLY DO IF NOT IN REPL OR YOU WILL BE HOSED\n",
    "future.result()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2. Write the multi-extractor and crawler\n",
    "\n",
    "Note that you are writing the multi-extractor using async syntax but the `yield from`s are serialized."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "@asyncio.coroutine\n",
    "def extract_multi_async(to_fetch, seen_urls):\n",
    "    #your code here\n",
    "    results = []\n",
    "    for url in to_fetch:\n",
    "        if url in seen_urls: \n",
    "            continue\n",
    "        seen_urls.add(url)\n",
    "        try:\n",
    "            new=yield from extract_async(url)\n",
    "            results.append(new)\n",
    "        except Exception:\n",
    "            continue\n",
    "    return results\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "@asyncio.coroutine\n",
    "def crawl_async(start_url, max_depth=1):\n",
    "    #your code here\n",
    "    seen_urls = set()\n",
    "    to_fetch = [canonicalize(start_url)]\n",
    "    results = []\n",
    "    for depth in range(max_depth + 1):\n",
    "        batch = yield from extract_multi_async(to_fetch, seen_urls)\n",
    "        to_fetch = []\n",
    "        for url, datalen, found_urls in batch:\n",
    "            results.append((depth, url, datalen))\n",
    "            to_fetch.extend(found_urls)\n",
    "\n",
    "    return results"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We run the entire crawler now:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Extracting http://www.xkcd.com/353\n",
      "Doing http://www.xkcd.com/353\n",
      "Extracting http://www.xkcd.com/\n",
      "Doing http://www.xkcd.com/\n",
      "Extracting http://www.xkcd.com/1/\n",
      "Doing http://www.xkcd.com/1/\n",
      "Extracting http://www.xkcd.com/150/\n",
      "Doing http://www.xkcd.com/150/\n",
      "Extracting http://www.xkcd.com/162/\n",
      "Doing http://www.xkcd.com/162/\n",
      "Extracting http://www.xkcd.com/352/\n",
      "Doing http://www.xkcd.com/352/\n",
      "Extracting http://www.xkcd.com/354/\n",
      "Doing http://www.xkcd.com/354/\n",
      "Extracting http://www.xkcd.com/556/\n",
      "Doing http://www.xkcd.com/556/\n",
      "Extracting http://www.xkcd.com/688/\n",
      "Doing http://www.xkcd.com/688/\n",
      "Extracting http://www.xkcd.com/730/\n",
      "Doing http://www.xkcd.com/730/\n",
      "Extracting http://www.xkcd.com/about\n",
      "Doing http://www.xkcd.com/about\n",
      "Extracting http://www.xkcd.com/archive\n",
      "Doing http://www.xkcd.com/archive\n",
      "Extracting http://www.xkcd.com/atom.xml\n",
      "Doing http://www.xkcd.com/atom.xml\n",
      "Extracting http://www.xkcd.com/license.html\n",
      "Doing http://www.xkcd.com/license.html\n",
      "Extracting http://www.xkcd.com/rss.xml\n",
      "Doing http://www.xkcd.com/rss.xml\n",
      "Extracting http://www.xkcd.com/s/919f27.ico\n",
      "Doing http://www.xkcd.com/s/919f27.ico\n",
      "Extracting http://www.xkcd.com/s/b0dcca.css\n",
      "Doing http://www.xkcd.com/s/b0dcca.css\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "[(0, 'http://www.xkcd.com/353', 6924),\n",
       " (1, 'http://www.xkcd.com/', 6219),\n",
       " (1, 'http://www.xkcd.com/1/', 6399),\n",
       " (1, 'http://www.xkcd.com/150/', 6904),\n",
       " (1, 'http://www.xkcd.com/162/', 6914),\n",
       " (1, 'http://www.xkcd.com/352/', 6540),\n",
       " (1, 'http://www.xkcd.com/354/', 6400),\n",
       " (1, 'http://www.xkcd.com/556/', 7977),\n",
       " (1, 'http://www.xkcd.com/688/', 7319),\n",
       " (1, 'http://www.xkcd.com/730/', 11800),\n",
       " (1, 'http://www.xkcd.com/about', 7649),\n",
       " (1, 'http://www.xkcd.com/archive', 103648),\n",
       " (1, 'http://www.xkcd.com/atom.xml', 2092),\n",
       " (1, 'http://www.xkcd.com/license.html', 2558),\n",
       " (1, 'http://www.xkcd.com/rss.xml', 2022),\n",
       " (1, 'http://www.xkcd.com/s/b0dcca.css', 3487)]"
      ]
     },
     "execution_count": 36,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "future = asyncio.Task(crawl_async('http://www.xkcd.com/353', max_depth=1))\n",
    "loop = asyncio.get_event_loop()\n",
    "loop.run_until_complete(future)\n",
    "future.result()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "###  3. Asynchronous crawler with `async def` and `await`: Many simultaneous fetches\n",
    "\n",
    "Rewrite all the code here. You will need to make two changes:\n",
    "\n",
    "1. `yield from` -> `await`, decorator -> `async def`\n",
    "2. note that `extract_multi_async` upstairs was seriealized. Use futures from `asyncio.as_completed` to change this.\n",
    "\n",
    "The first two are just copied over"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 85,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "async def fetch_async(url):\n",
    "    #your code here\n",
    "    print('Fetching ' + url)\n",
    "    response=await aiohttp.request('GET', url)\n",
    "    try:\n",
    "        assert response.status_code==200\n",
    "        data=await response.read()\n",
    "        assert data\n",
    "        return data.decode('utf-8')\n",
    "    finally:\n",
    "        response.close()\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 86,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "async def extract_async(url):\n",
    "    #your code here\n",
    "    print('Extracting ' + url)\n",
    "    data = await fetch_async(url)\n",
    "    found_urls = set()\n",
    "    for match in URL_EXPR.finditer(data):\n",
    "        found = canonicalize(match.group('url'))\n",
    "        if same_domain(url, found):\n",
    "            found_urls.add(urljoin(url, found))\n",
    "    return url, len(data), sorted(found_urls)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Surprisingly, one of these next two is unchanged except for the syntax. Which one? "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 93,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "async def extract_multi_async(to_fetch, seen_urls):\n",
    "    #your code here\n",
    "    results = []\n",
    "    for url in asyncio.as_completed([extract_async(url) for url in to_fetch]):\n",
    "        if url in seen_urls: \n",
    "            continue\n",
    "        seen_urls.add(url)\n",
    "        try:\n",
    "            new_result = await url\n",
    "            results.append(new_result)\n",
    "        except Exception:\n",
    "            continue\n",
    "    return results"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 94,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "async def crawl_async(start_url, max_depth=1):\n",
    "    #your code here\n",
    "    seen_urls = set()\n",
    "    to_fetch = [canonicalize(start_url)]\n",
    "    results = []\n",
    "    for depth in range(max_depth + 1):\n",
    "        batch = await extract_multi_async(to_fetch, seen_urls)\n",
    "        to_fetch = []\n",
    "        for url, datalen, found_urls in batch:\n",
    "            results.append((depth, url, datalen))\n",
    "            to_fetch.extend(found_urls)\n",
    "    return results"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 95,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Extracting http://www.xkcd.com/353\n",
      "Fetching http://www.xkcd.com/353\n",
      "Extracting http://www.xkcd.com/150/\n",
      "Fetching http://www.xkcd.com/150/\n",
      "Extracting http://www.xkcd.com/about\n",
      "Fetching http://www.xkcd.com/about\n",
      "Extracting http://www.xkcd.com/license.html\n",
      "Fetching http://www.xkcd.com/license.html\n",
      "Extracting http://www.xkcd.com/1/\n",
      "Fetching http://www.xkcd.com/1/\n",
      "Extracting http://www.xkcd.com/archive\n",
      "Fetching http://www.xkcd.com/archive\n",
      "Extracting http://www.xkcd.com/s/b0dcca.css\n",
      "Fetching http://www.xkcd.com/s/b0dcca.css\n",
      "Extracting http://www.xkcd.com/162/\n",
      "Fetching http://www.xkcd.com/162/\n",
      "Extracting http://www.xkcd.com/688/\n",
      "Fetching http://www.xkcd.com/688/\n",
      "Extracting http://www.xkcd.com/730/\n",
      "Fetching http://www.xkcd.com/730/\n",
      "Extracting http://www.xkcd.com/354/\n",
      "Fetching http://www.xkcd.com/354/\n",
      "Extracting http://www.xkcd.com/556/\n",
      "Fetching http://www.xkcd.com/556/\n",
      "Extracting http://www.xkcd.com/\n",
      "Fetching http://www.xkcd.com/\n",
      "Extracting http://www.xkcd.com/rss.xml\n",
      "Fetching http://www.xkcd.com/rss.xml\n",
      "Extracting http://www.xkcd.com/352/\n",
      "Fetching http://www.xkcd.com/352/\n",
      "Extracting http://www.xkcd.com/s/919f27.ico\n",
      "Fetching http://www.xkcd.com/s/919f27.ico\n",
      "Extracting http://www.xkcd.com/atom.xml\n",
      "Fetching http://www.xkcd.com/atom.xml\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "[(0, 'http://www.xkcd.com/353', 6924),\n",
       " (1, 'http://www.xkcd.com/150/', 6904),\n",
       " (1, 'http://www.xkcd.com/license.html', 2558),\n",
       " (1, 'http://www.xkcd.com/s/b0dcca.css', 3487),\n",
       " (1, 'http://www.xkcd.com/1/', 6399),\n",
       " (1, 'http://www.xkcd.com/162/', 6914),\n",
       " (1, 'http://www.xkcd.com/354/', 6400),\n",
       " (1, 'http://www.xkcd.com/', 6219),\n",
       " (1, 'http://www.xkcd.com/556/', 7977),\n",
       " (1, 'http://www.xkcd.com/730/', 11800),\n",
       " (1, 'http://www.xkcd.com/688/', 7319),\n",
       " (1, 'http://www.xkcd.com/352/', 6540),\n",
       " (1, 'http://www.xkcd.com/rss.xml', 2022),\n",
       " (1, 'http://www.xkcd.com/atom.xml', 2092),\n",
       " (1, 'http://www.xkcd.com/about', 7649),\n",
       " (1, 'http://www.xkcd.com/archive', 103648)]"
      ]
     },
     "execution_count": 95,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "future = asyncio.Task(crawl_async('http://www.xkcd.com/353', max_depth=1))\n",
    "loop = asyncio.get_event_loop()\n",
    "loop.run_until_complete(future)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 4. Concurrent Crawls\n",
    "\n",
    "We can even do concurrent crawls to multiple web sites. Implement this."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 96,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "urls = ['http://www.xkcd.com/353', 'http://what-if.xkcd.com/148/']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 97,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "async def crawl_multi_async(urls):\n",
    "    #your code here\n",
    "    results=[]\n",
    "    for url in asyncio.as_completed([crawl_async(url) for url in urls]):\n",
    "        results.append(await url)\n",
    "    return results\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 98,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Extracting http://www.xkcd.com/353\n",
      "Fetching http://www.xkcd.com/353\n",
      "Extracting http://what-if.xkcd.com/148/\n",
      "Fetching http://what-if.xkcd.com/148/\n",
      "Extracting http://what-if.xkcd.com/imgs/a/148/franchises.png\n",
      "Fetching http://what-if.xkcd.com/imgs/a/148/franchises.png\n",
      "Extracting http://what-if.xkcd.com/\n",
      "Fetching http://what-if.xkcd.com/\n",
      "Extracting http://what-if.xkcd.com/archive\n",
      "Fetching http://what-if.xkcd.com/archive\n",
      "Extracting http://what-if.xkcd.com/imgs/a/148/snakemeat.png\n",
      "Fetching http://what-if.xkcd.com/imgs/a/148/snakemeat.png\n",
      "Extracting http://what-if.xkcd.com/imgs/favicon.ico\n",
      "Fetching http://what-if.xkcd.com/imgs/favicon.ico\n",
      "Extracting http://what-if.xkcd.com/imgs/a/148/actualsize.png\n",
      "Fetching http://what-if.xkcd.com/imgs/a/148/actualsize.png\n",
      "Extracting http://what-if.xkcd.com/imgs/a/148/easy.png\n",
      "Fetching http://what-if.xkcd.com/imgs/a/148/easy.png\n",
      "Extracting http://what-if.xkcd.com/148/\n",
      "Fetching http://what-if.xkcd.com/148/\n",
      "Extracting https://what-if.xkcd.com/96/\n",
      "Fetching https://what-if.xkcd.com/96/\n",
      "Extracting http://what-if.xkcd.com/imgs/apple-touch-icon.png\n",
      "Fetching http://what-if.xkcd.com/imgs/apple-touch-icon.png\n",
      "Extracting http://what-if.xkcd.com/css/style.css\n",
      "Fetching http://what-if.xkcd.com/css/style.css\n",
      "Extracting http://what-if.xkcd.com/feed.atom\n",
      "Fetching http://what-if.xkcd.com/feed.atom\n",
      "Extracting http://what-if.xkcd.com/147/\n",
      "Fetching http://what-if.xkcd.com/147/\n",
      "Extracting http://what-if.xkcd.com/imgs/whatif-logo.png\n",
      "Fetching http://what-if.xkcd.com/imgs/whatif-logo.png\n",
      "Extracting http://www.xkcd.com/162/\n",
      "Fetching http://www.xkcd.com/162/\n",
      "Extracting http://www.xkcd.com/730/\n",
      "Fetching http://www.xkcd.com/730/\n",
      "Extracting http://www.xkcd.com/150/\n",
      "Fetching http://www.xkcd.com/150/\n",
      "Extracting http://www.xkcd.com/atom.xml\n",
      "Fetching http://www.xkcd.com/atom.xml\n",
      "Extracting http://www.xkcd.com/license.html\n",
      "Fetching http://www.xkcd.com/license.html\n",
      "Extracting http://www.xkcd.com/352/\n",
      "Fetching http://www.xkcd.com/352/\n",
      "Extracting http://www.xkcd.com/1/\n",
      "Fetching http://www.xkcd.com/1/\n",
      "Extracting http://www.xkcd.com/about\n",
      "Fetching http://www.xkcd.com/about\n",
      "Extracting http://www.xkcd.com/688/\n",
      "Fetching http://www.xkcd.com/688/\n",
      "Extracting http://www.xkcd.com/rss.xml\n",
      "Fetching http://www.xkcd.com/rss.xml\n",
      "Extracting http://www.xkcd.com/s/919f27.ico\n",
      "Fetching http://www.xkcd.com/s/919f27.ico\n",
      "Extracting http://www.xkcd.com/\n",
      "Fetching http://www.xkcd.com/\n",
      "Extracting http://www.xkcd.com/archive\n",
      "Fetching http://www.xkcd.com/archive\n",
      "Extracting http://www.xkcd.com/556/\n",
      "Fetching http://www.xkcd.com/556/\n",
      "Extracting http://www.xkcd.com/s/b0dcca.css\n",
      "Fetching http://www.xkcd.com/s/b0dcca.css\n",
      "Extracting http://www.xkcd.com/354/\n",
      "Fetching http://www.xkcd.com/354/\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "[[(0, 'http://what-if.xkcd.com/148/', 9414),\n",
       "  (1, 'http://what-if.xkcd.com/', 9414),\n",
       "  (1, 'http://what-if.xkcd.com/148/', 9414),\n",
       "  (1, 'http://what-if.xkcd.com/css/style.css', 9586),\n",
       "  (1, 'http://what-if.xkcd.com/147/', 12258),\n",
       "  (1, 'https://what-if.xkcd.com/96/', 10286),\n",
       "  (1, 'http://what-if.xkcd.com/feed.atom', 46115),\n",
       "  (1, 'http://what-if.xkcd.com/archive', 43541)],\n",
       " [(0, 'http://www.xkcd.com/353', 6924),\n",
       "  (1, 'http://www.xkcd.com/730/', 11800),\n",
       "  (1, 'http://www.xkcd.com/162/', 6914),\n",
       "  (1, 'http://www.xkcd.com/352/', 6540),\n",
       "  (1, 'http://www.xkcd.com/license.html', 2558),\n",
       "  (1, 'http://www.xkcd.com/150/', 6904),\n",
       "  (1, 'http://www.xkcd.com/1/', 6399),\n",
       "  (1, 'http://www.xkcd.com/atom.xml', 2092),\n",
       "  (1, 'http://www.xkcd.com/rss.xml', 2022),\n",
       "  (1, 'http://www.xkcd.com/688/', 7319),\n",
       "  (1, 'http://www.xkcd.com/', 6219),\n",
       "  (1, 'http://www.xkcd.com/354/', 6400),\n",
       "  (1, 'http://www.xkcd.com/s/b0dcca.css', 3487),\n",
       "  (1, 'http://www.xkcd.com/556/', 7977),\n",
       "  (1, 'http://www.xkcd.com/about', 7649),\n",
       "  (1, 'http://www.xkcd.com/archive', 103648)]]"
      ]
     },
     "execution_count": 98,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "future = asyncio.Task(crawl_multi_async(urls))\n",
    "loop = asyncio.get_event_loop()\n",
    "loop.run_until_complete(future)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}