C1-natural-language-processing-with-classification-and-vector-spaces/Week1/C1_W1_lecture_nb_01_preprocessing.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Preprocessing\n",
    "\n",
    "In this lab, we will be exploring how to preprocess tweets for sentiment analysis. We will provide a function for preprocessing tweets during this week's assignment, but it is still good to know what is going on under the hood. By the end of this lecture, you will see how to use the [NLTK](http://www.nltk.org) package to perform a preprocessing pipeline for Twitter datasets."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Setup\n",
    "\n",
    "You will be doing sentiment analysis on tweets in the first two weeks of this course. To help with that, we will be using the [Natural Language Toolkit (NLTK)](http://www.nltk.org/howto/twitter.html) package, an open-source Python library for natural language processing. It has modules for collecting, handling, and processing Twitter data, and you will be acquainted with them as we move along the course.\n",
    "\n",
    "For this exercise, we will use a Twitter dataset that comes with NLTK. This dataset has been manually annotated and serves to establish baselines for models quickly. Let us import them now as well as a few other libraries we will be using."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import nltk                                # Python library for NLP\n",
    "from nltk.corpus import twitter_samples    # sample Twitter dataset from NLTK\n",
    "import matplotlib.pyplot as plt            # library for visualization\n",
    "import random                              # pseudo-random number generator"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## About the Twitter dataset\n",
    "\n",
    "The sample dataset from NLTK is separated into positive and negative tweets. It contains 5000 positive tweets and 5000 negative tweets exactly. The exact match between these classes is not a coincidence. The intention is to have a balanced dataset. That does not reflect the real distributions of positive and negative classes in live Twitter streams. It is just because balanced datasets simplify the design of most computational methods that are required for sentiment analysis. However, it is better to be aware that this balance of classes is artificial. \n",
    "\n",
    "You can download the dataset in your workspace (or in your local computer) by doing:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[nltk_data] Downloading package twitter_samples to\n",
      "[nltk_data]     /Users/johnmoses/nltk_data...\n",
      "[nltk_data]   Package twitter_samples is already up-to-date!\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# downloads sample twitter dataset.\n",
    "nltk.download('twitter_samples')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can load the text fields of the positive and negative tweets by using the module's `strings()` method like this:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "# select the set of positive and negative tweets\n",
    "all_positive_tweets = twitter_samples.strings('positive_tweets.json')\n",
    "all_negative_tweets = twitter_samples.strings('negative_tweets.json')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, we'll print a report with the number of positive and negative tweets. It is also essential to know the data structure of the datasets"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of positive tweets:  5000\n",
      "Number of negative tweets:  5000\n",
      "\n",
      "The type of all_positive_tweets is:  <class 'list'>\n",
      "The type of a tweet entry is:  <class 'str'>\n"
     ]
    }
   ],
   "source": [
    "print('Number of positive tweets: ', len(all_positive_tweets))\n",
    "print('Number of negative tweets: ', len(all_negative_tweets))\n",
    "\n",
    "print('\\nThe type of all_positive_tweets is: ', type(all_positive_tweets))\n",
    "print('The type of a tweet entry is: ', type(all_negative_tweets[0]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can see that the data is stored in a list and as you might expect, individual tweets are stored as strings.\n",
    "\n",
    "You can make a more visually appealing report by using Matplotlib's [pyplot](https://matplotlib.org/tutorials/introductory/pyplot.html) library. Let us see how to create a [pie chart](https://matplotlib.org/3.2.1/gallery/pie_and_polar_charts/pie_features.html#sphx-glr-gallery-pie-and-polar-charts-pie-features-py) to show the same information as above. This simple snippet will serve you in future visualizations of this kind of data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAhIAAAGVCAYAAACrYBhCAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjYuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8o6BhiAAAACXBIWXMAAA9hAAAPYQGoP6dpAABLzElEQVR4nO3dd3zV9aH/8dfZOTnZi5AQCBCmLEFliCiKIu5x1VaqtbW2tcOfdtjb9vZqb3t7tXZYbbXWetVbsbVqHXXUhagoIrL3DisJ2eucnP39/REMxCQI4eR8T3Lez8fjPCJnhLcx5rzzWV+LYRgGIiIiIr1gNTuAiIiI9F8qEiIiItJrKhIiIiLSayoSIiIi0msqEiIiItJrKhIiIiLSayoSIiIi0msqEiIiItJrKhIiIiLSayoSIiIi0msqEiIiItJrKhIiIiLSayoSIiIi0msqEiIiItJrKhIiIiLSayoSIiIi0msqEiIiItJrKhIiIiLSayoSIiIi0msqEiIiItJrKhIiIiLSayoSIiIi0msqEiIiItJrKhIiIiLSayoSIiIi0mt2swOISN+r9wapbQ3Q4g/jDYTxBcO0BiJ4A2FaD/3ZG4gQCEeIRA2iBkQNgy9krmeq732wWMFiAYsN7CngSgOnB5zph/750J9d6e03TwF4cs3+1xaROFCREOnHDMPgQGMb5bU+KhrbqG7xU90S4GBz+8fq5gA1LQGCkWivPv9Vo9bCvr/2LpzNCWmDIL3w0MfBkD4I0gohcwjklrV/tFh69/lFJCGoSIj0A42+IDtrvOyu9bKrppXdte3/XF7nxR/qXUnoc5EgNO1rv/XE7oacEZA7sr1Y5I1q/5hbBqk58csqIr2mIiGSYGpaAmw40MT6Q7cNB5qobPKbHatvhNugemP77dMyimHwFCiacujjyZCWH+eAIvJZVCRETOQLhvm4vIE1+xoTsjT4vS3m/eXNB9pvW18+fN+R5aJ4Ggyd0b42Q0RMoyIhEkdtwQgf76ln2c46PtxVx/oDTYQihtmxelRduRdcZqc4wqfLhdXRPlIx/AwonQ0lM8CZam5GkSRjMQwjcX+KifRzoUiUFbvr+eBQcVi7vzGhi8On3et+lMuMN8yOceysjvaRitLZMHwODJsFNofZqUQGNI1IiMRYky/E21ureWPzQd7dWkNLIGx2pF6zWID+03sgGoJ9H7bf3vsVuDKg7BwYcwGMOhfc2WYnFBlwVCREYmBPnZc3Nh3kjU0HWbmngXC0P737DmCBZtj4XPvNaoehM2H0+TD2gvbdIiJywjS1IdJLe+q8PLf6AC+vq2R7davZcfrE71If5dJoP5raOB75Y+Gky2HS1SoVIidARULkONR7g/xzbQXPrznA6r2NZsfpcwO6SBxpyKkw6Ro46QqdyClynDS1IfIZ/KEIr22s4oU1Fby7rUbTFgPR/hXtt3/9O4w8p32UYuyF4HCbnUwk4alIiPRgc2Uz/7dsD/9cW0FrP14wKcchGobtr7XfnOkw8Uo49SYonGB2MpGEpSIhcoRgOMor6yv5y4d7WLmnwew4YqZgC6x8rP1WMgNO/QqMvxTsTrOTiSQUFQkRYH+Dj0XL9/L0x/uobQ2aHUcSzSdbSl/7IZx8HZzyZcgqMTuVSELQYktJah/uquPP7+1i8ZZqtPShq6RZbHm8LNb2baQzvwWlp5udRsRUGpGQpGMYBm9truaBJTtYlQQ7L6QPGFHY+kr7rWQ6zL6tvVjokuiShFQkJGlEogb/XFvBg0t2svWgiRejkoFl33L46+eg4CSYfStMuBKsNrNTicSNpjZkwPOHIjy9cj9/encn++rbzI7Tr2hqoxeyhsHpt8CUL4Ajxew0In1ORUIGrFAkyt8+2sv9i3dQ3RIwO06/pCJxAtIK4czvw9QbwKbBXxm49N0tA45hGLy4toLfvLGNPXU+s+NIsmqtgpe/C8v+AHN/3D7loTUUMgCpSMiAsmRrNfe8tpWNFc1mRxFpV78Lnr0R3v8dzLsDyuaZnUgkplQkZEBYvbeBu/+1hQ931ZsdRaR7VevgiSuh9AyY91MYMs3sRCIxoSIh/VpVk5+fv7yJl9ZVmh1F5NiUvwd/Prv9AmHz/xsyisxOJHJCVCSkXwpFojyydDf3v7UdbzBidhyR47fxH7D9dZjzfZj5TbA5zE4k0isqEtLvLN1eyx0vbmBnjdfsKCInJtgKb94BaxbBgl/CyLlmJxI5bioS0m9UNLbx85c38cr6KrOjiMRW7Tb4y2XtFwWb/z+QWWx2IpFjpiIhCS8cifLwe7u5f/F2fJrGkIFs0wuw/c328ydm3aITMqVfUJGQhLa1qoXvPb2G9Qe0nVOSRMgLb97ZXiouexAKxpmdSOSorGYHEOlOOBLl94u3c9H976lESHKqWA0PzYF3fwVRjcRJ4tKIhCScrVUtfPfva9igQ6Uk2UWCsPhnsOUljU5IwtKIhCSMcCTK/Yu3c+F976lEiBxJoxOSwDQiIQlhd62Xby1aycZKXd5bpFtHjk5c+QjkjjQ7kQigEQlJAM+t2s8F976jEiFyLCpWw0NnwrqnzU4iAmhEQkzUFozww2fX8PxanQshclyCLfCPr8DuJbDgHnCmmp1IkpiKhJhiS2UzNz2+nH2NQbOjiPRfq5+A/R/DVY9pIaaYRlMbEnePLd3Bxfe/pxIhEgs1W+BPc2Hl42YnkSSlEQmJG18wzLf/8hFvbW8wO4rIwBJug3/eArvfhUvu11SHxJWKhMTFvjovCx9ayt7msNlRRAauDc9A7Vb43F8hq8TsNJIkNLUhfe7tjfuZ/9u3VSJE4qFqPTw8F/YsMzuJJAkVCelTv3tlDTf+ZQ2+sMXsKCLJw1sDj1+sdRMSF5rakD4RCIb4+iPv8PaeAKASIRJ30VD7uomDG9ovTW7Tj3vpGxqRkJg7UNvM/F/+61CJEBFTffQneOJy8NWbnUQGKBUJianVOw5wwb1vU96qby2RhLH7XfjzPGjYY3YSGYD0015i5pXlm/n8Ix/TFNYQqkjCqd8Jj5zXvhhTJIZUJOSEGYbBQy99wC3PbcdvqESIJKzWKnj0QihfanYSGUBUJOSEBIMh/vPx17jrvTrC2MyOIyKfJdAEf7kCNr1gdhIZIFQkpNdaWr1844EX+cvmMIZF30oi/UYkAE/fACseMTuJDAD66S+9UlvfyJfuf5k3q1LAou2dIv2OEYWXvwNv/4/ZSaSfU5GQ47a/8iBf/P2/+LjJY3YUETlR79wFr/3Y7BTSj2llnByXHeV7+cb/vse2YJbZUUQkVpb9HqIRWHCX2UmkH1KRkGO2dtNWbl30EbsjOWZHEZFYW/4gRMNwwT2arpTjoiIhx+TDVeu4/e+r2Eu+2VFEpK+seLj944W/MjeH9CsqEnJUhmHw9gcruOOFDeyzDjI7joj0tRUPg9WuaQ45ZlpsKT0yDIO3li7njufXq0SIJJPlD2oBphwzFQnplmEYvPHeh/z8xbXssxWaHUdE4m3Z77U1VI6JioR0YRgGr73zAfe8sJJy2xCz44iIWd65S4dWyWdSkZBOPhmJuP+fH7HdUWp2HBEx2yvf03HaclQqEtLhkzURD7ywlM2OMkBbwESSnhGFZ2/Shb6kRyoS0uGdD1fy0PNL2OAYQ1TfGiLyiUgA/notVG0wO4kkIL1bCAAfrV7Pn597k3WOsbqKp4h0FWiCJ66Ehj1mJ5EEoyIhbNy2k4efeYWVllEEcJgdR0QSVWsV/OVy8NWbnUQSiIpEktu1Zz9//utzfBQsoc2SYnYcEUl09Tvh6S9CJGx2EkkQKhJJrOJgDX/+2z9Y1pxJky3T7Dgi0l/sfhde+6HZKSRBqEgkqbqGJv7813/wYWWEKkeR2XFEpL/56E+w8jGzU0gCUJFIQq1eH48+9TzLdtZSnjLK7Dgi0l+9/D3Y84HZKcRkKhJJJhQKs+j5V1i6fge70iYS1VkRItJb0RA8dR007jM7iZhIRSKJGIbBS2+9y+IPV1GeeTIBQ9s8ReQE+Wrhb5+HoM/sJGISFYkk8uGqdbz4xhL2p59EY9RldhwRGSiq1sOL3zY7hZhERSJJ7Cjfy19feJVKWwEHollmxxGRgWbDM7DycbNTiAlUJJJAbX0jjz39Ivsaw2y3DjM7jogMVK/+AKo3m51C4kxFYoBr8/v5v2f/yZbd+9mdPoGI/pOLSF8Jt8HTN2i9RJLRu8oAFo1GeeblN/lo9QYa8ibTGNHx1yLSx2q2wKu3m51C4khFYgB7f8Ua3njvQyL5ZewMZpgdR0SSxeq/wLqnzU4hcaIiMUDt2V/B0y+/TtiZxppgodlxRCTZvHQb1O00O4XEgYrEAOT1tbHouVeorm9km2sMIUP/mUUkzoIt8OyNurhXEtA7zABjGAbPv7aYdZu34y+YQG3IaXYkEUlWFavh/XvNTiF9TEVigFm+ej1vvPshnoIhrPNlmR1HRJLdO3drS+gApyIxgOyvPMhT/3wNi83K2sgQXUdDRMwXCcLzN2uKYwBTkRgg/IEATz7/KlXVtTRnjdGUhogkDk1xDGgqEgPEG+9+yOoNm8kdMoLVrdrqKSIJRlMcA5aKxACwo3wvryx+j+ysLD70FWhKQ0QSj6Y4BiwViX7OHwjw9Etv0NTipTp1mKY0RCRxVayGZb83O4XEmIpEP/fGux+ybvM2Bg0ZxprWdLPjiIgc3Tu/hKYDZqeQGFKR6Mc+mdLIzc5ijT+PsA6eEpFEF/LCaz8yO4XEkN55+qkjpzTCGUXs9rvNjiQicmw2PQ+7lpidQmJERaKf+mRKo3ToEJY1Z5odR0Tk+LzyfYiEzE4hMaAi0Q/t2V/Bq28vJSc7i+2hHJrCujy4iPQztdvgwwfMTiExoCLRz0QiEV584x0amppJz8lnTWua2ZFERHrnnV9Cc4XZKeQEqUj0MyvXbWLF2o0MG1LER81ZurKniPRfwVZ4/T/MTiEnSO9C/Uir18eLb76D1WrF58hklxZYikh/t+FZ2L/S7BRyAlQk+pE3l37Ijt37GDZkMCuadQy2iAwQb95hdgI5ASoS/cS+iipef2cZeTlZHAx7qAy6zI4kIhIb5e/BjrfMTiG9pCLRD0SjUV54fQn1Tc0U5OWyokUnWIrIAPPWT8EwzE4hvaAi0Q+s2rCFj9ZsYFjxYHYH3NTpehoiMtBUroWN/zA7hfSCikSCCwZD/GvJUgwMPB4PK7U2QkQGqsU/1yFV/ZCKRIL7aM0GNm/fzbDiwWzxpdIcsZsdSUSkb9TvglWPm51CjpOKRALztfn515L3cTod2J0u1mhthIgMdO/cAyG/2SnkOKhIJLD3P17Dzj37KCkqZJsvFV/UZnYkEZG+1VoFaxaZnUKOg4pEgmpqaeX1d5aRlubBbnewXkdhi0iy+OA+iEbMTiHHSEUiQb374Ur2VVQypLCAXW1uWrQ2QkSSRUM5bNAOjv5CRSIB1dQ18ObSD8nOysRqtbFWoxEikmyW/lbnSvQTKhIJ6N3lKzlYU8/ggjz2Blw06DLhIpJsqjfCttfMTiHHQEUiwdQ3NvHu8pXkZGditVpZq50aIpKslv7G7ARyDFQkEsyylWs5WFPHoLxcKgNOqnWKpYgkq33Lofx9s1PIZ1CRSCAtrV7e/uBjMjPSsdms2qkhIrLs92YnkM+gIpFAPlqzgf1VBxlckEdL2Ma+gK7wKSJJbtu/oHGv2SnkKFQkEkSb389b7y/Hk+rGbrezxZeKgcXsWCIi5jKi8PGjZqeQo1CRSBAr122mfH8FxYMKiBiw1ZdqdiQRkcSw+i8QDpqdQnqgIpEAQqEwb76/HKfdgdPpYHebG7+OwxYRaeetgU0vmJ1CeqAikQA2bNvBzvJ9FA8eBMAmr8fkRCIiCWbFn81OID1QkTCZYRgsW7mOaDSKO8VFXciuLZ8iIp+270Oo2mB2CumGioTJDlRVs3bTVgbl5wKwWaMRIiLd06hEQlKRMNnKdZtoam4hOzODsAE729xmRxIRSUzrn4FQm9kp5FNUJEzka/OzdMUaMjPSsVgs7PGnEDL0n0REpFvBFtj6itkp5FP0rmWitZu2UnGwmsL8PAB2aMuniMjRrfu72QnkU1QkTGIYBktXrMZqs+Jw2PFHrOzXSZYiIke34y3w1pmdQo6gImGSnXv2sWVHOYML8tv/7E/RSZYiIp8lGoKN/zA7hRxBRcIkazdtw9vWRrqnfTpjp6Y1RESOjaY3EoqKhAkCgSDLV68nMz0Ni8VCc9imsyNERI7V/o+gfrfZKeQQFQkTbNlZTmV1Lfm5OQDs0JZPEZHjo1GJhKEiYYI1G7cQiURIcbWPQuxSkRAROT5aJ5EwVCTirKXVy6oNm8nJygSgKWyjMewwOZWISD9TswXqd5mdQlCRiLtN23dRW99IXk4WAHv9KeYGEhHpr7a+anYCQUUi7lau24QFsNvtgIqEiEivqUgkBBWJOKqpa2D91h3kHVpkGYhaqApqt4aISK/sXQZtjWanSHoqEnG0dWc5Tc0t5GRlALBPh1CJiPReNAzb3zA7RdJTkYijDVt3YLVasVrbv+x7NK0hInJidBEv06lIxElLq5eN23Z2jEZEDXRtDRGRE7XjLYiEzE6R1FQk4mR7+V4amprJPrTtsyro1CXDRUROVKCpfa2EmEbvZHGyZcduItEojkO7NSo0GiEiEhu73zU7QVJTkYiDYDDEmo1byUxP67ivUrs1RERio3yp2QmSmopEHOzed4Dq2vqO0yzDUQs1KhIiIrFxYCUEfWanSFoqEnGwbfceAsEQqe72XRpVQSdRbfsUEYmNSBD2LTc7RdJSkehjhmGwZuNW3O7DayI0rSEiEmOa3jCNikQfq2tooqKqmqyM9I77KrXQUkQktsrfMztB0lKR6GN7DlTS3Ool49BCy1DUQm1IV/sUEYmpA6sg6DU7RVJSkehje/ZXEI0a2G02AA5qfYSISOxFQ1onYRIViT5kGAYbt+3sWGQJUKPRCBGRvrF/pdkJkpKKRB+qrW+koqqazIzD50doWkNEpI9UrjE7QVJSkehDew5U0tzS2ukgqlrt2BAR6RsVa8xOkJRUJPrQnv0VGIDt0PqItogVb9RmbigRkYGqeT94a81OkXRUJPqIYRhs2r4L9xHrIzStISLSxypWm50g6ahI9JHmVi9VNbVkpHk67lOREBHpY5reiDsViT5SVV1La6uPNE9qx30qEiIifUwLLuNORaKPVFbXEgyFcDoOlwcttBQR6WMakYg7FYk+cqDqIBaLBYul/fApf9SihZYiIn2teT/46s1OkVRUJPqAYRjsKN9Haqq7476msN3ERCIiSaRup9kJkoqKRB9obvVSXVdPWurh9REqEiIicVK3w+wESUVFog98stAyPU1FQkQk7uq2m50gqahI9IHuFlqqSIiIxIlGJOJKRaIPVNfWd1poCSoSIiJxozUScaUi0QcOVB3E5Ty81dMwoDmiHRsiInFRv6v9B6/EhYpEjEWjUSqqa3CnuDru80ZsRAx9qUVE4iLkg+YDZqdIGnp3i7GmllZavW2drrHRpNEIEZH40jqJuFGRiLH6xiba/P5OIxKtKhIiIvHVtN/sBElDRSLG6uobCQRDndZI+FQkRETiq6XK7ARJQ0Uixuoam7BApx0bbSoSIiLx1XrQ7ARJQ0UixmrqGjqVCABvVF9mEZG4aqk0O0HS0DtcjB2oqibF5ep0n0YkRETirEUjEvGiIhFDhmFQ19CIy9X5cuE+jUiIiMSX1kjEjd7hYsjX5scfCOJ0dD7FUostRUTiTGsk4kZFIoZavL72a2wcsWPDH7ESxXKUV4mISMxFAuCrNztFUlCRiCGvz0cw2PliXW2a1hARMUdrtdkJkoLe5WKopfXQiMQRUxtBQ6MRIiKmCLSYnSApqEjEUKvXhwFYrYe/rCGNSIiImCOoIhEPepeLoRavt8tqiJBGJEREzBH0mp0gKahIxNAnIxJHUpEQETFJoNXsBElBRSKGmlpasNs6b/VUkRARMUlQRSIeEqJIlJeXY7FYWLNmzVGfd9ZZZ3HrrbfGJVNveH1+bJ8uElojISJiDi22jAv7Zz/lsBtuuIHHH3+8/YV2OyUlJVxxxRX89Kc/xePx9DpESUkJlZWV5OXlAbBkyRLmzp1LQ0MDWVlZHc/7xz/+geOIrZWJpi0QwGbtXBwGyohE49JFNL3/1073WT1ZlHzrCaD9VM+m95+kde1rRP2tOAePJufcm3HmDzvq5/VufZ+m954g1FiJI2swWXOuI3X0rI7HWze+TeM7j2OE/KRNOo/suV/ueCzcdJCDT/2EwV+8F6srNYb/tiIyICThGonS0lJuvfXWuP7Sfdy/Lp9//vlUVlaya9cufv7zn/PAAw/wve9974RC2Gw2CgsLsduP3mtycnJIT08/ob+rL7W1dTMiMUCKBIAjbyhDvvmXjlvRl//Q8Vjz8mdpXvE8OfO+TuH1v8Hmyab67z8hGvD1+PkCBzZT+8LdeE6aS9GX7sdz0lxqXribQMVWACK+Jur/dT/Zc79MwdX/ReuGt/DtXNHx+rrXHiD7zBtUIqRfuXOJH8tPmzvdCn91+DdnwzC4c4mfol+34P7vZs56zMvG6shnft5nN4UY/4dWXD9vZvwfWnluc6jT44vWhSj5bQs5dzfz/df9nR4rb4wy+v5WmgOfXuXVz8V4auOGG27AYrFw1113dbr/+eef73Kxxr722GOPdfpF+xMrVqzgq1/9alyzHHeRcLlcFBYWUlJSwrXXXsvChQt5/vnnCQQC3HLLLRQUFJCSksLs2bNZseLwD/2GhgYWLlxIfn4+brebUaNG8eijjwKdpzbKy8uZO3cuANnZ2VgsFm644Qag89TGD3/4Q2bMmNEl36RJk7jjjjs6/vzoo48ybtw4UlJSGDt2LA888EDHY8FgkG9961sMHjyYlJQUSktL+Z//+Z/j/ZJ08PuD2Gydv6SRAVQksNqwpWUfvqVmAu0/+Fo+foHMmdeQOmYWzvxS8i78DtFQAO/md3r8dM0fv0hK6clkzrwaR24JmTOvJmXYZJo/fgGAcGMVFlcqnnFzcA0eTcrQSYRq9wLg3bQEi81O6phZPX5+kUR1Ur6Vyu+mddzW33x4RPeX7wf5zbIgv78ghRU3eShMs3DuX3y0HOVNftm+MNc808Z1kxys/bqH6yY5uPqZNpbvDwNQ64vylX+28atzU3jtCx4eXxvi5W2Hi8bNL7dx1zwXGa4B9PMKINQW80+ZkpLC3XffTUNDQ8w/dyzk5+eTmhrfX65OeALf7XYTCoW4/fbbefbZZ3n88cdZtWoVZWVlzJ8/n/r69iNKf/KTn7Bp0yZeffVVNm/ezIMPPtgxlXGkkpISnn32WQC2bt1KZWUlv/vd77o8b+HChSxfvpydO3d23Ldx40bWr1/PwoULAXj44Yf58Y9/zH//93+zefNmfvGLX/CTn/ykY3rmvvvu48UXX+Tvf/87W7du5YknnqC0tLRXX4dIJEIg1LVIDKR+H26oYP8frmf/H2+k5oW7CTW2XxQn3HSQiLcB9/CTO55rsTtIKZlA4MDmHj9f4MCWTq8BcA+f2vEae04xRihA8OBOIm0tBCu34cwvJdLWQuN7i8g59+t98G8p0vfsVihMs3bc8j3tPzcMw+De5UF+fIaLK8Y5mFBg4/HL3PhCBk+uD/X4+e5dHuTckTZ+eIaLsXntH88ZbuPe5UEAdjUYZLosXDPBwanFNuYOt7GpJgrAk+tDOG0WrhiXuNPGvWZEY/4p582bR2Fh4VF/6fzggw+YM2cObrebkpISbrnlFrzew9MslZWVXHjhhbjdboYPH86TTz5JaWkp9957b8dzfvOb3zBx4kQ8Hg8lJSV84xvfoLW1fYRlyZIlfOlLX6KpqQmLxYLFYuHOO+8E6PR5Pv/5z/O5z32uU7ZQKEReXl7HL/KGYfDLX/6SESNG4Ha7mTx5Ms8888xxfU1OqEh89NFHPPnkk8ydO5cHH3yQe+65hwULFjB+/Hgefvhh3G43jzzyCAB79+7l5JNP5pRTTqG0tJR58+Zx8cUXd/mcNpuNnJwcAAoKCigsLCQzM7PL8yZMmMCkSZN48sknO+5btGgRp556KqNHjwbgZz/7Gb/+9a+54oorGD58OFdccQW33XYbDz30UEemUaNGMXv2bIYNG8bs2bP5/Oc/36uvRTAUIhKJdpnaMAbIiIRr8BhyL/wOBVf/F7nnf5uIt4GqJ75HpK2ZSGt7M7emZnV6jc2T1fFYdyLeBmyebl7jbX+NLSWNvAtvo/al31D1f9/BM+Fs3COm0fD2I6RPu4hw00EqHr2Fike+gXfL0pj++4r0pe31UYp+3cLw37XwuWd87Gpof8Pb3WhQ1Wpw3sjD07wuu4UzS+18sL/n6Y1l+yKcN6Lz1PD8kXY+2Nf+mlE5Vnwhg9WVEerbDFYciDBpkI36NoP/fNvP7xek9MG/ZQIwYv+rnM1m4xe/+AX3338/+/fv7/L4+vXrmT9/PldccQXr1q3jqaeeYunSpXzrW9/qeM71119PRUUFS5Ys4dlnn+VPf/oT1dWdj/O2Wq3cd999bNiwgccff5zFixdz++23AzBr1izuvfdeMjIyqKyspLKystslBgsXLuTFF1/sKCAAr732Gl6vlyuvvBKA//iP/+DRRx/lwQcfZOPGjdx222184Qtf4J13eh5N/rTjWmwJ8NJLL5GWlkY4HCYUCnHppZfy7W9/m2eeeYbTTz+943kOh4PTTjuNzZvbf7u8+eabufLKK1m1ahXnnXcel112GbNmndiw9MKFC/nf//1ffvKTn2AYBn/96187pj5qamrYt28fN954IzfddFPHa8LhcEcxueGGGzj33HMZM2YM559/PhdddBHnnXder7IEQ2EikQgOx3F/SfsF98hTDv8hH1xFYznwp6/gXf8WzqKx7fd/eo7QMLre10Xnxw3D6HRf6uhZnRZf+veuI1Szh5xzv07Fn75K3sXfx+bJpvL/vkNKyYQuxUQk0UwvtvF/l7kZnWvloNfg5+8GmPWIl43f8FDV2l4oBqV1/v9ikMfCnqaef7uuajUYlNb598JBaVaqWtvfSLPdFh6/zM31z7fRFjK4frKD+WV2vvxCG98+zcnuxiiX/M1HKAJ3nuXi38YPkNGJPhiRALj88suZMmUKd9xxR8cvy5+45557uPbaazvei0aNGsV9993HmWeeyYMPPkh5eTlvvvkmK1as4JRT2n+u/vnPf2bUqFGdPs+RiyWHDx/Oz372M26++WYeeOABnE4nmZmZWCwWCgsLe8w5f/58PB4Pzz33HNdddx0ATz75JBdffDEZGRl4vV5+85vfsHjxYmbOnAnAiBEjWLp0KQ899BBnnnnmMX09jvtd75PRB4fDQVFREQ6Hg7Vr1wJ0WWxiGEbHfQsWLGDPnj28/PLLvPnmm5xzzjl885vf5Fe/+tXxRuhw7bXX8u///u+sWrWKtrY29u3b1zGME422fwM9/PDDTJ8+vdPrPhk1mDp1Krt37+bVV1/lzTff5Oqrr2bevHnHPawDEAyGiES7GZHozb9YP2B1puDMKyXUUIF7dPs3YNTbAGk5Hc+J+JqO+sZu82R3jD58InqU1xjhEPWvP0juRd8l3FCJEY2QMnQiAI6cYgKVW0ktm97ta0USxYJRh9+kJwIzh9gYeV8rj68NMWNI+8+PT9dvw+h636d91msuH+fg8iOmL5aUh1lfHeH3F6RQdl8rf73STWGahdP+7GXOMBsFngGwdb2PigTA3Xffzdlnn813v/vdTvevXLmSHTt2sGjRosMxDINoNMru3bvZtm0bdrudqVOndjxeVlZGdnZ2p8/z9ttv84tf/IJNmzbR3NxMOBzG7/fj9XqPeZekw+HgqquuYtGiRVx33XV4vV5eeOGFjpH8TZs24ff7Offcczu9LhgMcvLJJ3f3Kbt13N8pHo+HsrIyhg0b1rEVs6ysDKfTydKlh4eXQ6EQH3/8MePGjeu4Lz8/nxtuuIEnnniCe++9lz/96U/d/h2fXIY7Ejn6SuUhQ4YwZ84cFi1axKJFi5g3bx6DBg0CYNCgQRQXF7Nr1y7Kyso63YYPH97xOTIyMrjmmmt4+OGHeeqpp3j22Wc71nUcj3AkQjQaxfqpMjUwJja6MsIhQnX7sKXlYM8chM2TTVv56sOPR0L4923AVTyux8/hKh7b6TUAbbtX9/iaxg/+RsqIabgKy9p/QEQPf38Y0TBE++6Hhkhf8TgtTBxkZXtdlMJDowqfjCR8otrXdcThSIVplo7RjI7XeKNdRjY+EQgbfONlPw9d5GZHfZRwFM4stTMmz8boXCvLjzKN0q9Y+q4MzZkzh/nz5/OjH/2o0/3RaJSvfe1rrFmzpuO2du1atm/fzsiRIw+NunZ15P179uzhggsuYMKECTz77LOsXLmSP/yhfZdcKNTzWpnuLFy4kDfffJPq6mqef/55UlJSWLBgQUdWgJdffrlT3k2bNh3XL9QxGYf3eDzcfPPNfP/73ycnJ4ehQ4fyy1/+Ep/Px4033gjAf/7nfzJt2jROOukkAoEAL730UqeScaRhw4ZhsVh46aWXuOCCC3C73aSlpXX73IULF3LnnXcSDAb57W9/2+mxO++8k1tuuYWMjAwWLFhAIBDg448/pqGhge985zv89re/ZfDgwUyZMgWr1crTTz9NYWFht1tqPssn3wTx3gIULw2LH8Fddhq2jHyiviaaPvgb0aCPtAnnYLFYSD/lUpqWPY0juwh7dhFNy57G6nDhGXd4aKz2pV9jS88l+8wbAEifdgkHn/wBTR8+Q+qo6fi2L8e/Zw2FC3/Z5e8P1uzBt+VdBt9wPwD2nCFgsdKy9nVsadmE6vbjHDyqy+tEEl0gbLC5JsoZQ+0Mz7JQmGbhjV1hTh7cPjoRjBi8Ux7m7nk9r2OYWWLjjV0Rbpt5+L7Xd4WZVWLr9vk/ezfAgjI7UwfbWF0ZIRw9/CYWikBkoAyl9mGRALjrrruYMmVKx7o8aB/p3rhxI2VlZd2+ZuzYsYTDYVavXs20adMA2LFjB42NjR3P+fjjjwmHw/z617/uuAjk3//+906fx+l0fuYv29C+nqKkpISnnnqKV199lauuuqrjl/Xx48fjcrnYu3fvMU9jdCdmE/p33XUX0WiU6667jpaWFk455RRee+21juEap9PJD3/4Q8rLy3G73Zxxxhn87W9/6/ZzFRcX89Of/pR///d/50tf+hLXX389jz32WLfPveqqq/j2t7+NzWbjsssu6/TYV77yFVJTU7nnnnu4/fbb8Xg8TJw4sWPuKS0tjbvvvpvt27djs9k49dRTeeWVVzpdvfNYGYbBp6b3AbBYBsb/keGWWmr/eQ8RXzO21AxcRWMpvO7X2DMLAMiYfiVGOED96w8S8bfiKhpDwdX/1emMh3BzTaf/sVOGjCPvkttpfO8JGt97AntWIfmX/ABX0ZhOf7dhGNS/9nuyz74Jq7P9h6nV4SL3glupf+NBjEiInHO/jj296y4gkUTzvdf9XDzaztBMK9XeKD9/L0hzwOCLkx1YLBZune7kF+8FGJVjZVSulV+8FyDVYeHaiYenJa5/ro3idAv/c6hc/L/pTuY86uPupQEuHWvnhS1h3twVYemXum4D3Fgd4amNYdZ8rX14fGyeFavFwiOrghSmWdhSG+XUou4LSL/Tx7/YTZw4kYULF3L//fd33PeDH/yAGTNm8M1vfpObbroJj8fD5s2beeONN7j//vsZO3Ys8+bN46tf/WrHMoHvfve7uN3ujl9ER44cSTgc5v777+fiiy/m/fff549//GOnv7u0tJTW1lbeeustJk+eTGpqarfbPi0WC9deey1//OMf2bZtG2+//XbHY+np6Xzve9/jtttuIxqNMnv2bJqbm/nggw9IS0vji1/84jF9HSxGT+Msclz2Vx7kzt88SH5uDp5Ud8f97zVmstXX+1M/Rcz0u9RHuTT6htkxBpTPPePj3T0Ran0G+R4LM4bY+NlcF+Pz29+8DcPgp+8EeGhliIY2g+lDbPzhghQmFBx+cz/rMS+lWVYeu+zwz5pnNoX4j8UBdjVEGZlj5b/PdnXZ0mkYBrMf9fHD2U4uGn34sZe2hfjmK34CYfj52S6+MtXZx1+FOJn6Rbjkvph9uhtuuIHGxkaef/75jvv27NnDmDFjCAQCHSPTK1as4Mc//jHLli3DMAxGjhzJNddc0zENUllZyY033sjixYs7tpLeeuut/Nd//Rdf+9rXAPjtb3/LPffcQ2NjI3PmzGHhwoVcf/31nU58vvnmm3n66aepq6vjjjvu4M477+z2ZMtNmzZx0kknMWzYMHbv3t1p5NwwDO6//34eeOABdu3aRVZWFlOnTuVHP/oRc+bMOaavi4pEjPRUJD5oymCTt/tpGZFEpyIh/dr0r8OCu81O8Zn2799PSUlJx0aE/mZg7lVMII4BMrUhItLvOBPzl7jFixfT2trKxIkTqays5Pbbb6e0tPSYRwASjYpEjLQPFVkwPrXhU0VCRMQkzsScVg6FQvzoRz9i165dpKenM2vWLBYtWpTQF6U8GhWJGGk/ppQuB0c4rNqSKCJiCldiXuRx/vz5zJ8/3+wYMTMAThxJDBaLBQuWLnuENSIhImKSBJ3aGGhUJGLEYbdjtVo6DvjouF9FQkTEHC4ViXhQkYgRp8OOzWYjoiIhIpIYEnSNxECjIhEjLqcTm81K+FMnjWmNhIiISZyJuUZioFGRiBGHw47NaiMa6VwcnBqREBExR4IuthxoVCRixGq14nI5u0xtuDUiISJijrQCsxMkBRWJGHKnuLpMbaTYolgH7MXERUQSlM0FqTlmp0gKKhIx5Ha5iHZzNbZU2wC5JK+ISH+RNsjsBElDRSKG3O4UIpGuUxmpmt4QEYmv9EKzEyQNFYkYSkt1d5naAHBrREJEJL7SNSIRLyoSMZSZkd7tiIRHIxIiIvGVPtjsBElDRSKG0jypWLq5XyMSIiJxpjUScaMiEUPpnlS6axJabCkiEmdaIxE3KhIxlOZJxWKxdJneSFOREBGJr8whZidIGioSMZSWmorT6SAYCna6P1NFQkQkvnLLzE6QNFQkYijNk4rT4SAYDHW632OLYLNowaWISFw4UiGj2OwUSUNFIobS0w4ViVC40/0WC2RoVEJEJD5yRrT/4JW4UJGIIZfTiSfVTTAU6vJYpj3czStERCTmckeanSCpqEjEkMViIS87C38g0OUxFQkRkTjR+oi4UpGIsSGDCwgEgl3uV5EQEYmT3FFmJ0gqKhIxlpuT3e39KhIiInGiEYm4UpGIsdzsTCxANNp5l4aKhIhInGiNRFypSMRYblYmLpcL/6emN1KsBh6rdm6IiPSpjCGQmmN2iqSiIhFjOVmZuN0u2vz+Lo/lObuunRARkRgqmmJ2gqSjIhFjaZ5UMtPSaPN33bmR5+i6LVRERGJo8BSzEyQdFYkYs1gsFBcWdD8ioSIhItK3NCIRdyoSfaBoUD6hUNfFlSoSIiJ9rOhksxMkHRWJPjAoPxcAwzA63e+2RbXgUkSkr2QMAU+e2SmSjopEHyjMzyPF5ep+nYQWXIqI9A1Na5hCRaIPFBbkkeZx0+r1dXlM0xsiIn1ECy1NoSLRB1LdKRQNKui2SOSrSIiI9I0h08xOkJRUJPpIWWkJbd1cvGuQM4gVo5tXiIhIr1kdUDLd7BRJSUWijxQNysdC1wWXDquh6Q0RkVgrngpOj9kpkpKKRB8pzM/D5XJ2u+BysKvrfSIicgJKzzA7QdJSkegjhQV5pHtSu10nMVg7N0REYqt0ttkJkpaKRB9JdadQUlRIc6u3y2OFWichIhI7NqfWR5hIRaIPjRs1gkA3Cy7tVoN8jUqIiMRG8TRwppqdImmpSPShYcWDsdlsBENdF1dqekNEJEY0rWEqFYk+NKx4MFkZ6TS1tHZ5rEgLLkVEYmP4HLMTJDUViT6UkZ7G8JJimppaujxW6AzisERNSCUiMoC4MmHoTLNTJDUViT42tmw4gWDXqQ2rBYZoVEJE5MSUnQM2h9kpkpqKRB8bNmQwdruNYDdlYliK34REIiIDyJgLzE6Q9FQk+tiw4sFkpqd1u06iJMWPRdtARUR6x2qHUeeanSLpqUj0sfQ0DyOGDaGxubnLYy6rQaF2b4iI9M7QmeDOMjtF0lORiIOJY0cRCIa6XHcDYKimN0REemfMArMTCCoScTF6+FDSUt3dHpetIiEi0ktaH5EQVCTioKSokKJBBdQ1NHV5LNMeIcuuq4GKiByX/LGQM9zsFIKKRFxYrVamThiL19d1RAJghLstzolERPq5k64wO4EcoiIRJ6NHluJwOPB3c+2NMhUJEZHjM+lqsxPIISoScTJy6BAKcrOp72Z6I8MeocCh3RsiIsdkyGma1kggKhJx4nI5mTx+DI3NXc+TABiZ2v20h4iIfIpGIxKKikQcjS0bjsUC4XC4y2MjdTiViMhnszq0PiLBqEjE0diRpeTlZFFb39jlsRRbVNfeEBH5LGXngCfX7BRyBBWJOEpP83DKpJOoa+y6TgKgTNMbIiJHp2mNhKMiEWeTxo3GbrPhD3RdXDksxa9Li4uI9MSZrkOoEpCKRJyNHVlK0aB8aurquzxmt8BIbQUVEenexH8Dh9vsFPIpKhJx5nI5OW3KhG6vBgowzuONcyIRkX7i1K+YnUC6oSJhgknjRuNOcdHazUmXuY6wzpQQEfm0khlQOMHsFNINFQkTjBhazLDiwdTUdp3eABivUQkRkc40GpGwVCRMYLPZmDF1El6fv9tLiw93t5FijZiQTEQkAXnyYfylZqeQHqhImGTy+DFkZaZT381WUJsFxmgrqIhIu5OvA7vT7BTSAxUJkwwuyGPqhLFU9zC9MTbVp5MuRUQsVjjly2ankKNQkTDRjKmTsNmstPn9XR5Lt0co0UmXIpLsRp8PWSVmp5CjUJEw0fhRIxheMoTK6tpuH5+Y1v0WURGRpDHzW2YnkM+gImEiu93OnOlTaWvzE4l0PdFysCuoraAikrxKpkPp6WankM+gImGykyeMJT83m5r67tdKTE5viXMiEZEEMfs7ZieQY6AiYbKcrEymnzyR2rrGbh8f6gqQbQ/FN5SIiNkKToLR881OIcdARSIBnDZlIqnuFJpbux5EZbHAZK2VEJFkM/u29h+AkvBUJBLAyGFDmDC2jAOVB7t9fIS7jXRbOM6pRERMkl0KE64wO4UcIxWJBGC1Wjl71mnYbFa8bV2v/mm1aAeHiCSRWbeA1WZ2CjlGKhIJYuLYMsaVjeBAZXW3j49O9ZGqY7NFZKBLK4STv2B2CjkOKhIJwmazcc7ppxGNRvEHum75tFtginZwiMhAd+b3we4yO4UcBxWJBDLlpLGUlZb0uFZibKqPDK2VEJGBKmcETL3B7BRynFQkEojT6eDs008jEAoSDHXd8mm1wLSMZhOSiYjEwdn/ATa72SnkOKlIJJhTJ59EaXERFVU13T4+IsVPrk67FJGBZvBkOEk7NfojFYkE405J4ezTT8Pb1kYo3HUaw2KBU7VWQkQGmnPu0LkR/ZSKRAKaOW0yw0uK2F9R1e3jQ1ICDHbqyqAiMkAMnwNl55idQnpJRSIBeVLdnH/W6fgDQQLB7qcxTtVaCREZKM650+wEcgJUJBLU9CkTGT1iGHsPdD8qUeAMMSKl6+FVIiL9yoQrYcg0s1PICVCRSFAul5MFZ51ONBLB1+bv9jnTM5twWLpeflxEpF9wpsF5Pzc7hZwgFYkENnXiOCaOG83eA5XdPu6xRZmio7NFpL8683bIKDI7hZwgFYkEZrfbOf+sWdjtNlq6uTIotF+DI1OXGReR/iZvDMz4htkpJAZUJBLchDFlTJ0wjn0VVRiG0eVxqwVmZTaZkExE5ARc8EuwOcxOITGgIpHgrFYrF5w9m7TUVOoaGrt9TrEryHAtvBSR/mL8ZTDiLLNTSIyoSPQDZaVDOWvWKVRV1xKJdH8F0OmZTdi18FJEEpzh8MD8X5gdQ2JIRaKfWDB3NkOLB7O3h0Oq0mxRTtbCSxFJcJYzb4fMYrNjSAypSPQT2ZkZXHjOGfj9gR63g05MayVP1+EQkQRlFJ0Ms75tdgyJMRWJfmTWtClMHj+G8n0Helx4OSerEStdHxMRMZNhc2K57EGw2syOIjGmItGPOBx2Lj3vLDypbuoaut+pkeMIM1UX9RKRBGM58wdQMM7sGNIHVCT6mdEjhnHWzFOoqq4hEul+ceUkTXGISAKJDp4Cs28zO4b0ERWJfsZisbBg7mxKigrZe6Ci2+doikNEEoVhdWK9/I+a0hjAVCT6oZysTC6bP5dgKExzDydeaopDRBLCWZrSGOhUJPqpmdMmM/vUKezZX0E0qikOEUk80cLJWDSlMeCpSPRTVquVyxecQ8ngQvbs73mKY252g64QKiJxF7WnYr3qUU1pJAEViX6sIDeHy88/+hRHpj3C6boWh4jEmeWS+yB3pNkxJA5UJPq5Y5niKEttY3Rq90VDRCTWwhM/j2XSVWbHkDhRkejnjmWKA2BWRjNZuty4iPSxYOZw7Bf/xuwYEkcqEgPAkVMcTS3dX2/DbjU4J7sBm9ZLiEgfiVgcOK59EpypZkeROFKRGCBmTpvM3JmnsHd/JaFQuNvnZDvCzMxojnMyEUkW0fN+gWXQeLNjSJypSAwQVquVf7vwXMaNHsGO8r3dXosDYKzHx0i3L87pRGSgaxu5AMfMr5odQ0ygIjGAZKSnsfCyC8hIT6PiYE2Pzzsjs4lcrZcQkRhpSx+O+5r/NTuGmERFYoAZNXwoVyw4h5bWVlp62BJqtxqcm1NPijUS53QiMtAEbGm4vvyC1kUkMRWJAejsWadyxmlTKd9XQTjc/XqJNHuEedkNuh6HiPRaBCvGVY9jzR5mdhQxkYrEAGSz2bj64vmMGjGUHeX7elwvUegKMkuHVYlIL7XM+hEpY+eZHUNMpiIxQGVnZnDtZReQluo+6nqJsR4f43VYlYgcp5ohC8g67/tmx5AEoCIxgI0fNYIrLphHa6uXxqaerwQ6I7OJImcgjslEpD+rSx1FzhcfNzuGJAgViQFu3uzpnHfmLPZVVtHm93f7HKsFzs6pJ8PW/XoKEZFPtNiy8dz4PDaHy+wokiBUJAa49vMl5jF9ygR2lu8jHOl+p0aK1WBBbh2p2skhIj3wkUp04bOk5A4xO4okEBWJJOBOSeG6Ky+mbPhQtu/a0+Piy3R7hPm5dbrsuIh0ETAcNF/0EJkjppkdRRKMikSSyM/N5oarLiEnK5M9+yt7fF6uI8y5OfXYtC1URA4JG1YOzPgphadcYnYUSUAqEkmkrHQo1162gEg0QnVdfY/PK3IFOSu7AYvKhEjSixqwteyrDD//G2ZHkQSlIpFkZkydxCXnnkVdfSNNzT3v5Bju9uuMCRFhQ/4ljL7mZ1gsFrOjSIKymx1A4stisXDxvDk0t7TyryXvY7PbSEvt/mjbcR4fvoiN1a3pcU4pIolgo2c2ZTc8gMPpNDuKJDCNSCQhu93O5y45nzOmT2P33gP4Az2fITEto4UJntY4phORRLDJNZWSr/wfqWn6RUKOTkUiSblcTq6/8iJOnXwS23fvJRTq+QyJGZnNnKQyIZI0NtgmMOjLT5CRnWt2FOkHVCSSWJonlS9fcxkTxpSxdVc5kR7OmACYmdnMOB2lLTLgbWAMOdc/Tu6gYrOjSD+hIpHkcrIy+crnr2DE0GK27iwnGu35DIlZmU26LofIALYuOgrPNQ9TNKzM7CjSj6hICIML8rjp81cyuCCf7bv39nhglcUCs7KaNM0hMgCtCpfhvPx+ho+bbHYU6WdUJASA4UOL+crnLyc3O5Ntu3s+/RLapzm0AFNk4Pg4WIbtwnsYe/JMs6NIP6QiIR3GjRrBTddeSXZG+lFHJqB9AebJaT2fQyEi/cMHgTHYLriLSTPmmh1F+ikVCenkpNEjuenaK8lMT2NH+b6jlolpGS2cntmoEzBF+qGoYeF1/yRcC37GlNPn6cAp6TUVCeli4thR3HTtFWSkeT5zZGKcx8fZ2Q26NodIPxI2rPzTP42cBT9i6hnnqUTICVGRkG5NGjeary68kqyMtM8sE8Pdfubn1uHUVUNFEl7AcPBs20xKLvoB0+acrxIhJ0xFQno0cewovvaFq8jJymDbrj1H3Rpa5ApyYV4tqdaez6IQEXO1Gik81Tab0Zd+h5Nnn6sSITGhIiFHddLokXxt4b9RkJvDlh27j3poVa4jzMV5tWTYej4lU0TMUR/18PfgXCZf/v+YMusclQiJGRUJ+UzjRo3gWzd8jtKSIjbv2H3U47TT7REuya+hyNnz9TtEJL72hHP5B+dz+uduY/LMuSoRElMqEnJMhg8t5ttfupaTRo9k687yo17oK8VqcH5uHeN11oSI6T72D+N150Wc+4XvMGbydLPjyACkIiHHbHBBHt/84jWcMnk823fvpdXn6/G5VgvMymxmdmYjVu3oEIm7CFZebhnP+pwFXPylWxk2eoLZkWSAspsdQPqXnKxMvv6Fq0h1u3lv+UqGDB5EZkbPlxke6/GRZQ/zZkM2/qgtjklFklebxc3TDSdhGzGHSz/3VXIKBpsdSQYwjUjIcUvzpHLjNZdx/lmnU3Gwhuq6+qM+v9AV5LK8WnLsoTglFEledZZcHqmdRur4+Vxy/bdVIqTPqUhIr7hcThZefgH/duG5tLR6Kd9XcdSzJtLsES7Jq2Wku+fpEBE5MduMUh6tm8rwGRdx4cJvkJ6VY3YkSQKa2pBes9vtXDZ/LoX5uTz5/Kts2bGbUSOGYbd1P4VhtxrMzW5ksDPIsuYMIoZ6rEgshLGzODCR9aFhzLrwMqafczE2u368S3zoO01OiMViYea0yeTnZvPY0y+yZfsuRpaW4E5J6fE1Yz0+BjmDvNWQTWPYEce0IgNPozWHZxrGE8wo5YKrFjL25Jna3ilxpV8JJSbKSofy/768kNOmTGBn+X4ampqP+vxsR5jL8moZneqNU0KRgWebfRx/OjgFx5ApXPalWxk3dZZKhMSdRiQkZvJzs7n5+qvJz32T19/5AF+bn6JB+T3+YLNbDeZkNVHkDPJ+UyYhTXWIHJOQxck7lpl8VOVi7MnTmXfFDWTk5JkdS5KUioTElDslhYWXX0BhQR7PvvwGW3eWU1Zagv0o87VlqW3kO4O83ZBNbcgZx7Qi/U+dbRAveKdQF3Aw/ez5nHHh1ThdPU8livQ1FQmJOavVyrlnzKC4sIAnn3+FLdt3M6ykiPQ0T4+vyTy0q2NdaxqrWtKJouFZkSNFsLHGeSpvVGaQkVvAgiuvYfy007FaNZIn5rIYR9uzJ3KCGpqa+fs/X+O9j1aR5vFQXFjwmXO4DSE77zRmaXQiAfwu9VEujb5hdoykV2cbxFvGDHYc9DFi/MmcfdkXKCgeZnYsEUBFQuIgEomw+IMV/OPVt2hp8TKytASH4+iDYVEDjU4kABUJc0WwsS5lBm/X5RGNGkw7Yz6z5l+By51qdjSRDprakD5ns9k494wZlA4pYtFzL7Nlx25KigqPerS21QJT0lsZluLX6IQkpTrbIN5zzmXTvnpyB2Vz5kWfY8yUGdqVIQlHIxISV80trTzz8hss+XAlDruNYUOKPnOON2rA+tY0VremEdbOjrjSiET8hXCwwT2DD1qLaW5soGzCNM6+7AvkDio2O5pIt1QkJO6i0SgfrlrHP15dzL7KKoYNKSLjKAsxP9EasbK8KZPdfnccUgqoSMTbHudolttnsmv/QVLTMznt7IuYOvs87cqQhKYiIaaprqvnuVffYumKNTgd9mManQA4EHDyQVMmTToVs8+pSMRHky2HFalns6XRSUtjHSPGTeGMC6+maFiZ2dFEPpOKhJiqt6MTUQM2eD2sbknXQVZ9SEWib4VwsCF1Juss46ncV65RCOmXtNhSTGW1Wpl1yhTKhg/tGJ2oq29gaHERNlvPBcFqgUlpXka621jelMkuTXdIP1PuHMOq1DlU1HlpadytUQjptzQiIQnjk9GJ5/61mH0VVRQW5JGbnXVMq9Srgw5WNGdQGXTFIWny0IhE7FXZh7LGcwYH/B6qD5STlpXDqXMv1CiE9FsqEpJw6hubePXtpbyzbCXetjaGDSki1X1sP2D3+12saEmnTttFY0JFInbqbQWsST2D/dZiqveXY0QNxkw+jRnnXqrDpaRfU5GQhGQYBtt37+WF199mzcatpKS4GFpUiM1mO4bXwi5/CiubM2iOaPbuRKhInLgWaxZrU09nj2M0DbUHaWmoZXBpGTPmXcroSafpiGvp91QkJKGFQmE+WLmGl996j70HKhlUkEfeMU53RA3Y4ktlTUs6vuhnFxDpSkWi93wWDxtSZ7LTNRGvt5WaA3tJz8pm6pzzOfn0c3F70syOKBITKhLSL9Q3NvGvJe+zZNnHtHh9lBQVHtPuDoCwAdt9qaxrTaNFIxTHRUXi+LVYM9nsPpVdrpPwB8NUH9iDxWJhzOTpzDz3UvKLhpodUSSmVCSkX9lRvpdXFi9l1YbNhMMRhhYPPub1E1EDdrW5WduaRoPOoDgmKhLHrsGWxyb3aex1jiEUDlNTsZdwKEhJ2XhOPXMBIydM0zSGDEgqEtLvRKNRNmzdwb+WfMD6zduw2mwMLS7E5Ty2BZaGAfsCLta0pFOtRZlHpSLx2artRWxyn0aFYwSRaITayv34fS0MHlbGqWddyOhJp+E4xu9Nkf5I47zS71itViaNG834USNYtX4zry55n2279pDicjKkqBCH/ejf1hYLDE0JMDQlQGXAyXqvh33+FAxdZVSOURQLFY4RbHafQo1jCNFolIaDFbQ01VNQNJSzLv4846edrqt0SlLQiIT0e4FAkOVr1vOvJe+za+8BPKluigsLcDqOffqiJWxjiy+Vrb5U/FqY2UEjEp21WVLZmTKRHa5J+GwZRCMRGmqraGmsIyuvkKmzz2Xi9LPwpGeaHVUkblQkZMDw+tpYtnIti9//iD0HKnE4HAwZXECK69gPqYoYsLvNzWZfKgd1uJWKxCHV9mK2p0xmn3M0UYuNSDhMXXUFvuZGsvMLmThjLhNPm0NmTr7ZUUXiTkVCBpw2v5+P127irQ8+YufuvWC1MGTwIDzu4ztGuy5kZ7PXw842d9JezyOZi0TI4qTcOZbtKVNotLcXhFAwSN3B/QTafOQVDmHKrHMYN3UWaZnZJqcVMY+KhAxYwWCINRu38Nb7H7F5x24ikQjFgwcd87bRT4QN2ONPYYcvlQMBF9EkWkuRbEUigpUqRym7XeM44BxJxNI+PRb0t1FbtZ9wKEhhyXAmz5rHmMnTSU1LNzmxiPlUJGTAi0QirN+yg8UffMTGrTvwtfnJzckiPyfnqBcG644/YmXXoVKRDDs+kqVI1NgHU+4az17naALW9gWShmHQ0lhPY+1BrDYbRcPKmHL6PEZNPAVXii4SJ/IJFQlJGtFolJ179vPRmvUsX72emroG3CkpFBbkHfNZFEdqDtvY0eZmV5ubxgF6LsVALhKNtlz2OsdQ7hpHqy2r4/5wKEhDTRXe5kY8GVmMHH8y46bNYtioCdiPYwGvSLJQkZCk1NDUzKr1m3nvo1Xs2rufcDhKQV4OudmZx3T89qc1h23s8aew159CVdA5YLaSDqQiEcVKtb2YA86RHHCO7FQeDMOgzdtCfXUl0UiYvMIhnHTqGYyedCq5g4rNCy3SD6hISFILh8Ns2r6LD1etY/WGLTQ0teBJTSE/L4e01N6dARCIWth3qFTsD7gI9uOFmv29SAQtLiodpex3jqTCMZyQtfPIUzDgp6muGm9LEympHkpGjmPCqWcwYtwUnQEhcoxUJEQOqayuZe2mrXy4ah17DlTS1uYnMyOd/NwcUly9Ww8RNaAq6KQi4KIy6KQm6OxXizX7W5GIYKPOXki1o4QqRwk19mIMS+dzQSLhME31NbQ01mGzO8gtKGLMlOmMGD+FwUNH9mpESiSZqUiIfEokEmHX3gOs27yNj9ZsoOJgDeFIhNysTPJysrB/xsmZRxOOWjgYcrQXi4CL2pAjoYtFoheJCFbq7YUcdJRw0F5CraOoY6fFkaLRKC2NdTTX12JEo2Tm5jNq4qmMPGkKQ0aMxeHUmSEivaUiIXIUgUCQrbvKWbtpGx+v20htfSOGYZCZkU5uduZxHXbVnVDUwsGgk5qQg9qQg9qgE28CnayZaEXCa02n3jaIevsg6uyF1DiKuy0OAOFQiNamelob64lEI6RlZlM6eiKjJk5j6KgJ2ropEiMqEiLHqKXVy5ad5WzZsZu1m7ZSXddAMBQiLTWV3OxM0jypMRkWb4tY20vFETevSZc/N7NIHFkaPrl9sjWzJ4E2Hy2NdXhbmrBarKRn5zBs1ASGjZlIycixOnlSpA+oSIj0QiAQZOfe/WzbWc6qjVuoqKqh1evD6XSQlZlOVno6Tmfstgr6oxaaw3aawnYaw/aOf26K2Ij04WLOvi4SYey02LJpsWXT/MlHa/vHoPWzz2qIRML4WppobW4k2ObD4UohK7eAkSdNpWTkWIpKR2vkQaSPqUiInKBoNMq+iiq2797L+i072LV3P43NLYTDEVxOB5kZ6WRmpB3XRcSOlWGAN2KjKWLDG7Hhi9jwRa3tH4/4596uwziRIhHBRpvVc+iW1v7Rkobf6sFrS6fFmo3Pmt5+OdZj/ZyRCL6WJrwtjQTafFitVlLTMsgtHMKIcVMYMnw0g0pG6LLdInGkIiESYy2tXvYcqGTP/kq27NjN7n0HaGppJRKJ4HQ6yUjzkOZJxZ3iitsOAX/UQlvERsiwEIxaCRsWQp/colZ8wRA1e7eTm1+AJy0DAzCAK+3LmGlZd+jPFsBCxGInZHEStjgOfXQS4vCfQxYXfmvqMY0oHI1hGIRDQdq8Lfhamwn4fFisVlLTM8gtKGLY6AkUlgynoLiU9Kwc7bYQMYmKhEgfa25pZW9FFeX7K9i8fRf7Kqpo9bbhDwQASHG5SPO4SfN44loujtTma2XLG4s49eSJDB02PO5/v2EYhIIB/L5W2rwt+H0+wMDucOL2pJGdV8iw0RMYNKSUQcWlpGfnqjiIJAgVCZE4a/P7qaquo7K6hoqDNezcs58DVdW0en34AwEMw8DhcOBOceFOScHtduF2ubBa+24tRLyKhGEYhAJ+Av42gv42An4foWAQMHA4nKR40sjKHURR6ShyBxWTWzCYnEFFuD3pKg4iCcqcpeAiScydksLwocUMH3r46OUjy8XBmjoqqms4UFlNU2srzdWth0YvLICBy+UkxeXC4bDjcjhwOh04HQ5sNvO3jUajUcKhIOFQkFAw0PEx6PdjGO0rNexOF64UNympaRQPH01uYTFZuYNUGkT6KRUJkQTQXbkwDINWr4/6xibqGpuob2iitqGRA1XV1NQ14Gtro6XVSyAUIhgM8cngomGAzWbFZrMd/mi1YrfZsNqs2Kw27DYrVqu14w076G+fZgkHA7R5WwEDDIgaUaKRCNFImGg0QjQSJdLxz+23SDiMpWOwxILd4cTucOJwOklxe8gvGkp+YQmZufmkZ+WQnpVLRlYuqemZfTrKIiLxoakNkX7IMAwCwSCt3jZavT5avF5avT5afW20tnppbG7B62vD5w/Q5vfjDwTxBwJEIhEikSiRaJRoJIpB+//+0XCY0O5ljBmSR6rbDRYLlkM3m82G1WbDarXhcLpwprhxuty43G5cKR48GZmkpmXg9qSTkurB7UnH7Ukn1ZOOM8Wt0QWRAU5FQiRJGIZBMBgiGAoRCIYIhcMd9xuGQTjQhsdlO/TGf6hE2O2HRhdc2B1OjSCISBcqEiIiItJr+vVCREREek1FQkRERHpNRUJERER6TUVCREREek1FQkRERHpNRUJERER6TUVCREREek1FQkRERHpNRUJERER6TUVCREREek1FQkRERHpNRUJERER6TUVCREREek1FQkRERHpNRUJERER6TUVCREREek1FQkRERHpNRUJERER6TUVCREREeu3/A3BKLN40fzAMAAAAAElFTkSuQmCC",
      "text/plain": [
       "<Figure size 500x500 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Declare a figure with a custom size\n",
    "fig = plt.figure(figsize=(5, 5))\n",
    "\n",
    "# labels for the two classes\n",
    "labels = 'Positives', 'Negative'\n",
    "\n",
    "# Sizes for each slide\n",
    "sizes = [len(all_positive_tweets), len(all_negative_tweets)] \n",
    "\n",
    "# Declare pie chart, where the slices will be ordered and plotted counter-clockwise:\n",
    "plt.pie(sizes, labels=labels, autopct='%1.1f%%',\n",
    "        shadow=True, startangle=90)\n",
    "\n",
    "# Equal aspect ratio ensures that pie is drawn as a circle.\n",
    "plt.axis('equal')  \n",
    "\n",
    "# Display the chart\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Looking at raw texts\n",
    "\n",
    "Before anything else, we can print a couple of tweets from the dataset to see how they look. Understanding the data is responsible for 80% of the success or failure in data science projects. We can use this time to observe aspects we'd like to consider when preprocessing our data.\n",
    "\n",
    "Below, you will print one random positive and one random negative tweet. We have added a color mark at the beginning of the string to further distinguish the two. (Warning: This is taken from a public dataset of real tweets and a very small portion has explicit content.)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[92m.@1234Mania #jedzjabłka or #pijjabłka - whatever :) This article re Polish cider by @AB_Chapman is a #MustRead anyway @nawacki @TheEconomist\n",
      "\u001b[91m@keancipriano I want to see you :( sana makita na kita and ang @callalilyband .\n"
     ]
    }
   ],
   "source": [
    "# print positive in greeen\n",
    "print('\\033[92m' + all_positive_tweets[random.randint(0,5000)])\n",
    "\n",
    "# print negative in red\n",
    "print('\\033[91m' + all_negative_tweets[random.randint(0,5000)])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "One observation you may have is the presence of [emoticons](https://en.wikipedia.org/wiki/Emoticon) and URLs in many of the tweets. This info will come in handy in the next steps."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Preprocess raw text for Sentiment analysis"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Data preprocessing is one of the critical steps in any machine learning project. It includes cleaning and formatting the data before feeding into a machine learning algorithm. For NLP, the preprocessing steps are comprised of the following tasks:\n",
    "\n",
    "* Tokenizing the string\n",
    "* Lowercasing\n",
    "* Removing stop words and punctuation\n",
    "* Stemming\n",
    "\n",
    "The videos explained each of these steps and why they are important. Let's see how we can do these to a given tweet. We will choose just one and see how this is transformed by each preprocessing step."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "My beautiful sunflowers on a sunny Friday morning off :) #sunflowers #favourites #happy #Friday off… https://t.co/3tfYom0N1i\n"
     ]
    }
   ],
   "source": [
    "# Our selected sample. Complex enough to exemplify each step\n",
    "tweet = all_positive_tweets[2277]\n",
    "print(tweet)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's import a few more libraries for this purpose."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[nltk_data] Downloading package stopwords to\n",
      "[nltk_data]     /Users/johnmoses/nltk_data...\n",
      "[nltk_data]   Package stopwords is already up-to-date!\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# download the stopwords from NLTK\n",
    "nltk.download('stopwords')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "import re                                  # library for regular expression operations\n",
    "import string                              # for string operations\n",
    "\n",
    "from nltk.corpus import stopwords          # module for stop words that come with NLTK\n",
    "from nltk.stem import PorterStemmer        # module for stemming\n",
    "from nltk.tokenize import TweetTokenizer   # module for tokenizing strings"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Remove hyperlinks,  Twitter marks and styles\n",
    "\n",
    "Since we have a Twitter dataset, we'd like to remove some substrings commonly used on the platform like the hashtag, retweet marks, and hyperlinks. We'll use the [re](https://docs.python.org/3/library/re.html) library to perform regular expression operations on our tweet. We'll define our search pattern and use the `sub()` method to remove matches by substituting with an empty character (i.e. `''`)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[92mMy beautiful sunflowers on a sunny Friday morning off :) #sunflowers #favourites #happy #Friday off… https://t.co/3tfYom0N1i\n",
      "\u001b[94m\n",
      "My beautiful sunflowers on a sunny Friday morning off :) sunflowers favourites happy Friday off… \n"
     ]
    }
   ],
   "source": [
    "print('\\033[92m' + tweet)\n",
    "print('\\033[94m')\n",
    "\n",
    "# remove old style retweet text \"RT\"\n",
    "tweet2 = re.sub(r'^RT[\\s]+', '', tweet)\n",
    "\n",
    "# remove hyperlinks\n",
    "tweet2 = re.sub(r'https?://[^\\s\\n\\r]+', '', tweet2)\n",
    "\n",
    "# remove hashtags\n",
    "# only removing the hash # sign from the word\n",
    "tweet2 = re.sub(r'#', '', tweet2)\n",
    "\n",
    "print(tweet2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Tokenize the string\n",
    "\n",
    "To tokenize means to split the strings into individual words without blanks or tabs. In this same step, we will also convert each word in the string to lower case. The [tokenize](https://www.nltk.org/api/nltk.tokenize.html#module-nltk.tokenize.casual) module from NLTK allows us to do these easily:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "\u001b[92mMy beautiful sunflowers on a sunny Friday morning off :) sunflowers favourites happy Friday off… \n",
      "\u001b[94m\n",
      "\n",
      "Tokenized string:\n",
      "['my', 'beautiful', 'sunflowers', 'on', 'a', 'sunny', 'friday', 'morning', 'off', ':)', 'sunflowers', 'favourites', 'happy', 'friday', 'off', '…']\n"
     ]
    }
   ],
   "source": [
    "print()\n",
    "print('\\033[92m' + tweet2)\n",
    "print('\\033[94m')\n",
    "\n",
    "# instantiate tokenizer class\n",
    "tokenizer = TweetTokenizer(preserve_case=False, strip_handles=True,\n",
    "                               reduce_len=True)\n",
    "\n",
    "# tokenize tweets\n",
    "tweet_tokens = tokenizer.tokenize(tweet2)\n",
    "\n",
    "print()\n",
    "print('Tokenized string:')\n",
    "print(tweet_tokens)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Remove stop words and punctuations\n",
    "\n",
    "The next step is to remove stop words and punctuation. Stop words are words that don't add significant meaning to the text. You'll see the list provided by NLTK when you run the cells below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Stop words\n",
      "\n",
      "['i', 'me', 'my', 'myself', 'we', 'our', 'ours', 'ourselves', 'you', \"you're\", \"you've\", \"you'll\", \"you'd\", 'your', 'yours', 'yourself', 'yourselves', 'he', 'him', 'his', 'himself', 'she', \"she's\", 'her', 'hers', 'herself', 'it', \"it's\", 'its', 'itself', 'they', 'them', 'their', 'theirs', 'themselves', 'what', 'which', 'who', 'whom', 'this', 'that', \"that'll\", 'these', 'those', 'am', 'is', 'are', 'was', 'were', 'be', 'been', 'being', 'have', 'has', 'had', 'having', 'do', 'does', 'did', 'doing', 'a', 'an', 'the', 'and', 'but', 'if', 'or', 'because', 'as', 'until', 'while', 'of', 'at', 'by', 'for', 'with', 'about', 'against', 'between', 'into', 'through', 'during', 'before', 'after', 'above', 'below', 'to', 'from', 'up', 'down', 'in', 'out', 'on', 'off', 'over', 'under', 'again', 'further', 'then', 'once', 'here', 'there', 'when', 'where', 'why', 'how', 'all', 'any', 'both', 'each', 'few', 'more', 'most', 'other', 'some', 'such', 'no', 'nor', 'not', 'only', 'own', 'same', 'so', 'than', 'too', 'very', 's', 't', 'can', 'will', 'just', 'don', \"don't\", 'should', \"should've\", 'now', 'd', 'll', 'm', 'o', 're', 've', 'y', 'ain', 'aren', \"aren't\", 'couldn', \"couldn't\", 'didn', \"didn't\", 'doesn', \"doesn't\", 'hadn', \"hadn't\", 'hasn', \"hasn't\", 'haven', \"haven't\", 'isn', \"isn't\", 'ma', 'mightn', \"mightn't\", 'mustn', \"mustn't\", 'needn', \"needn't\", 'shan', \"shan't\", 'shouldn', \"shouldn't\", 'wasn', \"wasn't\", 'weren', \"weren't\", 'won', \"won't\", 'wouldn', \"wouldn't\"]\n",
      "\n",
      "Punctuation\n",
      "\n",
      "!\"#$%&'()*+,-./:;<=>?@[\\]^_`{|}~\n"
     ]
    }
   ],
   "source": [
    "#Import the english stop words list from NLTK\n",
    "stopwords_english = stopwords.words('english') \n",
    "\n",
    "print('Stop words\\n')\n",
    "print(stopwords_english)\n",
    "\n",
    "print('\\nPunctuation\\n')\n",
    "print(string.punctuation)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can see that the stop words list above contains some words that could be important in some contexts. \n",
    "These could be words like _i, not, between, because, won, against_. You might need to customize the stop words list for some applications. For our exercise, we will use the entire list.\n",
    "\n",
    "For the punctuation, we saw earlier that certain groupings like ':)' and '...'  should be retained when dealing with tweets because they are used to express emotions. In other contexts, like medical analysis, these should also be removed.\n",
    "\n",
    "Time to clean up our tokenized tweet!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "\u001b[92m\n",
      "['my', 'beautiful', 'sunflowers', 'on', 'a', 'sunny', 'friday', 'morning', 'off', ':)', 'sunflowers', 'favourites', 'happy', 'friday', 'off', '…']\n",
      "\u001b[94m\n",
      "removed stop words and punctuation:\n",
      "['beautiful', 'sunflowers', 'sunny', 'friday', 'morning', ':)', 'sunflowers', 'favourites', 'happy', 'friday', '…']\n"
     ]
    }
   ],
   "source": [
    "print()\n",
    "print('\\033[92m')\n",
    "print(tweet_tokens)\n",
    "print('\\033[94m')\n",
    "\n",
    "tweets_clean = []\n",
    "\n",
    "for word in tweet_tokens: # Go through every word in your tokens list\n",
    "    if (word not in stopwords_english and  # remove stopwords\n",
    "        word not in string.punctuation):  # remove punctuation\n",
    "        tweets_clean.append(word)\n",
    "\n",
    "print('removed stop words and punctuation:')\n",
    "print(tweets_clean)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Please note that the words **happy** and **sunny** in this list are correctly spelled. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Stemming\n",
    "\n",
    "Stemming is the process of converting a word to its most general form, or stem. This helps in reducing the size of our vocabulary.\n",
    "\n",
    "Consider the words: \n",
    " * **learn**\n",
    " * **learn**ing\n",
    " * **learn**ed\n",
    " * **learn**t\n",
    " \n",
    "All these words are stemmed from its common root **learn**. However, in some cases, the stemming process produces words that are not correct spellings of the root word. For example, **happi** and **sunni**. That's because it chooses the most common stem for related words. For example, we can look at the set of words that comprises the different forms of happy:\n",
    "\n",
    " * **happ**y\n",
    " * **happi**ness\n",
    " * **happi**er\n",
    " \n",
    "We can see that the prefix **happi** is more commonly used. We cannot choose **happ** because it is the stem of unrelated words like **happen**.\n",
    " \n",
    "NLTK has different modules for stemming and we will be using the [PorterStemmer](https://www.nltk.org/api/nltk.stem.html#module-nltk.stem.porter) module which uses the [Porter Stemming Algorithm](https://tartarus.org/martin/PorterStemmer/). Let's see how we can use it in the cell below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "\u001b[92m\n",
      "['beautiful', 'sunflowers', 'sunny', 'friday', 'morning', ':)', 'sunflowers', 'favourites', 'happy', 'friday', '…']\n",
      "\u001b[94m\n",
      "stemmed words:\n",
      "['beauti', 'sunflow', 'sunni', 'friday', 'morn', ':)', 'sunflow', 'favourit', 'happi', 'friday', '…']\n"
     ]
    }
   ],
   "source": [
    "print()\n",
    "print('\\033[92m')\n",
    "print(tweets_clean)\n",
    "print('\\033[94m')\n",
    "\n",
    "# Instantiate stemming class\n",
    "stemmer = PorterStemmer() \n",
    "\n",
    "# Create an empty list to store the stems\n",
    "tweets_stem = [] \n",
    "\n",
    "for word in tweets_clean:\n",
    "    stem_word = stemmer.stem(word)  # stemming word\n",
    "    tweets_stem.append(stem_word)  # append to the list\n",
    "\n",
    "print('stemmed words:')\n",
    "print(tweets_stem)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "That's it! Now we have a set of words we can feed into to the next stage of our machine learning project."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## process_tweet()\n",
    "\n",
    "As shown above, preprocessing consists of multiple steps before you arrive at the final list of words. We will not ask you to replicate these however. In the week's assignment, you will use the function `process_tweet(tweet)` available in _utils.py_. We encourage you to open the file and you'll see that this function's implementation is very similar to the steps above.\n",
    "\n",
    "To obtain the same result as in the previous code cells, you will only need to call the function `process_tweet()`. Let's do that in the next cell."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "\u001b[92m\n",
      "My beautiful sunflowers on a sunny Friday morning off :) #sunflowers #favourites #happy #Friday off… https://t.co/3tfYom0N1i\n",
      "\u001b[94m\n",
      "preprocessed tweet:\n",
      "['beauti', 'sunflow', 'sunni', 'friday', 'morn', ':)', 'sunflow', 'favourit', 'happi', 'friday', '…']\n"
     ]
    }
   ],
   "source": [
    "from utils import process_tweet # Import the process_tweet function\n",
    "\n",
    "# choose the same tweet\n",
    "tweet = all_positive_tweets[2277]\n",
    "\n",
    "print()\n",
    "print('\\033[92m')\n",
    "print(tweet)\n",
    "print('\\033[94m')\n",
    "\n",
    "# call the imported function\n",
    "tweets_stem = process_tweet(tweet); # Preprocess a given tweet\n",
    "\n",
    "print('preprocessed tweet:')\n",
    "print(tweets_stem) # Print the result"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "That's it for this lab! You now know what is going on when you call the preprocessing helper function in this week's assignment. Hopefully, this exercise has also given you some insights on how to tweak this for other types of text datasets."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "mlx",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.15"
  },
  "vscode": {
   "interpreter": {
    "hash": "4ea7dd683ad3989c9acf43b2110a0915f0e0607f4ae017ada7681046b918478b"
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}