model/twitter-sentiment-analysis.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Twitter Sentiment Analysis\n",
    "\n",
    "This notebook shows how a classification model is built to perform sentiment analysis on tweets. The end result is to be able to determin the *Polarity*, *Positive* or *Negative*, of each tweet coming from a real-time Twitter API.\n",
    "\n",
    "It is part of a larger project available on my GitHub: [twitter-sentiment-analysis](https://github.com/redouane-dev/twitter-sentiment-analysis).\n",
    "\n",
    "This notebook is inspired from [this DigitialOcean tutorial](https://www.digitalocean.com/community/tutorials/how-to-perform-sentiment-analysis-in-python-3-using-the-natural-language-toolkit-nltk).\n",
    "\n",
    "###### Steps\n",
    "\n",
    "1. Install the NLTK package (Natural Language Toolkit) and additional libraries to process tweets.\n",
    "2. Load datasets of positive and negative tweets\n",
    "3. Tokenize, normalize, and remove noise and stopwords from each tweet.\n",
    "4. Determine Word Density of the Dataset.\n",
    "5. Assemble the cleaned data into a dataset and split it into a training and testing sets.\n",
    "6. Training Day: Train a Naive Bayes classification model and validate it.\n",
    "7. Saving the model into binary format.\n",
    "\n",
    "###### Disclaimer\n",
    "\n",
    "Code snippets in the following notebook are used for demonstration only. They are not written with code formatting and optimization concerns in mind. See the GitHub repo mentioned above for classes and methods used in the live project."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Install NLTK and Dependencies\n",
    "\n",
    "To repeat the experiment, we will need to install Jupyter and the dependencies of this project.\n",
    "\n",
    "Following steps are optional:\n",
    "\n",
    "```bash\n",
    "# Create a virtual environment to isolate this project from other Python projects and avoid dependency conflicts\n",
    "virtualenv -p python3 venv\n",
    "\n",
    "# Activate your virtual env. You will see a (venv) before your usual terminal prompt\n",
    "source venv/bin/activate\n",
    "\n",
    "# If you want to use Jupyter and have it installed in this virtual environment\n",
    "pip install jupyter\n",
    "```\n",
    "\n",
    "Then comes the installation part:\n",
    "\n",
    "```bash\n",
    "# Install the single main dependency\n",
    "pip install nltk==3.4.5\n",
    "```\n",
    "\n",
    "...and voila!\n",
    "\n",
    "Or almost. We still need to install libraries that will help up process the tweets."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import nltk\n",
    "\n",
    "nltk.download('punkt')        # Contains a pre-trained model to help tokenize sentences into single words\n",
    "nltk.download('wordnet')      # Lexical database that will be used during normalization\n",
    "nltk.download('averaged_perceptron_tagger')    # Tagger to find nature of words (verb, noun, ...)\n",
    "nltk.download('stopwords')    # "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Load Datasets"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Download and store datasets locally\n",
    "nltk.download('twitter_samples')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['negative_tweets.json', 'positive_tweets.json', 'tweets.20150430-223406.json']"
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from nltk.corpus import twitter_samples\n",
    "\n",
    "# To see what are the available files\n",
    "twitter_samples.fileids()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Load the training set\n",
    "positive_tweets = twitter_samples.strings('positive_tweets.json')\n",
    "negative_tweets = twitter_samples.strings('negative_tweets.json')\n",
    "\n",
    "# Load the test set\n",
    "text = twitter_samples.strings('tweets.20150430-223406.json')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Tokenize, Normalize, and Remove Noise and Stopwords"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Tokenization\n",
    "\n",
    "This means splitting sentences into single words called *tokens*, including emojis :)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Example of a positive tweet:\n",
      "#FollowFriday @France_Inte @PKuchly57 @Milipol_Paris for being top engaged members in my community this week :)\n",
      "\n",
      "Tokens:\n",
      "['#FollowFriday', '@France_Inte', '@PKuchly57', '@Milipol_Paris', 'for', 'being', 'top', 'engaged', 'members', 'in', 'my', 'community', 'this', 'week', ':)']\n"
     ]
    }
   ],
   "source": [
    "from nltk.tokenize import TweetTokenizer\n",
    "\n",
    "# Instantiate a tweet tokenizer that will preserve each word (or token) as it is\n",
    "tweet_tokenizer = TweetTokenizer(\n",
    "    preserve_case = True,\n",
    "    reduce_len    = False,\n",
    "    strip_handles = False)\n",
    "\n",
    "tokens_positive = [tweet_tokenizer.tokenize(p) for p in positive_tweets]\n",
    "tokens_negative = [tweet_tokenizer.tokenize(n) for n in negative_tweets]\n",
    "\n",
    "print(\"Example of a positive tweet:\\n{}\\n\".format(positive_tweets[0]))\n",
    "print(\"Tokens:\\n{}\".format(tokens_positive[0]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Normalization\n",
    "\n",
    "Bringing words to their canonical form. We will use Lemmatization as a normalization process.\n",
    "\n",
    "We will need to find the nature of each word by using a tagger:\n",
    "- NNP: Noun, proper, singular\n",
    "- NN: Noun, common, singular or mass\n",
    "- IN: Preposition or conjunction, subordinating\n",
    "- VBG: Verb, gerund or present participle\n",
    "- VBN: Verb, past participle\n",
    "- JJ: adjective ‘big’\n",
    "- JJR: adjective, comparative ‘bigger’\n",
    "- JJS: adjective, superlative ‘biggest’\n",
    "- ...\n",
    "\n",
    "After getting the types (Verb, noun, or others), we can extract the lemma of each word."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('#FollowFriday', 'JJ'),\n",
       " ('@France_Inte', 'NNP'),\n",
       " ('@PKuchly57', 'NNP'),\n",
       " ('@Milipol_Paris', 'NNP'),\n",
       " ('for', 'IN'),\n",
       " ('being', 'VBG'),\n",
       " ('top', 'JJ'),\n",
       " ('engaged', 'VBN'),\n",
       " ('members', 'NNS'),\n",
       " ('in', 'IN'),\n",
       " ('my', 'PRP$'),\n",
       " ('community', 'NN'),\n",
       " ('this', 'DT'),\n",
       " ('week', 'NN'),\n",
       " (':)', 'NN')]"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from nltk.tag import pos_tag    # Part-of-speech tagger\n",
    "\n",
    "tags_positive = [pos_tag(p) for p in tokens_positive]\n",
    "tags_negative = [pos_tag(n) for n in tokens_negative]\n",
    "\n",
    "# print\n",
    "tags_positive[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Example of a positive tweet:\n",
      "#FollowFriday @France_Inte @PKuchly57 @Milipol_Paris for being top engaged members in my community this week :)\n",
      "\n",
      "Lemmatized:\n",
      "['#FollowFriday', '@France_Inte', '@PKuchly57', '@Milipol_Paris', 'for', 'be', 'top', 'engage', 'member', 'in', 'my', 'community', 'this', 'week', ':)']\n"
     ]
    }
   ],
   "source": [
    "from nltk.stem.wordnet import WordNetLemmatizer\n",
    "\n",
    "# All we need is to know the type (Noun, Verb, or others) of each word\n",
    "def _tag2type(tag):\n",
    "    '''\n",
    "    Take a tag and return a type.\n",
    "    return 'n' for noun, 'v' for verb, and 'a' for any\n",
    "    '''\n",
    "    if tag.startswith('NN'):\n",
    "        return 'n'\n",
    "    elif tag.startswith('VB'):\n",
    "        return 'v'\n",
    "    else:\n",
    "        return 'a'\n",
    "\n",
    "lemmatizer = WordNetLemmatizer()\n",
    "\n",
    "lemma_positive = [[lemmatizer.lemmatize(word, _tag2type(tag)) for (word, tag) in tags] for tags in tags_positive]\n",
    "lemma_negative = [[lemmatizer.lemmatize(word, _tag2type(tag)) for (word, tag) in tags] for tags in tags_negative]\n",
    "\n",
    "\n",
    "print(\"Example of a positive tweet:\\n{}\\n\".format(positive_tweets[0]))\n",
    "print(\"Lemmatized:\\n{}\".format(lemma_positive[0]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can notice that the verb *being* is converted to *be*, and the noun *members* to *member*."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## De-noising or Noise Reduction\n",
    "\n",
    "We consider the following as noise:\n",
    "1. Stopwords: Most common words in a language, such as \"a\", \"the\", and \"it\", generally don't convey a meaning, unless otherwise specified.\n",
    "2. Hyperlinks: Twitter uses t.co to shorten hyperlinks, which doesn't leave any value in the information left as URLs.\n",
    "3. Mentions: Usernames and pages that start with a @.\n",
    "4. Punctuation: It adds context and meaning, but makes the text more complex to process. For simplicity, we'll remove all punctuation.\n",
    "\n",
    "We will the dictionary *Stopwords* from NLTK, plus regular expressions to de-noise."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['i', 'me', 'my', 'myself', 'we', 'our', 'ours', 'ourselves', 'you', \"you're\"]"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from nltk.corpus import stopwords\n",
    "stopwords = stopwords.words('english')\n",
    "\n",
    "# print\n",
    "stopwords[:10]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Example of a positive tweet:\n",
      "#FollowFriday @France_Inte @PKuchly57 @Milipol_Paris for being top engaged members in my community this week :)\n",
      "\n",
      "Denoised:\n",
      "['#followfriday', 'top', 'engage', 'member', 'community', 'week', ':)']\n"
     ]
    }
   ],
   "source": [
    "import re\n",
    "from string import punctuation\n",
    "\n",
    "def _is_noise(word):\n",
    "    pattern = 'http[s]?://(?:[a-zA-Z]|[0-9]|[$-_@.&+#]|[!*\\(\\),]|(?:%[0-9a-fA-F][0-9a-fA-F]))+|(@[A-Za-z0-9_]+)'\n",
    "    return word in punctuation \\\n",
    "        or word.lower() in stopwords \\\n",
    "        or re.search(pattern, word, re.IGNORECASE) != None\n",
    "\n",
    "denoised_positive = [[p.lower() for p in _list if not _is_noise(p)] for _list in lemma_positive]\n",
    "denoised_negative = [[n.lower() for n in _list if not _is_noise(n)] for _list in lemma_negative]\n",
    "\n",
    "print(\"Example of a positive tweet:\\n{}\\n\".format(positive_tweets[0]))\n",
    "print(\"Denoised:\\n{}\".format(denoised_positive[0]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Determining Word Density of the Dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The 10 most common words in a set of positive tweets:\n",
      "[(':)', 3691), (':-)', 701), (':d', 658), ('thanks', 388), ('follow', 357), ('love', 333), ('...', 290), ('good', 283), ('get', 263), ('thank', 253)]\n",
      "\n",
      "The 10 most common words in a set of negative tweets:\n",
      "[(':(', 4585), (':-(', 501), (\"i'm\", 343), ('...', 332), ('get', 325), ('miss', 291), ('go', 275), ('please', 275), ('want', 246), ('like', 218)]\n"
     ]
    }
   ],
   "source": [
    "from nltk import FreqDist\n",
    "\n",
    "def get_all_words(tokens_list):\n",
    "    '''\n",
    "    Generator function to get a flat mapping of all words in the dataset.\n",
    "    \n",
    "    @arg tokens_list: A 2-D list of (preferably cleaned) tokens\n",
    "    @return A list of all words\n",
    "    '''\n",
    "    for tokens in tokens_list:\n",
    "        for token in tokens:\n",
    "            yield token\n",
    "\n",
    "all_pos_words = get_all_words(denoised_positive)\n",
    "all_neg_words = get_all_words(denoised_negative)\n",
    "\n",
    "freq_dist_pos = FreqDist(all_pos_words)\n",
    "freq_dist_neg = FreqDist(all_neg_words)\n",
    "\n",
    "print(\"The 10 most common words in a set of positive tweets:\\n{}\\n\".format(freq_dist_pos.most_common(10)))\n",
    "print(\"The 10 most common words in a set of negative tweets:\\n{}\".format(freq_dist_neg.most_common(10)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Preparing Training and Testing Datasets\n",
    "\n",
    "We split our dataset into a training set for building the model, and a testing set for testing the performance of our model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_tweets_for_model(tokens_list):\n",
    "    '''\n",
    "    Generator function that associates a boolean 'True' to each token in a list of tokens,\n",
    "    which represents the label of each token.\n",
    "    This step is required by the NLTK classifier we'll be using:\n",
    "    - Documentation: https://www.nltk.org/book/ch06.html\n",
    "    \n",
    "    @arg tokens_list a 2-D list of (preferably cleaned) tokens\n",
    "    @return A 2-D list of tuples (original_token, True) containing the unaltered token and a boolean label\n",
    "    '''\n",
    "    for tweet_tokens in tokens_list:\n",
    "        yield dict([token, True] for token in tweet_tokens)\n",
    "\n",
    "positive_tokens_for_model = get_tweets_for_model(denoised_positive)\n",
    "negative_tokens_for_model = get_tweets_for_model(denoised_negative)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "import random\n",
    "\n",
    "TRAIN_SIZE_RATIO = 0.7    # We use 70% as a training set\n",
    "\n",
    "positive_dataset = [(tweet_dict, \"Positive\") for tweet_dict in positive_tokens_for_model]\n",
    "negative_dataset = [(tweet_dict, \"Negative\") for tweet_dict in negative_tokens_for_model]\n",
    "\n",
    "# Merge the positive and negative sets, then shuffle to avoid any bias\n",
    "# that could come from the arrangement of tweets.\n",
    "dataset = positive_dataset + negative_dataset\n",
    "random.shuffle(dataset)\n",
    "\n",
    "train_data = dataset[: round(len(dataset) * TRAIN_SIZE_RATIO)]\n",
    "test_data = dataset[round(len(dataset) * TRAIN_SIZE_RATIO) :]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Training Day\n",
    "\n",
    "For the sake of simplicity, speed, and a limited dataset, we will use a Naive Bayes classifier."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training accuracy is:0.9995714285714286\n",
      "\n",
      "Testing accuracy is:0.9953333333333333\n",
      "\n",
      "Most Informative Features\n",
      "                      :) = True           Positi : Negati =    988.2 : 1.0\n",
      "                     sad = True           Negati : Positi =     31.1 : 1.0\n",
      "                follower = True           Positi : Negati =     23.6 : 1.0\n",
      "                     bam = True           Positi : Negati =     21.9 : 1.0\n",
      "                    glad = True           Positi : Negati =     19.8 : 1.0\n",
      "                     x15 = True           Negati : Positi =     16.9 : 1.0\n",
      "                 welcome = True           Positi : Negati =     15.1 : 1.0\n",
      "               community = True           Positi : Negati =     14.5 : 1.0\n",
      "                     ugh = True           Negati : Positi =     12.9 : 1.0\n",
      "                    dont = True           Negati : Positi =     12.6 : 1.0\n",
      "None\n"
     ]
    }
   ],
   "source": [
    "from nltk import classify\n",
    "from nltk import NaiveBayesClassifier\n",
    "\n",
    "classifier = NaiveBayesClassifier.train(train_data)\n",
    "\n",
    "print(\"Training accuracy is:{}\\n\".format(classify.accuracy(classifier, train_data)))\n",
    "print(\"Testing accuracy is:{}\\n\".format(classify.accuracy(classifier, test_data)))\n",
    "print(classifier.show_most_informative_features(10))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Custom testing\n",
    "\n",
    "We wrap our classification algorithm into a function for ease of use, then we perform tests on various emotions."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "def classify(tweet):\n",
    "    '''\n",
    "    Wrapper function for the pre-processing and classification steps previously performed.\n",
    "    \n",
    "    @arg tweet: String representing a tweet\n",
    "    @return String representing a polarity. (Positive or Negative)\n",
    "    '''\n",
    "    tokens = tweet_tokenizer.tokenize(tweet)\n",
    "    tokens = [\n",
    "        lemmatizer.lemmatize(word, _tag2type(tag)).lower()\n",
    "        for word, tag in pos_tag(tokens)\n",
    "        if not _is_noise(word)\n",
    "    ]\n",
    "    \n",
    "    return tokens, classifier.classify(dict([token, True] for token in tokens))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Denoised tokens: ['thanks', 'pie', 'really', 'appreciate', ':)', '#yummy', '#pie_day']\n",
      "Polarity: Positive\n",
      "\n"
     ]
    }
   ],
   "source": [
    "positive_tweet = \"@bakery_brothers Thanks for the Pie! Really appreciate it :) #yummy #pie_day\"\n",
    "tokens, polarity = classify(positive_tweet)\n",
    "\n",
    "print(\"Denoised tokens: {}\\nPolarity: {}\\n\".format(tokens, polarity))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Denoised tokens: ['really', 'sad', 'lose', 'qualification', 'final', '#no_luck']\n",
      "Polarity: Negative\n",
      "\n"
     ]
    }
   ],
   "source": [
    "negative_tweet = \"@raptors really sad that you lost the qualifications to the final. #no_luck\"\n",
    "tokens, polarity = classify(negative_tweet)\n",
    "\n",
    "print(\"Denoised tokens: {}\\nPolarity: {}\\n\".format(tokens, polarity))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Denoised tokens: ['thank', 'much', 'close', 'half', 'road', 'city', 'middle', 'day', '#traffic']\n",
      "Polarity: Positive\n",
      "\n"
     ]
    }
   ],
   "source": [
    "sarcasme_tweet = \"@police thank you so much for closing half the roads to the city in the middle of the day! #traffic\"\n",
    "tokens, polarity = classify(sarcasme_tweet)\n",
    "\n",
    "print(\"Denoised tokens: {}\\nPolarity: {}\\n\".format(tokens, polarity))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Conclusions\n",
    "\n",
    "We can see that the model is not able to recognize sarcasme for lack of data in the training set.\n",
    "\n",
    "Training a more complex model that would recognize more evolved emotions requires a training set that contains all of those emotions, and evantually a classification algorithm that can cope with this complexity."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Saving Model into Binary File"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pickle\n",
    "\n",
    "with open('./model.pickle', 'wb') as f:\n",
    "    pickle.dump(classifier, f)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}