Week_05_Natural_Language_Processing.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "Week_05_Natural_Language_Processing.ipynb",
      "provenance": []
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "L8opw2UZFaQv"
      },
      "source": [
        "# Natural Language Processing with Python\n",
        "\n",
        "In this workshop we will use the [Sentiment Labelled Sentences Data Set](https://archive.ics.uci.edu/ml/datasets/Sentiment+Labelled+Sentences)  that contains 500 positive and 500 negative sentences from imdb.com, amazon.com and yelp.com.  \n",
        "\n",
        "* First, we will explore preliminary text analytics and text pre-processing .  \n",
        "* Second, we will evaluate different feature extraction mechanisms for text.  \n",
        "* Third, we will evaluate a simple text classification for Amazon review dataset, and an advanced deep neural network for classification accuracy improvement.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "4_z-LSYHJVxx"
      },
      "source": [
        "## Load the data\n",
        "\n",
        "Load the reviews.csv file downloaded from LMS to the Google Colab file repository."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "4XCT8rJKJv0j"
      },
      "source": [
        "import pandas as pd"
      ],
      "execution_count": 4,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "MWP9fyKaNRkN"
      },
      "source": [
        "# Dataset reveiws.csv MUST be uploaded to Google Colab before executing this line\n",
        "df = pd.read_csv('tp6_reviews.csv')"
      ],
      "execution_count": 5,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "KblPiHu6ZfT1"
      },
      "source": [
        "View a summary of the dataset."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "qlFws42RNS3Q",
        "outputId": "9f01ca8d-82ba-4692-beee-0f5163015c7b",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "source": [
        "df.info()"
      ],
      "execution_count": 6,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "<class 'pandas.core.frame.DataFrame'>\n",
            "RangeIndex: 2748 entries, 0 to 2747\n",
            "Data columns (total 3 columns):\n",
            " #   Column    Non-Null Count  Dtype \n",
            "---  ------    --------------  ----- \n",
            " 0   sentence  2748 non-null   object\n",
            " 1   label     2748 non-null   int64 \n",
            " 2   source    2748 non-null   object\n",
            "dtypes: int64(1), object(2)\n",
            "memory usage: 64.5+ KB\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "uroM_ht8N9Xt",
        "outputId": "5c77d9c7-c9f5-4605-fdf1-46bec51e8e44",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 206
        }
      },
      "source": [
        "df.head()"
      ],
      "execution_count": 7,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "                                            sentence  label source\n",
              "0                           Wow... Loved this place.      1   yelp\n",
              "1                                 Crust is not good.      0   yelp\n",
              "2          Not tasty and the texture was just nasty.      0   yelp\n",
              "3  Stopped by during the late May bank holiday of...      1   yelp\n",
              "4  The selection on the menu was great and so wer...      1   yelp"
            ],
            "text/html": [
              "\n",
              "  <div id=\"df-ee88e2bf-ddc9-4ae3-bfe3-a665e1514cdc\" class=\"colab-df-container\">\n",
              "    <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>sentence</th>\n",
              "      <th>label</th>\n",
              "      <th>source</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>Wow... Loved this place.</td>\n",
              "      <td>1</td>\n",
              "      <td>yelp</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>Crust is not good.</td>\n",
              "      <td>0</td>\n",
              "      <td>yelp</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>Not tasty and the texture was just nasty.</td>\n",
              "      <td>0</td>\n",
              "      <td>yelp</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>Stopped by during the late May bank holiday of...</td>\n",
              "      <td>1</td>\n",
              "      <td>yelp</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>The selection on the menu was great and so wer...</td>\n",
              "      <td>1</td>\n",
              "      <td>yelp</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>\n",
              "    <div class=\"colab-df-buttons\">\n",
              "\n",
              "  <div class=\"colab-df-container\">\n",
              "    <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-ee88e2bf-ddc9-4ae3-bfe3-a665e1514cdc')\"\n",
              "            title=\"Convert this dataframe to an interactive table.\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\" viewBox=\"0 -960 960 960\">\n",
              "    <path d=\"M120-120v-720h720v720H120Zm60-500h600v-160H180v160Zm220 220h160v-160H400v160Zm0 220h160v-160H400v160ZM180-400h160v-160H180v160Zm440 0h160v-160H620v160ZM180-180h160v-160H180v160Zm440 0h160v-160H620v160Z\"/>\n",
              "  </svg>\n",
              "    </button>\n",
              "\n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    .colab-df-buttons div {\n",
              "      margin-bottom: 4px;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "    <script>\n",
              "      const buttonEl =\n",
              "        document.querySelector('#df-ee88e2bf-ddc9-4ae3-bfe3-a665e1514cdc button.colab-df-convert');\n",
              "      buttonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "      async function convertToInteractive(key) {\n",
              "        const element = document.querySelector('#df-ee88e2bf-ddc9-4ae3-bfe3-a665e1514cdc');\n",
              "        const dataTable =\n",
              "          await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                    [key], {});\n",
              "        if (!dataTable) return;\n",
              "\n",
              "        const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "          '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "          + ' to learn more about interactive tables.';\n",
              "        element.innerHTML = '';\n",
              "        dataTable['output_type'] = 'display_data';\n",
              "        await google.colab.output.renderOutput(dataTable, element);\n",
              "        const docLink = document.createElement('div');\n",
              "        docLink.innerHTML = docLinkHtml;\n",
              "        element.appendChild(docLink);\n",
              "      }\n",
              "    </script>\n",
              "  </div>\n",
              "\n",
              "\n",
              "<div id=\"df-d7c728eb-e369-4d3e-90d5-740e84d2f9df\">\n",
              "  <button class=\"colab-df-quickchart\" onclick=\"quickchart('df-d7c728eb-e369-4d3e-90d5-740e84d2f9df')\"\n",
              "            title=\"Suggest charts\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "<svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "     width=\"24px\">\n",
              "    <g>\n",
              "        <path d=\"M19 3H5c-1.1 0-2 .9-2 2v14c0 1.1.9 2 2 2h14c1.1 0 2-.9 2-2V5c0-1.1-.9-2-2-2zM9 17H7v-7h2v7zm4 0h-2V7h2v10zm4 0h-2v-4h2v4z\"/>\n",
              "    </g>\n",
              "</svg>\n",
              "  </button>\n",
              "\n",
              "<style>\n",
              "  .colab-df-quickchart {\n",
              "      --bg-color: #E8F0FE;\n",
              "      --fill-color: #1967D2;\n",
              "      --hover-bg-color: #E2EBFA;\n",
              "      --hover-fill-color: #174EA6;\n",
              "      --disabled-fill-color: #AAA;\n",
              "      --disabled-bg-color: #DDD;\n",
              "  }\n",
              "\n",
              "  [theme=dark] .colab-df-quickchart {\n",
              "      --bg-color: #3B4455;\n",
              "      --fill-color: #D2E3FC;\n",
              "      --hover-bg-color: #434B5C;\n",
              "      --hover-fill-color: #FFFFFF;\n",
              "      --disabled-bg-color: #3B4455;\n",
              "      --disabled-fill-color: #666;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart {\n",
              "    background-color: var(--bg-color);\n",
              "    border: none;\n",
              "    border-radius: 50%;\n",
              "    cursor: pointer;\n",
              "    display: none;\n",
              "    fill: var(--fill-color);\n",
              "    height: 32px;\n",
              "    padding: 0;\n",
              "    width: 32px;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart:hover {\n",
              "    background-color: var(--hover-bg-color);\n",
              "    box-shadow: 0 1px 2px rgba(60, 64, 67, 0.3), 0 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "    fill: var(--button-hover-fill-color);\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart-complete:disabled,\n",
              "  .colab-df-quickchart-complete:disabled:hover {\n",
              "    background-color: var(--disabled-bg-color);\n",
              "    fill: var(--disabled-fill-color);\n",
              "    box-shadow: none;\n",
              "  }\n",
              "\n",
              "  .colab-df-spinner {\n",
              "    border: 2px solid var(--fill-color);\n",
              "    border-color: transparent;\n",
              "    border-bottom-color: var(--fill-color);\n",
              "    animation:\n",
              "      spin 1s steps(1) infinite;\n",
              "  }\n",
              "\n",
              "  @keyframes spin {\n",
              "    0% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "      border-left-color: var(--fill-color);\n",
              "    }\n",
              "    20% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    30% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    40% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    60% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    80% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "    90% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "  }\n",
              "</style>\n",
              "\n",
              "  <script>\n",
              "    async function quickchart(key) {\n",
              "      const quickchartButtonEl =\n",
              "        document.querySelector('#' + key + ' button');\n",
              "      quickchartButtonEl.disabled = true;  // To prevent multiple clicks.\n",
              "      quickchartButtonEl.classList.add('colab-df-spinner');\n",
              "      try {\n",
              "        const charts = await google.colab.kernel.invokeFunction(\n",
              "            'suggestCharts', [key], {});\n",
              "      } catch (error) {\n",
              "        console.error('Error during call to suggestCharts:', error);\n",
              "      }\n",
              "      quickchartButtonEl.classList.remove('colab-df-spinner');\n",
              "      quickchartButtonEl.classList.add('colab-df-quickchart-complete');\n",
              "    }\n",
              "    (() => {\n",
              "      let quickchartButtonEl =\n",
              "        document.querySelector('#df-d7c728eb-e369-4d3e-90d5-740e84d2f9df button');\n",
              "      quickchartButtonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "    })();\n",
              "  </script>\n",
              "</div>\n",
              "\n",
              "    </div>\n",
              "  </div>\n"
            ],
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "dataframe",
              "variable_name": "df",
              "summary": "{\n  \"name\": \"df\",\n  \"rows\": 2748,\n  \"fields\": [\n    {\n      \"column\": \"sentence\",\n      \"properties\": {\n        \"dtype\": \"string\",\n        \"num_unique_values\": 2728,\n        \"samples\": [\n          \"I have to jiggle the plug to get it to line up right to get decent volume.\",\n          \"Unfortunately, any virtue in this film's production work was lost on a regrettable script.  \",\n          \"A great film by a great director.  \"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"label\",\n      \"properties\": {\n        \"dtype\": \"number\",\n        \"std\": 0,\n        \"min\": 0,\n        \"max\": 1,\n        \"num_unique_values\": 2,\n        \"samples\": [\n          0,\n          1\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"source\",\n      \"properties\": {\n        \"dtype\": \"category\",\n        \"num_unique_values\": 3,\n        \"samples\": [\n          \"yelp\",\n          \"amazon\"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    }\n  ]\n}"
            }
          },
          "metadata": {},
          "execution_count": 7
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "3FlpqeMJ4dyW"
      },
      "source": [
        "Check what are the sources of data available in the dataset."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "sqjbA0GZ4Zlc",
        "outputId": "fefaf925-29b8-4978-f442-2ecdb92d7caf",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "source": [
        "df['source'].unique()"
      ],
      "execution_count": 8,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "array(['yelp', 'amazon', 'imdb'], dtype=object)"
            ]
          },
          "metadata": {},
          "execution_count": 8
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "kFPeBYVw3gDz"
      },
      "source": [
        "## Preliminary analysis"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "O3xamOU23qJ0"
      },
      "source": [
        "***Pandas dataframe's df.apply() function***  \n",
        "* This allow the users to pass a function and apply it on every single value of the Pandas series, i.e., column. [API documentation.](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.apply.html)\n",
        "*  Efficient way to update values in a dataframe column\n",
        "\n",
        "In the following example,\n",
        "\n",
        "\n",
        "*   Count the number of words in each sentence\n",
        "*   Assign the word count to a new attribute  named 'word_count'\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "fEYoC-bJ6Uj5"
      },
      "source": [
        "def word_counter(document):\n",
        "  split_word = str(document).split(\" \") # split by white space\n",
        "  word_count = len(split_word) # count the words\n",
        "  return word_count\n",
        "\n",
        "df['word_count_function'] = df['sentence'].apply(word_counter)"
      ],
      "execution_count": 9,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "cIyW2QaG6pbS",
        "outputId": "c96af467-f105-4921-fbc3-4b8e4bcb7b64",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 206
        }
      },
      "source": [
        "df.head(5)"
      ],
      "execution_count": 10,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "                                            sentence  label source  \\\n",
              "0                           Wow... Loved this place.      1   yelp   \n",
              "1                                 Crust is not good.      0   yelp   \n",
              "2          Not tasty and the texture was just nasty.      0   yelp   \n",
              "3  Stopped by during the late May bank holiday of...      1   yelp   \n",
              "4  The selection on the menu was great and so wer...      1   yelp   \n",
              "\n",
              "   word_count_function  \n",
              "0                    4  \n",
              "1                    4  \n",
              "2                    8  \n",
              "3                   15  \n",
              "4                   12  "
            ],
            "text/html": [
              "\n",
              "  <div id=\"df-9a506c76-fd12-4ff4-9e35-2a9d3b691471\" class=\"colab-df-container\">\n",
              "    <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>sentence</th>\n",
              "      <th>label</th>\n",
              "      <th>source</th>\n",
              "      <th>word_count_function</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>Wow... Loved this place.</td>\n",
              "      <td>1</td>\n",
              "      <td>yelp</td>\n",
              "      <td>4</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>Crust is not good.</td>\n",
              "      <td>0</td>\n",
              "      <td>yelp</td>\n",
              "      <td>4</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>Not tasty and the texture was just nasty.</td>\n",
              "      <td>0</td>\n",
              "      <td>yelp</td>\n",
              "      <td>8</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>Stopped by during the late May bank holiday of...</td>\n",
              "      <td>1</td>\n",
              "      <td>yelp</td>\n",
              "      <td>15</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>The selection on the menu was great and so wer...</td>\n",
              "      <td>1</td>\n",
              "      <td>yelp</td>\n",
              "      <td>12</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>\n",
              "    <div class=\"colab-df-buttons\">\n",
              "\n",
              "  <div class=\"colab-df-container\">\n",
              "    <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-9a506c76-fd12-4ff4-9e35-2a9d3b691471')\"\n",
              "            title=\"Convert this dataframe to an interactive table.\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\" viewBox=\"0 -960 960 960\">\n",
              "    <path d=\"M120-120v-720h720v720H120Zm60-500h600v-160H180v160Zm220 220h160v-160H400v160Zm0 220h160v-160H400v160ZM180-400h160v-160H180v160Zm440 0h160v-160H620v160ZM180-180h160v-160H180v160Zm440 0h160v-160H620v160Z\"/>\n",
              "  </svg>\n",
              "    </button>\n",
              "\n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    .colab-df-buttons div {\n",
              "      margin-bottom: 4px;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "    <script>\n",
              "      const buttonEl =\n",
              "        document.querySelector('#df-9a506c76-fd12-4ff4-9e35-2a9d3b691471 button.colab-df-convert');\n",
              "      buttonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "      async function convertToInteractive(key) {\n",
              "        const element = document.querySelector('#df-9a506c76-fd12-4ff4-9e35-2a9d3b691471');\n",
              "        const dataTable =\n",
              "          await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                    [key], {});\n",
              "        if (!dataTable) return;\n",
              "\n",
              "        const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "          '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "          + ' to learn more about interactive tables.';\n",
              "        element.innerHTML = '';\n",
              "        dataTable['output_type'] = 'display_data';\n",
              "        await google.colab.output.renderOutput(dataTable, element);\n",
              "        const docLink = document.createElement('div');\n",
              "        docLink.innerHTML = docLinkHtml;\n",
              "        element.appendChild(docLink);\n",
              "      }\n",
              "    </script>\n",
              "  </div>\n",
              "\n",
              "\n",
              "<div id=\"df-ea2e9e1d-987b-4011-987b-b0a328cd307f\">\n",
              "  <button class=\"colab-df-quickchart\" onclick=\"quickchart('df-ea2e9e1d-987b-4011-987b-b0a328cd307f')\"\n",
              "            title=\"Suggest charts\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "<svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "     width=\"24px\">\n",
              "    <g>\n",
              "        <path d=\"M19 3H5c-1.1 0-2 .9-2 2v14c0 1.1.9 2 2 2h14c1.1 0 2-.9 2-2V5c0-1.1-.9-2-2-2zM9 17H7v-7h2v7zm4 0h-2V7h2v10zm4 0h-2v-4h2v4z\"/>\n",
              "    </g>\n",
              "</svg>\n",
              "  </button>\n",
              "\n",
              "<style>\n",
              "  .colab-df-quickchart {\n",
              "      --bg-color: #E8F0FE;\n",
              "      --fill-color: #1967D2;\n",
              "      --hover-bg-color: #E2EBFA;\n",
              "      --hover-fill-color: #174EA6;\n",
              "      --disabled-fill-color: #AAA;\n",
              "      --disabled-bg-color: #DDD;\n",
              "  }\n",
              "\n",
              "  [theme=dark] .colab-df-quickchart {\n",
              "      --bg-color: #3B4455;\n",
              "      --fill-color: #D2E3FC;\n",
              "      --hover-bg-color: #434B5C;\n",
              "      --hover-fill-color: #FFFFFF;\n",
              "      --disabled-bg-color: #3B4455;\n",
              "      --disabled-fill-color: #666;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart {\n",
              "    background-color: var(--bg-color);\n",
              "    border: none;\n",
              "    border-radius: 50%;\n",
              "    cursor: pointer;\n",
              "    display: none;\n",
              "    fill: var(--fill-color);\n",
              "    height: 32px;\n",
              "    padding: 0;\n",
              "    width: 32px;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart:hover {\n",
              "    background-color: var(--hover-bg-color);\n",
              "    box-shadow: 0 1px 2px rgba(60, 64, 67, 0.3), 0 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "    fill: var(--button-hover-fill-color);\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart-complete:disabled,\n",
              "  .colab-df-quickchart-complete:disabled:hover {\n",
              "    background-color: var(--disabled-bg-color);\n",
              "    fill: var(--disabled-fill-color);\n",
              "    box-shadow: none;\n",
              "  }\n",
              "\n",
              "  .colab-df-spinner {\n",
              "    border: 2px solid var(--fill-color);\n",
              "    border-color: transparent;\n",
              "    border-bottom-color: var(--fill-color);\n",
              "    animation:\n",
              "      spin 1s steps(1) infinite;\n",
              "  }\n",
              "\n",
              "  @keyframes spin {\n",
              "    0% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "      border-left-color: var(--fill-color);\n",
              "    }\n",
              "    20% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    30% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    40% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    60% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    80% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "    90% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "  }\n",
              "</style>\n",
              "\n",
              "  <script>\n",
              "    async function quickchart(key) {\n",
              "      const quickchartButtonEl =\n",
              "        document.querySelector('#' + key + ' button');\n",
              "      quickchartButtonEl.disabled = true;  // To prevent multiple clicks.\n",
              "      quickchartButtonEl.classList.add('colab-df-spinner');\n",
              "      try {\n",
              "        const charts = await google.colab.kernel.invokeFunction(\n",
              "            'suggestCharts', [key], {});\n",
              "      } catch (error) {\n",
              "        console.error('Error during call to suggestCharts:', error);\n",
              "      }\n",
              "      quickchartButtonEl.classList.remove('colab-df-spinner');\n",
              "      quickchartButtonEl.classList.add('colab-df-quickchart-complete');\n",
              "    }\n",
              "    (() => {\n",
              "      let quickchartButtonEl =\n",
              "        document.querySelector('#df-ea2e9e1d-987b-4011-987b-b0a328cd307f button');\n",
              "      quickchartButtonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "    })();\n",
              "  </script>\n",
              "</div>\n",
              "\n",
              "    </div>\n",
              "  </div>\n"
            ],
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "dataframe",
              "variable_name": "df",
              "summary": "{\n  \"name\": \"df\",\n  \"rows\": 2748,\n  \"fields\": [\n    {\n      \"column\": \"sentence\",\n      \"properties\": {\n        \"dtype\": \"string\",\n        \"num_unique_values\": 2728,\n        \"samples\": [\n          \"I have to jiggle the plug to get it to line up right to get decent volume.\",\n          \"Unfortunately, any virtue in this film's production work was lost on a regrettable script.  \",\n          \"A great film by a great director.  \"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"label\",\n      \"properties\": {\n        \"dtype\": \"number\",\n        \"std\": 0,\n        \"min\": 0,\n        \"max\": 1,\n        \"num_unique_values\": 2,\n        \"samples\": [\n          0,\n          1\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"source\",\n      \"properties\": {\n        \"dtype\": \"category\",\n        \"num_unique_values\": 3,\n        \"samples\": [\n          \"yelp\",\n          \"amazon\"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"word_count_function\",\n      \"properties\": {\n        \"dtype\": \"number\",\n        \"std\": 35,\n        \"min\": 1,\n        \"max\": 1393,\n        \"num_unique_values\": 57,\n        \"samples\": [\n          4,\n          6\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    }\n  ]\n}"
            }
          },
          "metadata": {},
          "execution_count": 10
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ZNKXacvx60GL"
      },
      "source": [
        "Same above function can be achieved through a simple lambda function."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "6767ZK4ZOAhd",
        "outputId": "81632b88-b562-4ba0-a21a-d43b7d5ea23c",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 206
        }
      },
      "source": [
        "df['word_count'] = df['sentence'].apply(lambda x: len(str(x).split(\" \")))\n",
        "df.head(5)"
      ],
      "execution_count": 11,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "                                            sentence  label source  \\\n",
              "0                           Wow... Loved this place.      1   yelp   \n",
              "1                                 Crust is not good.      0   yelp   \n",
              "2          Not tasty and the texture was just nasty.      0   yelp   \n",
              "3  Stopped by during the late May bank holiday of...      1   yelp   \n",
              "4  The selection on the menu was great and so wer...      1   yelp   \n",
              "\n",
              "   word_count_function  word_count  \n",
              "0                    4           4  \n",
              "1                    4           4  \n",
              "2                    8           8  \n",
              "3                   15          15  \n",
              "4                   12          12  "
            ],
            "text/html": [
              "\n",
              "  <div id=\"df-49917c2f-7c8b-4e73-805f-f7d8b0e48739\" class=\"colab-df-container\">\n",
              "    <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>sentence</th>\n",
              "      <th>label</th>\n",
              "      <th>source</th>\n",
              "      <th>word_count_function</th>\n",
              "      <th>word_count</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>Wow... Loved this place.</td>\n",
              "      <td>1</td>\n",
              "      <td>yelp</td>\n",
              "      <td>4</td>\n",
              "      <td>4</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>Crust is not good.</td>\n",
              "      <td>0</td>\n",
              "      <td>yelp</td>\n",
              "      <td>4</td>\n",
              "      <td>4</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>Not tasty and the texture was just nasty.</td>\n",
              "      <td>0</td>\n",
              "      <td>yelp</td>\n",
              "      <td>8</td>\n",
              "      <td>8</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>Stopped by during the late May bank holiday of...</td>\n",
              "      <td>1</td>\n",
              "      <td>yelp</td>\n",
              "      <td>15</td>\n",
              "      <td>15</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>The selection on the menu was great and so wer...</td>\n",
              "      <td>1</td>\n",
              "      <td>yelp</td>\n",
              "      <td>12</td>\n",
              "      <td>12</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>\n",
              "    <div class=\"colab-df-buttons\">\n",
              "\n",
              "  <div class=\"colab-df-container\">\n",
              "    <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-49917c2f-7c8b-4e73-805f-f7d8b0e48739')\"\n",
              "            title=\"Convert this dataframe to an interactive table.\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\" viewBox=\"0 -960 960 960\">\n",
              "    <path d=\"M120-120v-720h720v720H120Zm60-500h600v-160H180v160Zm220 220h160v-160H400v160Zm0 220h160v-160H400v160ZM180-400h160v-160H180v160Zm440 0h160v-160H620v160ZM180-180h160v-160H180v160Zm440 0h160v-160H620v160Z\"/>\n",
              "  </svg>\n",
              "    </button>\n",
              "\n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    .colab-df-buttons div {\n",
              "      margin-bottom: 4px;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "    <script>\n",
              "      const buttonEl =\n",
              "        document.querySelector('#df-49917c2f-7c8b-4e73-805f-f7d8b0e48739 button.colab-df-convert');\n",
              "      buttonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "      async function convertToInteractive(key) {\n",
              "        const element = document.querySelector('#df-49917c2f-7c8b-4e73-805f-f7d8b0e48739');\n",
              "        const dataTable =\n",
              "          await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                    [key], {});\n",
              "        if (!dataTable) return;\n",
              "\n",
              "        const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "          '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "          + ' to learn more about interactive tables.';\n",
              "        element.innerHTML = '';\n",
              "        dataTable['output_type'] = 'display_data';\n",
              "        await google.colab.output.renderOutput(dataTable, element);\n",
              "        const docLink = document.createElement('div');\n",
              "        docLink.innerHTML = docLinkHtml;\n",
              "        element.appendChild(docLink);\n",
              "      }\n",
              "    </script>\n",
              "  </div>\n",
              "\n",
              "\n",
              "<div id=\"df-42de81bb-78db-4617-beb1-11cdc0d83888\">\n",
              "  <button class=\"colab-df-quickchart\" onclick=\"quickchart('df-42de81bb-78db-4617-beb1-11cdc0d83888')\"\n",
              "            title=\"Suggest charts\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "<svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "     width=\"24px\">\n",
              "    <g>\n",
              "        <path d=\"M19 3H5c-1.1 0-2 .9-2 2v14c0 1.1.9 2 2 2h14c1.1 0 2-.9 2-2V5c0-1.1-.9-2-2-2zM9 17H7v-7h2v7zm4 0h-2V7h2v10zm4 0h-2v-4h2v4z\"/>\n",
              "    </g>\n",
              "</svg>\n",
              "  </button>\n",
              "\n",
              "<style>\n",
              "  .colab-df-quickchart {\n",
              "      --bg-color: #E8F0FE;\n",
              "      --fill-color: #1967D2;\n",
              "      --hover-bg-color: #E2EBFA;\n",
              "      --hover-fill-color: #174EA6;\n",
              "      --disabled-fill-color: #AAA;\n",
              "      --disabled-bg-color: #DDD;\n",
              "  }\n",
              "\n",
              "  [theme=dark] .colab-df-quickchart {\n",
              "      --bg-color: #3B4455;\n",
              "      --fill-color: #D2E3FC;\n",
              "      --hover-bg-color: #434B5C;\n",
              "      --hover-fill-color: #FFFFFF;\n",
              "      --disabled-bg-color: #3B4455;\n",
              "      --disabled-fill-color: #666;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart {\n",
              "    background-color: var(--bg-color);\n",
              "    border: none;\n",
              "    border-radius: 50%;\n",
              "    cursor: pointer;\n",
              "    display: none;\n",
              "    fill: var(--fill-color);\n",
              "    height: 32px;\n",
              "    padding: 0;\n",
              "    width: 32px;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart:hover {\n",
              "    background-color: var(--hover-bg-color);\n",
              "    box-shadow: 0 1px 2px rgba(60, 64, 67, 0.3), 0 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "    fill: var(--button-hover-fill-color);\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart-complete:disabled,\n",
              "  .colab-df-quickchart-complete:disabled:hover {\n",
              "    background-color: var(--disabled-bg-color);\n",
              "    fill: var(--disabled-fill-color);\n",
              "    box-shadow: none;\n",
              "  }\n",
              "\n",
              "  .colab-df-spinner {\n",
              "    border: 2px solid var(--fill-color);\n",
              "    border-color: transparent;\n",
              "    border-bottom-color: var(--fill-color);\n",
              "    animation:\n",
              "      spin 1s steps(1) infinite;\n",
              "  }\n",
              "\n",
              "  @keyframes spin {\n",
              "    0% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "      border-left-color: var(--fill-color);\n",
              "    }\n",
              "    20% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    30% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    40% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    60% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    80% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "    90% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "  }\n",
              "</style>\n",
              "\n",
              "  <script>\n",
              "    async function quickchart(key) {\n",
              "      const quickchartButtonEl =\n",
              "        document.querySelector('#' + key + ' button');\n",
              "      quickchartButtonEl.disabled = true;  // To prevent multiple clicks.\n",
              "      quickchartButtonEl.classList.add('colab-df-spinner');\n",
              "      try {\n",
              "        const charts = await google.colab.kernel.invokeFunction(\n",
              "            'suggestCharts', [key], {});\n",
              "      } catch (error) {\n",
              "        console.error('Error during call to suggestCharts:', error);\n",
              "      }\n",
              "      quickchartButtonEl.classList.remove('colab-df-spinner');\n",
              "      quickchartButtonEl.classList.add('colab-df-quickchart-complete');\n",
              "    }\n",
              "    (() => {\n",
              "      let quickchartButtonEl =\n",
              "        document.querySelector('#df-42de81bb-78db-4617-beb1-11cdc0d83888 button');\n",
              "      quickchartButtonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "    })();\n",
              "  </script>\n",
              "</div>\n",
              "\n",
              "    </div>\n",
              "  </div>\n"
            ],
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "dataframe",
              "variable_name": "df",
              "summary": "{\n  \"name\": \"df\",\n  \"rows\": 2748,\n  \"fields\": [\n    {\n      \"column\": \"sentence\",\n      \"properties\": {\n        \"dtype\": \"string\",\n        \"num_unique_values\": 2728,\n        \"samples\": [\n          \"I have to jiggle the plug to get it to line up right to get decent volume.\",\n          \"Unfortunately, any virtue in this film's production work was lost on a regrettable script.  \",\n          \"A great film by a great director.  \"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"label\",\n      \"properties\": {\n        \"dtype\": \"number\",\n        \"std\": 0,\n        \"min\": 0,\n        \"max\": 1,\n        \"num_unique_values\": 2,\n        \"samples\": [\n          0,\n          1\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"source\",\n      \"properties\": {\n        \"dtype\": \"category\",\n        \"num_unique_values\": 3,\n        \"samples\": [\n          \"yelp\",\n          \"amazon\"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"word_count_function\",\n      \"properties\": {\n        \"dtype\": \"number\",\n        \"std\": 35,\n        \"min\": 1,\n        \"max\": 1393,\n        \"num_unique_values\": 57,\n        \"samples\": [\n          4,\n          6\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"word_count\",\n      \"properties\": {\n        \"dtype\": \"number\",\n        \"std\": 35,\n        \"min\": 1,\n        \"max\": 1393,\n        \"num_unique_values\": 57,\n        \"samples\": [\n          4,\n          6\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    }\n  ]\n}"
            }
          },
          "metadata": {},
          "execution_count": 11
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "def func(x):\n",
        "  return len( str(x).split(\" \") )"
      ],
      "metadata": {
        "id": "uKAgrGaI6Ymk"
      },
      "execution_count": 12,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "5RYDf5StaS5s"
      },
      "source": [
        "Similarly, count the number of characters of each sentence"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "PrT9GLxt5RgZ",
        "outputId": "f21be232-46c3-4b07-ee28-483149877194",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 206
        }
      },
      "source": [
        "df['char_count'] = df['sentence'].str.len()  # Includes the spaces\n",
        "df.head(5)"
      ],
      "execution_count": 13,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "                                            sentence  label source  \\\n",
              "0                           Wow... Loved this place.      1   yelp   \n",
              "1                                 Crust is not good.      0   yelp   \n",
              "2          Not tasty and the texture was just nasty.      0   yelp   \n",
              "3  Stopped by during the late May bank holiday of...      1   yelp   \n",
              "4  The selection on the menu was great and so wer...      1   yelp   \n",
              "\n",
              "   word_count_function  word_count  char_count  \n",
              "0                    4           4          24  \n",
              "1                    4           4          18  \n",
              "2                    8           8          41  \n",
              "3                   15          15          87  \n",
              "4                   12          12          59  "
            ],
            "text/html": [
              "\n",
              "  <div id=\"df-b15c1de8-a6b1-441d-95b4-526a09ff501c\" class=\"colab-df-container\">\n",
              "    <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>sentence</th>\n",
              "      <th>label</th>\n",
              "      <th>source</th>\n",
              "      <th>word_count_function</th>\n",
              "      <th>word_count</th>\n",
              "      <th>char_count</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>Wow... Loved this place.</td>\n",
              "      <td>1</td>\n",
              "      <td>yelp</td>\n",
              "      <td>4</td>\n",
              "      <td>4</td>\n",
              "      <td>24</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>Crust is not good.</td>\n",
              "      <td>0</td>\n",
              "      <td>yelp</td>\n",
              "      <td>4</td>\n",
              "      <td>4</td>\n",
              "      <td>18</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>Not tasty and the texture was just nasty.</td>\n",
              "      <td>0</td>\n",
              "      <td>yelp</td>\n",
              "      <td>8</td>\n",
              "      <td>8</td>\n",
              "      <td>41</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>Stopped by during the late May bank holiday of...</td>\n",
              "      <td>1</td>\n",
              "      <td>yelp</td>\n",
              "      <td>15</td>\n",
              "      <td>15</td>\n",
              "      <td>87</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>The selection on the menu was great and so wer...</td>\n",
              "      <td>1</td>\n",
              "      <td>yelp</td>\n",
              "      <td>12</td>\n",
              "      <td>12</td>\n",
              "      <td>59</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>\n",
              "    <div class=\"colab-df-buttons\">\n",
              "\n",
              "  <div class=\"colab-df-container\">\n",
              "    <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-b15c1de8-a6b1-441d-95b4-526a09ff501c')\"\n",
              "            title=\"Convert this dataframe to an interactive table.\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\" viewBox=\"0 -960 960 960\">\n",
              "    <path d=\"M120-120v-720h720v720H120Zm60-500h600v-160H180v160Zm220 220h160v-160H400v160Zm0 220h160v-160H400v160ZM180-400h160v-160H180v160Zm440 0h160v-160H620v160ZM180-180h160v-160H180v160Zm440 0h160v-160H620v160Z\"/>\n",
              "  </svg>\n",
              "    </button>\n",
              "\n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    .colab-df-buttons div {\n",
              "      margin-bottom: 4px;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "    <script>\n",
              "      const buttonEl =\n",
              "        document.querySelector('#df-b15c1de8-a6b1-441d-95b4-526a09ff501c button.colab-df-convert');\n",
              "      buttonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "      async function convertToInteractive(key) {\n",
              "        const element = document.querySelector('#df-b15c1de8-a6b1-441d-95b4-526a09ff501c');\n",
              "        const dataTable =\n",
              "          await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                    [key], {});\n",
              "        if (!dataTable) return;\n",
              "\n",
              "        const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "          '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "          + ' to learn more about interactive tables.';\n",
              "        element.innerHTML = '';\n",
              "        dataTable['output_type'] = 'display_data';\n",
              "        await google.colab.output.renderOutput(dataTable, element);\n",
              "        const docLink = document.createElement('div');\n",
              "        docLink.innerHTML = docLinkHtml;\n",
              "        element.appendChild(docLink);\n",
              "      }\n",
              "    </script>\n",
              "  </div>\n",
              "\n",
              "\n",
              "<div id=\"df-cb1f2e0d-738e-4422-b15d-97c564e91a65\">\n",
              "  <button class=\"colab-df-quickchart\" onclick=\"quickchart('df-cb1f2e0d-738e-4422-b15d-97c564e91a65')\"\n",
              "            title=\"Suggest charts\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "<svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "     width=\"24px\">\n",
              "    <g>\n",
              "        <path d=\"M19 3H5c-1.1 0-2 .9-2 2v14c0 1.1.9 2 2 2h14c1.1 0 2-.9 2-2V5c0-1.1-.9-2-2-2zM9 17H7v-7h2v7zm4 0h-2V7h2v10zm4 0h-2v-4h2v4z\"/>\n",
              "    </g>\n",
              "</svg>\n",
              "  </button>\n",
              "\n",
              "<style>\n",
              "  .colab-df-quickchart {\n",
              "      --bg-color: #E8F0FE;\n",
              "      --fill-color: #1967D2;\n",
              "      --hover-bg-color: #E2EBFA;\n",
              "      --hover-fill-color: #174EA6;\n",
              "      --disabled-fill-color: #AAA;\n",
              "      --disabled-bg-color: #DDD;\n",
              "  }\n",
              "\n",
              "  [theme=dark] .colab-df-quickchart {\n",
              "      --bg-color: #3B4455;\n",
              "      --fill-color: #D2E3FC;\n",
              "      --hover-bg-color: #434B5C;\n",
              "      --hover-fill-color: #FFFFFF;\n",
              "      --disabled-bg-color: #3B4455;\n",
              "      --disabled-fill-color: #666;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart {\n",
              "    background-color: var(--bg-color);\n",
              "    border: none;\n",
              "    border-radius: 50%;\n",
              "    cursor: pointer;\n",
              "    display: none;\n",
              "    fill: var(--fill-color);\n",
              "    height: 32px;\n",
              "    padding: 0;\n",
              "    width: 32px;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart:hover {\n",
              "    background-color: var(--hover-bg-color);\n",
              "    box-shadow: 0 1px 2px rgba(60, 64, 67, 0.3), 0 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "    fill: var(--button-hover-fill-color);\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart-complete:disabled,\n",
              "  .colab-df-quickchart-complete:disabled:hover {\n",
              "    background-color: var(--disabled-bg-color);\n",
              "    fill: var(--disabled-fill-color);\n",
              "    box-shadow: none;\n",
              "  }\n",
              "\n",
              "  .colab-df-spinner {\n",
              "    border: 2px solid var(--fill-color);\n",
              "    border-color: transparent;\n",
              "    border-bottom-color: var(--fill-color);\n",
              "    animation:\n",
              "      spin 1s steps(1) infinite;\n",
              "  }\n",
              "\n",
              "  @keyframes spin {\n",
              "    0% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "      border-left-color: var(--fill-color);\n",
              "    }\n",
              "    20% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    30% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    40% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    60% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    80% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "    90% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "  }\n",
              "</style>\n",
              "\n",
              "  <script>\n",
              "    async function quickchart(key) {\n",
              "      const quickchartButtonEl =\n",
              "        document.querySelector('#' + key + ' button');\n",
              "      quickchartButtonEl.disabled = true;  // To prevent multiple clicks.\n",
              "      quickchartButtonEl.classList.add('colab-df-spinner');\n",
              "      try {\n",
              "        const charts = await google.colab.kernel.invokeFunction(\n",
              "            'suggestCharts', [key], {});\n",
              "      } catch (error) {\n",
              "        console.error('Error during call to suggestCharts:', error);\n",
              "      }\n",
              "      quickchartButtonEl.classList.remove('colab-df-spinner');\n",
              "      quickchartButtonEl.classList.add('colab-df-quickchart-complete');\n",
              "    }\n",
              "    (() => {\n",
              "      let quickchartButtonEl =\n",
              "        document.querySelector('#df-cb1f2e0d-738e-4422-b15d-97c564e91a65 button');\n",
              "      quickchartButtonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "    })();\n",
              "  </script>\n",
              "</div>\n",
              "\n",
              "    </div>\n",
              "  </div>\n"
            ],
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "dataframe",
              "variable_name": "df",
              "summary": "{\n  \"name\": \"df\",\n  \"rows\": 2748,\n  \"fields\": [\n    {\n      \"column\": \"sentence\",\n      \"properties\": {\n        \"dtype\": \"string\",\n        \"num_unique_values\": 2728,\n        \"samples\": [\n          \"I have to jiggle the plug to get it to line up right to get decent volume.\",\n          \"Unfortunately, any virtue in this film's production work was lost on a regrettable script.  \",\n          \"A great film by a great director.  \"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"label\",\n      \"properties\": {\n        \"dtype\": \"number\",\n        \"std\": 0,\n        \"min\": 0,\n        \"max\": 1,\n        \"num_unique_values\": 2,\n        \"samples\": [\n          0,\n          1\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"source\",\n      \"properties\": {\n        \"dtype\": \"category\",\n        \"num_unique_values\": 3,\n        \"samples\": [\n          \"yelp\",\n          \"amazon\"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"word_count_function\",\n      \"properties\": {\n        \"dtype\": \"number\",\n        \"std\": 35,\n        \"min\": 1,\n        \"max\": 1393,\n        \"num_unique_values\": 57,\n        \"samples\": [\n          4,\n          6\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"word_count\",\n      \"properties\": {\n        \"dtype\": \"number\",\n        \"std\": 35,\n        \"min\": 1,\n        \"max\": 1393,\n        \"num_unique_values\": 57,\n        \"samples\": [\n          4,\n          6\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"char_count\",\n      \"properties\": {\n        \"dtype\": \"number\",\n        \"std\": 201,\n        \"min\": 6,\n        \"max\": 7944,\n        \"num_unique_values\": 214,\n        \"samples\": [\n          14,\n          237\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    }\n  ]\n}"
            }
          },
          "metadata": {},
          "execution_count": 13
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "LmygJjvfabcd"
      },
      "source": [
        "Calculate the average word length for each sentence.\n",
        "\n",
        "*   First, construct a method (avg_word()) which takes a sentence, split the sentence to words, then calculate the average word length.\n",
        "*   Using pandas dataframe apply() function and avg_word() method, calculate the average word length\n",
        "*  Assign the value to new column names 'avg_word'\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Z-FyuyvnbMS3",
        "outputId": "cb474d21-e93e-4171-b813-1efe50ec4566",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 206
        }
      },
      "source": [
        "def avg_word(sentence):\n",
        "  words = sentence.split() # split the sentence into words\n",
        "  avg_of_words = (sum(len(word) for word in words)/len(words))\n",
        "  return avg_of_words\n",
        "\n",
        "df['avg_word'] = df['sentence'].apply(avg_word)\n",
        "df.head(5)"
      ],
      "execution_count": 14,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "                                            sentence  label source  \\\n",
              "0                           Wow... Loved this place.      1   yelp   \n",
              "1                                 Crust is not good.      0   yelp   \n",
              "2          Not tasty and the texture was just nasty.      0   yelp   \n",
              "3  Stopped by during the late May bank holiday of...      1   yelp   \n",
              "4  The selection on the menu was great and so wer...      1   yelp   \n",
              "\n",
              "   word_count_function  word_count  char_count  avg_word  \n",
              "0                    4           4          24  5.250000  \n",
              "1                    4           4          18  3.750000  \n",
              "2                    8           8          41  4.250000  \n",
              "3                   15          15          87  4.866667  \n",
              "4                   12          12          59  4.000000  "
            ],
            "text/html": [
              "\n",
              "  <div id=\"df-a1510d0e-da2c-4980-a5dd-25137227db9b\" class=\"colab-df-container\">\n",
              "    <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>sentence</th>\n",
              "      <th>label</th>\n",
              "      <th>source</th>\n",
              "      <th>word_count_function</th>\n",
              "      <th>word_count</th>\n",
              "      <th>char_count</th>\n",
              "      <th>avg_word</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>Wow... Loved this place.</td>\n",
              "      <td>1</td>\n",
              "      <td>yelp</td>\n",
              "      <td>4</td>\n",
              "      <td>4</td>\n",
              "      <td>24</td>\n",
              "      <td>5.250000</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>Crust is not good.</td>\n",
              "      <td>0</td>\n",
              "      <td>yelp</td>\n",
              "      <td>4</td>\n",
              "      <td>4</td>\n",
              "      <td>18</td>\n",
              "      <td>3.750000</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>Not tasty and the texture was just nasty.</td>\n",
              "      <td>0</td>\n",
              "      <td>yelp</td>\n",
              "      <td>8</td>\n",
              "      <td>8</td>\n",
              "      <td>41</td>\n",
              "      <td>4.250000</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>Stopped by during the late May bank holiday of...</td>\n",
              "      <td>1</td>\n",
              "      <td>yelp</td>\n",
              "      <td>15</td>\n",
              "      <td>15</td>\n",
              "      <td>87</td>\n",
              "      <td>4.866667</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>The selection on the menu was great and so wer...</td>\n",
              "      <td>1</td>\n",
              "      <td>yelp</td>\n",
              "      <td>12</td>\n",
              "      <td>12</td>\n",
              "      <td>59</td>\n",
              "      <td>4.000000</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>\n",
              "    <div class=\"colab-df-buttons\">\n",
              "\n",
              "  <div class=\"colab-df-container\">\n",
              "    <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-a1510d0e-da2c-4980-a5dd-25137227db9b')\"\n",
              "            title=\"Convert this dataframe to an interactive table.\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\" viewBox=\"0 -960 960 960\">\n",
              "    <path d=\"M120-120v-720h720v720H120Zm60-500h600v-160H180v160Zm220 220h160v-160H400v160Zm0 220h160v-160H400v160ZM180-400h160v-160H180v160Zm440 0h160v-160H620v160ZM180-180h160v-160H180v160Zm440 0h160v-160H620v160Z\"/>\n",
              "  </svg>\n",
              "    </button>\n",
              "\n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    .colab-df-buttons div {\n",
              "      margin-bottom: 4px;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "    <script>\n",
              "      const buttonEl =\n",
              "        document.querySelector('#df-a1510d0e-da2c-4980-a5dd-25137227db9b button.colab-df-convert');\n",
              "      buttonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "      async function convertToInteractive(key) {\n",
              "        const element = document.querySelector('#df-a1510d0e-da2c-4980-a5dd-25137227db9b');\n",
              "        const dataTable =\n",
              "          await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                    [key], {});\n",
              "        if (!dataTable) return;\n",
              "\n",
              "        const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "          '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "          + ' to learn more about interactive tables.';\n",
              "        element.innerHTML = '';\n",
              "        dataTable['output_type'] = 'display_data';\n",
              "        await google.colab.output.renderOutput(dataTable, element);\n",
              "        const docLink = document.createElement('div');\n",
              "        docLink.innerHTML = docLinkHtml;\n",
              "        element.appendChild(docLink);\n",
              "      }\n",
              "    </script>\n",
              "  </div>\n",
              "\n",
              "\n",
              "<div id=\"df-a68699a2-58cb-42fa-9a55-a567c55a73fd\">\n",
              "  <button class=\"colab-df-quickchart\" onclick=\"quickchart('df-a68699a2-58cb-42fa-9a55-a567c55a73fd')\"\n",
              "            title=\"Suggest charts\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "<svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "     width=\"24px\">\n",
              "    <g>\n",
              "        <path d=\"M19 3H5c-1.1 0-2 .9-2 2v14c0 1.1.9 2 2 2h14c1.1 0 2-.9 2-2V5c0-1.1-.9-2-2-2zM9 17H7v-7h2v7zm4 0h-2V7h2v10zm4 0h-2v-4h2v4z\"/>\n",
              "    </g>\n",
              "</svg>\n",
              "  </button>\n",
              "\n",
              "<style>\n",
              "  .colab-df-quickchart {\n",
              "      --bg-color: #E8F0FE;\n",
              "      --fill-color: #1967D2;\n",
              "      --hover-bg-color: #E2EBFA;\n",
              "      --hover-fill-color: #174EA6;\n",
              "      --disabled-fill-color: #AAA;\n",
              "      --disabled-bg-color: #DDD;\n",
              "  }\n",
              "\n",
              "  [theme=dark] .colab-df-quickchart {\n",
              "      --bg-color: #3B4455;\n",
              "      --fill-color: #D2E3FC;\n",
              "      --hover-bg-color: #434B5C;\n",
              "      --hover-fill-color: #FFFFFF;\n",
              "      --disabled-bg-color: #3B4455;\n",
              "      --disabled-fill-color: #666;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart {\n",
              "    background-color: var(--bg-color);\n",
              "    border: none;\n",
              "    border-radius: 50%;\n",
              "    cursor: pointer;\n",
              "    display: none;\n",
              "    fill: var(--fill-color);\n",
              "    height: 32px;\n",
              "    padding: 0;\n",
              "    width: 32px;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart:hover {\n",
              "    background-color: var(--hover-bg-color);\n",
              "    box-shadow: 0 1px 2px rgba(60, 64, 67, 0.3), 0 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "    fill: var(--button-hover-fill-color);\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart-complete:disabled,\n",
              "  .colab-df-quickchart-complete:disabled:hover {\n",
              "    background-color: var(--disabled-bg-color);\n",
              "    fill: var(--disabled-fill-color);\n",
              "    box-shadow: none;\n",
              "  }\n",
              "\n",
              "  .colab-df-spinner {\n",
              "    border: 2px solid var(--fill-color);\n",
              "    border-color: transparent;\n",
              "    border-bottom-color: var(--fill-color);\n",
              "    animation:\n",
              "      spin 1s steps(1) infinite;\n",
              "  }\n",
              "\n",
              "  @keyframes spin {\n",
              "    0% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "      border-left-color: var(--fill-color);\n",
              "    }\n",
              "    20% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    30% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    40% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    60% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    80% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "    90% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "  }\n",
              "</style>\n",
              "\n",
              "  <script>\n",
              "    async function quickchart(key) {\n",
              "      const quickchartButtonEl =\n",
              "        document.querySelector('#' + key + ' button');\n",
              "      quickchartButtonEl.disabled = true;  // To prevent multiple clicks.\n",
              "      quickchartButtonEl.classList.add('colab-df-spinner');\n",
              "      try {\n",
              "        const charts = await google.colab.kernel.invokeFunction(\n",
              "            'suggestCharts', [key], {});\n",
              "      } catch (error) {\n",
              "        console.error('Error during call to suggestCharts:', error);\n",
              "      }\n",
              "      quickchartButtonEl.classList.remove('colab-df-spinner');\n",
              "      quickchartButtonEl.classList.add('colab-df-quickchart-complete');\n",
              "    }\n",
              "    (() => {\n",
              "      let quickchartButtonEl =\n",
              "        document.querySelector('#df-a68699a2-58cb-42fa-9a55-a567c55a73fd button');\n",
              "      quickchartButtonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "    })();\n",
              "  </script>\n",
              "</div>\n",
              "\n",
              "    </div>\n",
              "  </div>\n"
            ],
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "dataframe",
              "variable_name": "df",
              "summary": "{\n  \"name\": \"df\",\n  \"rows\": 2748,\n  \"fields\": [\n    {\n      \"column\": \"sentence\",\n      \"properties\": {\n        \"dtype\": \"string\",\n        \"num_unique_values\": 2728,\n        \"samples\": [\n          \"I have to jiggle the plug to get it to line up right to get decent volume.\",\n          \"Unfortunately, any virtue in this film's production work was lost on a regrettable script.  \",\n          \"A great film by a great director.  \"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"label\",\n      \"properties\": {\n        \"dtype\": \"number\",\n        \"std\": 0,\n        \"min\": 0,\n        \"max\": 1,\n        \"num_unique_values\": 2,\n        \"samples\": [\n          0,\n          1\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"source\",\n      \"properties\": {\n        \"dtype\": \"category\",\n        \"num_unique_values\": 3,\n        \"samples\": [\n          \"yelp\",\n          \"amazon\"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"word_count_function\",\n      \"properties\": {\n        \"dtype\": \"number\",\n        \"std\": 35,\n        \"min\": 1,\n        \"max\": 1393,\n        \"num_unique_values\": 57,\n        \"samples\": [\n          4,\n          6\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"word_count\",\n      \"properties\": {\n        \"dtype\": \"number\",\n        \"std\": 35,\n        \"min\": 1,\n        \"max\": 1393,\n        \"num_unique_values\": 57,\n        \"samples\": [\n          4,\n          6\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"char_count\",\n      \"properties\": {\n        \"dtype\": \"number\",\n        \"std\": 201,\n        \"min\": 6,\n        \"max\": 7944,\n        \"num_unique_values\": 214,\n        \"samples\": [\n          14,\n          237\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"avg_word\",\n      \"properties\": {\n        \"dtype\": \"number\",\n        \"std\": 1.0909993611067916,\n        \"min\": 2.0,\n        \"max\": 14.0,\n        \"num_unique_values\": 518,\n        \"samples\": [\n          3.8461538461538463,\n          4.294117647058823\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    }\n  ]\n}"
            }
          },
          "metadata": {},
          "execution_count": 14
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "c7NajYVd8t-r"
      },
      "source": [
        "## Text pre-processing\n",
        "\n",
        "Pre-processing is mandatory for most text analytics tasks, as text in its raw format is unstructured and noisy.\n",
        "\n",
        "In the following snippets you will run several pre-processing steps.  \n",
        "\n",
        "**Please note that pre-processing is to be used with clear understanding of the expected outcome of text analytics, as each pre-processing step is not relevant or applicable to every NLP task.**"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "mZ56xCM0dxLy"
      },
      "source": [
        "Uppercase and lowercase characters are used for clarity in human communication. However, for a machine such distinction would create unnecessary complexities. Therefore, we transform all characters to lowercase."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "gowXmkYk50h_",
        "outputId": "5fe5ea1e-dfa5-4b4f-db69-737a381f8fd1",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 206
        }
      },
      "source": [
        "df['sentence'] = df['sentence'].str.lower()\n",
        "df.head()"
      ],
      "execution_count": 15,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "                                            sentence  label source  \\\n",
              "0                           wow... loved this place.      1   yelp   \n",
              "1                                 crust is not good.      0   yelp   \n",
              "2          not tasty and the texture was just nasty.      0   yelp   \n",
              "3  stopped by during the late may bank holiday of...      1   yelp   \n",
              "4  the selection on the menu was great and so wer...      1   yelp   \n",
              "\n",
              "   word_count_function  word_count  char_count  avg_word  \n",
              "0                    4           4          24  5.250000  \n",
              "1                    4           4          18  3.750000  \n",
              "2                    8           8          41  4.250000  \n",
              "3                   15          15          87  4.866667  \n",
              "4                   12          12          59  4.000000  "
            ],
            "text/html": [
              "\n",
              "  <div id=\"df-5cde01ae-b8e0-4a39-9e22-0c4c2825dba7\" class=\"colab-df-container\">\n",
              "    <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>sentence</th>\n",
              "      <th>label</th>\n",
              "      <th>source</th>\n",
              "      <th>word_count_function</th>\n",
              "      <th>word_count</th>\n",
              "      <th>char_count</th>\n",
              "      <th>avg_word</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>wow... loved this place.</td>\n",
              "      <td>1</td>\n",
              "      <td>yelp</td>\n",
              "      <td>4</td>\n",
              "      <td>4</td>\n",
              "      <td>24</td>\n",
              "      <td>5.250000</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>crust is not good.</td>\n",
              "      <td>0</td>\n",
              "      <td>yelp</td>\n",
              "      <td>4</td>\n",
              "      <td>4</td>\n",
              "      <td>18</td>\n",
              "      <td>3.750000</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>not tasty and the texture was just nasty.</td>\n",
              "      <td>0</td>\n",
              "      <td>yelp</td>\n",
              "      <td>8</td>\n",
              "      <td>8</td>\n",
              "      <td>41</td>\n",
              "      <td>4.250000</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>stopped by during the late may bank holiday of...</td>\n",
              "      <td>1</td>\n",
              "      <td>yelp</td>\n",
              "      <td>15</td>\n",
              "      <td>15</td>\n",
              "      <td>87</td>\n",
              "      <td>4.866667</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>the selection on the menu was great and so wer...</td>\n",
              "      <td>1</td>\n",
              "      <td>yelp</td>\n",
              "      <td>12</td>\n",
              "      <td>12</td>\n",
              "      <td>59</td>\n",
              "      <td>4.000000</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>\n",
              "    <div class=\"colab-df-buttons\">\n",
              "\n",
              "  <div class=\"colab-df-container\">\n",
              "    <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-5cde01ae-b8e0-4a39-9e22-0c4c2825dba7')\"\n",
              "            title=\"Convert this dataframe to an interactive table.\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\" viewBox=\"0 -960 960 960\">\n",
              "    <path d=\"M120-120v-720h720v720H120Zm60-500h600v-160H180v160Zm220 220h160v-160H400v160Zm0 220h160v-160H400v160ZM180-400h160v-160H180v160Zm440 0h160v-160H620v160ZM180-180h160v-160H180v160Zm440 0h160v-160H620v160Z\"/>\n",
              "  </svg>\n",
              "    </button>\n",
              "\n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    .colab-df-buttons div {\n",
              "      margin-bottom: 4px;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "    <script>\n",
              "      const buttonEl =\n",
              "        document.querySelector('#df-5cde01ae-b8e0-4a39-9e22-0c4c2825dba7 button.colab-df-convert');\n",
              "      buttonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "      async function convertToInteractive(key) {\n",
              "        const element = document.querySelector('#df-5cde01ae-b8e0-4a39-9e22-0c4c2825dba7');\n",
              "        const dataTable =\n",
              "          await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                    [key], {});\n",
              "        if (!dataTable) return;\n",
              "\n",
              "        const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "          '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "          + ' to learn more about interactive tables.';\n",
              "        element.innerHTML = '';\n",
              "        dataTable['output_type'] = 'display_data';\n",
              "        await google.colab.output.renderOutput(dataTable, element);\n",
              "        const docLink = document.createElement('div');\n",
              "        docLink.innerHTML = docLinkHtml;\n",
              "        element.appendChild(docLink);\n",
              "      }\n",
              "    </script>\n",
              "  </div>\n",
              "\n",
              "\n",
              "<div id=\"df-5cf0fe66-5722-44d9-810b-c2272ab76f99\">\n",
              "  <button class=\"colab-df-quickchart\" onclick=\"quickchart('df-5cf0fe66-5722-44d9-810b-c2272ab76f99')\"\n",
              "            title=\"Suggest charts\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "<svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "     width=\"24px\">\n",
              "    <g>\n",
              "        <path d=\"M19 3H5c-1.1 0-2 .9-2 2v14c0 1.1.9 2 2 2h14c1.1 0 2-.9 2-2V5c0-1.1-.9-2-2-2zM9 17H7v-7h2v7zm4 0h-2V7h2v10zm4 0h-2v-4h2v4z\"/>\n",
              "    </g>\n",
              "</svg>\n",
              "  </button>\n",
              "\n",
              "<style>\n",
              "  .colab-df-quickchart {\n",
              "      --bg-color: #E8F0FE;\n",
              "      --fill-color: #1967D2;\n",
              "      --hover-bg-color: #E2EBFA;\n",
              "      --hover-fill-color: #174EA6;\n",
              "      --disabled-fill-color: #AAA;\n",
              "      --disabled-bg-color: #DDD;\n",
              "  }\n",
              "\n",
              "  [theme=dark] .colab-df-quickchart {\n",
              "      --bg-color: #3B4455;\n",
              "      --fill-color: #D2E3FC;\n",
              "      --hover-bg-color: #434B5C;\n",
              "      --hover-fill-color: #FFFFFF;\n",
              "      --disabled-bg-color: #3B4455;\n",
              "      --disabled-fill-color: #666;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart {\n",
              "    background-color: var(--bg-color);\n",
              "    border: none;\n",
              "    border-radius: 50%;\n",
              "    cursor: pointer;\n",
              "    display: none;\n",
              "    fill: var(--fill-color);\n",
              "    height: 32px;\n",
              "    padding: 0;\n",
              "    width: 32px;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart:hover {\n",
              "    background-color: var(--hover-bg-color);\n",
              "    box-shadow: 0 1px 2px rgba(60, 64, 67, 0.3), 0 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "    fill: var(--button-hover-fill-color);\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart-complete:disabled,\n",
              "  .colab-df-quickchart-complete:disabled:hover {\n",
              "    background-color: var(--disabled-bg-color);\n",
              "    fill: var(--disabled-fill-color);\n",
              "    box-shadow: none;\n",
              "  }\n",
              "\n",
              "  .colab-df-spinner {\n",
              "    border: 2px solid var(--fill-color);\n",
              "    border-color: transparent;\n",
              "    border-bottom-color: var(--fill-color);\n",
              "    animation:\n",
              "      spin 1s steps(1) infinite;\n",
              "  }\n",
              "\n",
              "  @keyframes spin {\n",
              "    0% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "      border-left-color: var(--fill-color);\n",
              "    }\n",
              "    20% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    30% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    40% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    60% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    80% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "    90% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "  }\n",
              "</style>\n",
              "\n",
              "  <script>\n",
              "    async function quickchart(key) {\n",
              "      const quickchartButtonEl =\n",
              "        document.querySelector('#' + key + ' button');\n",
              "      quickchartButtonEl.disabled = true;  // To prevent multiple clicks.\n",
              "      quickchartButtonEl.classList.add('colab-df-spinner');\n",
              "      try {\n",
              "        const charts = await google.colab.kernel.invokeFunction(\n",
              "            'suggestCharts', [key], {});\n",
              "      } catch (error) {\n",
              "        console.error('Error during call to suggestCharts:', error);\n",
              "      }\n",
              "      quickchartButtonEl.classList.remove('colab-df-spinner');\n",
              "      quickchartButtonEl.classList.add('colab-df-quickchart-complete');\n",
              "    }\n",
              "    (() => {\n",
              "      let quickchartButtonEl =\n",
              "        document.querySelector('#df-5cf0fe66-5722-44d9-810b-c2272ab76f99 button');\n",
              "      quickchartButtonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "    })();\n",
              "  </script>\n",
              "</div>\n",
              "\n",
              "    </div>\n",
              "  </div>\n"
            ],
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "dataframe",
              "variable_name": "df",
              "summary": "{\n  \"name\": \"df\",\n  \"rows\": 2748,\n  \"fields\": [\n    {\n      \"column\": \"sentence\",\n      \"properties\": {\n        \"dtype\": \"string\",\n        \"num_unique_values\": 2725,\n        \"samples\": [\n          \"i believe the screenwriter did a good job of tying up the loose ends.  \",\n          \"their network coverage in los angeles is horrible.\",\n          \"it's really easy.\"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"label\",\n      \"properties\": {\n        \"dtype\": \"number\",\n        \"std\": 0,\n        \"min\": 0,\n        \"max\": 1,\n        \"num_unique_values\": 2,\n        \"samples\": [\n          0,\n          1\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"source\",\n      \"properties\": {\n        \"dtype\": \"category\",\n        \"num_unique_values\": 3,\n        \"samples\": [\n          \"yelp\",\n          \"amazon\"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"word_count_function\",\n      \"properties\": {\n        \"dtype\": \"number\",\n        \"std\": 35,\n        \"min\": 1,\n        \"max\": 1393,\n        \"num_unique_values\": 57,\n        \"samples\": [\n          4,\n          6\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"word_count\",\n      \"properties\": {\n        \"dtype\": \"number\",\n        \"std\": 35,\n        \"min\": 1,\n        \"max\": 1393,\n        \"num_unique_values\": 57,\n        \"samples\": [\n          4,\n          6\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"char_count\",\n      \"properties\": {\n        \"dtype\": \"number\",\n        \"std\": 201,\n        \"min\": 6,\n        \"max\": 7944,\n        \"num_unique_values\": 214,\n        \"samples\": [\n          14,\n          237\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"avg_word\",\n      \"properties\": {\n        \"dtype\": \"number\",\n        \"std\": 1.0909993611067916,\n        \"min\": 2.0,\n        \"max\": 14.0,\n        \"num_unique_values\": 518,\n        \"samples\": [\n          3.8461538461538463,\n          4.294117647058823\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    }\n  ]\n}"
            }
          },
          "metadata": {},
          "execution_count": 15
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "iXvngUDDelRX"
      },
      "source": [
        "Same with punctuation marks, we remove all using [regular expressions](https://www.w3schools.com/python/python_regex.asp) .  \n",
        "**Regular Expressions** - A regular expression is a special sequence of characters that helps you match or find other strings or sets of strings."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "WHsiGbbY9Hqg",
        "outputId": "eb66379f-1a21-42c4-dee6-090edec88106",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 206
        }
      },
      "source": [
        "# This regular expression only keeps words and characters\n",
        "df['sentence'] = df['sentence'].str.replace('[^\\w\\s]','')\n",
        "df.head()"
      ],
      "execution_count": 16,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "                                            sentence  label source  \\\n",
              "0                           wow... loved this place.      1   yelp   \n",
              "1                                 crust is not good.      0   yelp   \n",
              "2          not tasty and the texture was just nasty.      0   yelp   \n",
              "3  stopped by during the late may bank holiday of...      1   yelp   \n",
              "4  the selection on the menu was great and so wer...      1   yelp   \n",
              "\n",
              "   word_count_function  word_count  char_count  avg_word  \n",
              "0                    4           4          24  5.250000  \n",
              "1                    4           4          18  3.750000  \n",
              "2                    8           8          41  4.250000  \n",
              "3                   15          15          87  4.866667  \n",
              "4                   12          12          59  4.000000  "
            ],
            "text/html": [
              "\n",
              "  <div id=\"df-b37e7afa-7076-4847-9db1-19b4802dd635\" class=\"colab-df-container\">\n",
              "    <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>sentence</th>\n",
              "      <th>label</th>\n",
              "      <th>source</th>\n",
              "      <th>word_count_function</th>\n",
              "      <th>word_count</th>\n",
              "      <th>char_count</th>\n",
              "      <th>avg_word</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>wow... loved this place.</td>\n",
              "      <td>1</td>\n",
              "      <td>yelp</td>\n",
              "      <td>4</td>\n",
              "      <td>4</td>\n",
              "      <td>24</td>\n",
              "      <td>5.250000</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>crust is not good.</td>\n",
              "      <td>0</td>\n",
              "      <td>yelp</td>\n",
              "      <td>4</td>\n",
              "      <td>4</td>\n",
              "      <td>18</td>\n",
              "      <td>3.750000</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>not tasty and the texture was just nasty.</td>\n",
              "      <td>0</td>\n",
              "      <td>yelp</td>\n",
              "      <td>8</td>\n",
              "      <td>8</td>\n",
              "      <td>41</td>\n",
              "      <td>4.250000</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>stopped by during the late may bank holiday of...</td>\n",
              "      <td>1</td>\n",
              "      <td>yelp</td>\n",
              "      <td>15</td>\n",
              "      <td>15</td>\n",
              "      <td>87</td>\n",
              "      <td>4.866667</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>the selection on the menu was great and so wer...</td>\n",
              "      <td>1</td>\n",
              "      <td>yelp</td>\n",
              "      <td>12</td>\n",
              "      <td>12</td>\n",
              "      <td>59</td>\n",
              "      <td>4.000000</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>\n",
              "    <div class=\"colab-df-buttons\">\n",
              "\n",
              "  <div class=\"colab-df-container\">\n",
              "    <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-b37e7afa-7076-4847-9db1-19b4802dd635')\"\n",
              "            title=\"Convert this dataframe to an interactive table.\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\" viewBox=\"0 -960 960 960\">\n",
              "    <path d=\"M120-120v-720h720v720H120Zm60-500h600v-160H180v160Zm220 220h160v-160H400v160Zm0 220h160v-160H400v160ZM180-400h160v-160H180v160Zm440 0h160v-160H620v160ZM180-180h160v-160H180v160Zm440 0h160v-160H620v160Z\"/>\n",
              "  </svg>\n",
              "    </button>\n",
              "\n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    .colab-df-buttons div {\n",
              "      margin-bottom: 4px;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "    <script>\n",
              "      const buttonEl =\n",
              "        document.querySelector('#df-b37e7afa-7076-4847-9db1-19b4802dd635 button.colab-df-convert');\n",
              "      buttonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "      async function convertToInteractive(key) {\n",
              "        const element = document.querySelector('#df-b37e7afa-7076-4847-9db1-19b4802dd635');\n",
              "        const dataTable =\n",
              "          await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                    [key], {});\n",
              "        if (!dataTable) return;\n",
              "\n",
              "        const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "          '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "          + ' to learn more about interactive tables.';\n",
              "        element.innerHTML = '';\n",
              "        dataTable['output_type'] = 'display_data';\n",
              "        await google.colab.output.renderOutput(dataTable, element);\n",
              "        const docLink = document.createElement('div');\n",
              "        docLink.innerHTML = docLinkHtml;\n",
              "        element.appendChild(docLink);\n",
              "      }\n",
              "    </script>\n",
              "  </div>\n",
              "\n",
              "\n",
              "<div id=\"df-9f0c6008-fd2c-41dd-a549-8028a8c5a878\">\n",
              "  <button class=\"colab-df-quickchart\" onclick=\"quickchart('df-9f0c6008-fd2c-41dd-a549-8028a8c5a878')\"\n",
              "            title=\"Suggest charts\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "<svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "     width=\"24px\">\n",
              "    <g>\n",
              "        <path d=\"M19 3H5c-1.1 0-2 .9-2 2v14c0 1.1.9 2 2 2h14c1.1 0 2-.9 2-2V5c0-1.1-.9-2-2-2zM9 17H7v-7h2v7zm4 0h-2V7h2v10zm4 0h-2v-4h2v4z\"/>\n",
              "    </g>\n",
              "</svg>\n",
              "  </button>\n",
              "\n",
              "<style>\n",
              "  .colab-df-quickchart {\n",
              "      --bg-color: #E8F0FE;\n",
              "      --fill-color: #1967D2;\n",
              "      --hover-bg-color: #E2EBFA;\n",
              "      --hover-fill-color: #174EA6;\n",
              "      --disabled-fill-color: #AAA;\n",
              "      --disabled-bg-color: #DDD;\n",
              "  }\n",
              "\n",
              "  [theme=dark] .colab-df-quickchart {\n",
              "      --bg-color: #3B4455;\n",
              "      --fill-color: #D2E3FC;\n",
              "      --hover-bg-color: #434B5C;\n",
              "      --hover-fill-color: #FFFFFF;\n",
              "      --disabled-bg-color: #3B4455;\n",
              "      --disabled-fill-color: #666;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart {\n",
              "    background-color: var(--bg-color);\n",
              "    border: none;\n",
              "    border-radius: 50%;\n",
              "    cursor: pointer;\n",
              "    display: none;\n",
              "    fill: var(--fill-color);\n",
              "    height: 32px;\n",
              "    padding: 0;\n",
              "    width: 32px;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart:hover {\n",
              "    background-color: var(--hover-bg-color);\n",
              "    box-shadow: 0 1px 2px rgba(60, 64, 67, 0.3), 0 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "    fill: var(--button-hover-fill-color);\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart-complete:disabled,\n",
              "  .colab-df-quickchart-complete:disabled:hover {\n",
              "    background-color: var(--disabled-bg-color);\n",
              "    fill: var(--disabled-fill-color);\n",
              "    box-shadow: none;\n",
              "  }\n",
              "\n",
              "  .colab-df-spinner {\n",
              "    border: 2px solid var(--fill-color);\n",
              "    border-color: transparent;\n",
              "    border-bottom-color: var(--fill-color);\n",
              "    animation:\n",
              "      spin 1s steps(1) infinite;\n",
              "  }\n",
              "\n",
              "  @keyframes spin {\n",
              "    0% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "      border-left-color: var(--fill-color);\n",
              "    }\n",
              "    20% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    30% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    40% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    60% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    80% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "    90% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "  }\n",
              "</style>\n",
              "\n",
              "  <script>\n",
              "    async function quickchart(key) {\n",
              "      const quickchartButtonEl =\n",
              "        document.querySelector('#' + key + ' button');\n",
              "      quickchartButtonEl.disabled = true;  // To prevent multiple clicks.\n",
              "      quickchartButtonEl.classList.add('colab-df-spinner');\n",
              "      try {\n",
              "        const charts = await google.colab.kernel.invokeFunction(\n",
              "            'suggestCharts', [key], {});\n",
              "      } catch (error) {\n",
              "        console.error('Error during call to suggestCharts:', error);\n",
              "      }\n",
              "      quickchartButtonEl.classList.remove('colab-df-spinner');\n",
              "      quickchartButtonEl.classList.add('colab-df-quickchart-complete');\n",
              "    }\n",
              "    (() => {\n",
              "      let quickchartButtonEl =\n",
              "        document.querySelector('#df-9f0c6008-fd2c-41dd-a549-8028a8c5a878 button');\n",
              "      quickchartButtonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "    })();\n",
              "  </script>\n",
              "</div>\n",
              "\n",
              "    </div>\n",
              "  </div>\n"
            ],
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "dataframe",
              "variable_name": "df",
              "summary": "{\n  \"name\": \"df\",\n  \"rows\": 2748,\n  \"fields\": [\n    {\n      \"column\": \"sentence\",\n      \"properties\": {\n        \"dtype\": \"string\",\n        \"num_unique_values\": 2725,\n        \"samples\": [\n          \"i believe the screenwriter did a good job of tying up the loose ends.  \",\n          \"their network coverage in los angeles is horrible.\",\n          \"it's really easy.\"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"label\",\n      \"properties\": {\n        \"dtype\": \"number\",\n        \"std\": 0,\n        \"min\": 0,\n        \"max\": 1,\n        \"num_unique_values\": 2,\n        \"samples\": [\n          0,\n          1\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"source\",\n      \"properties\": {\n        \"dtype\": \"category\",\n        \"num_unique_values\": 3,\n        \"samples\": [\n          \"yelp\",\n          \"amazon\"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"word_count_function\",\n      \"properties\": {\n        \"dtype\": \"number\",\n        \"std\": 35,\n        \"min\": 1,\n        \"max\": 1393,\n        \"num_unique_values\": 57,\n        \"samples\": [\n          4,\n          6\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"word_count\",\n      \"properties\": {\n        \"dtype\": \"number\",\n        \"std\": 35,\n        \"min\": 1,\n        \"max\": 1393,\n        \"num_unique_values\": 57,\n        \"samples\": [\n          4,\n          6\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"char_count\",\n      \"properties\": {\n        \"dtype\": \"number\",\n        \"std\": 201,\n        \"min\": 6,\n        \"max\": 7944,\n        \"num_unique_values\": 214,\n        \"samples\": [\n          14,\n          237\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"avg_word\",\n      \"properties\": {\n        \"dtype\": \"number\",\n        \"std\": 1.0909993611067916,\n        \"min\": 2.0,\n        \"max\": 14.0,\n        \"num_unique_values\": 518,\n        \"samples\": [\n          3.8461538461538463,\n          4.294117647058823\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    }\n  ]\n}"
            }
          },
          "metadata": {},
          "execution_count": 16
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "YbhTrjC6lD1D"
      },
      "source": [
        "### Remove digits"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "k-PafqgZe7x7"
      },
      "source": [
        "For a sentiment analytics task, numbers or digits are not needed. Thus, we remove digits from the text dataset.\n",
        "\n",
        "However, for other tasks, numbers may be needed."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "RKRv9btDlLNz",
        "outputId": "e4cc55c0-be96-4a3a-a8c0-43cff2d1657f",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 206
        }
      },
      "source": [
        "def remove_digits(sent):\n",
        "  return \" \".join(w for w in sent.split() if not w.isdigit())\n",
        "\n",
        "df['sentence'] = df['sentence'].apply(remove_digits)\n",
        "df.head()"
      ],
      "execution_count": 17,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "                                            sentence  label source  \\\n",
              "0                           wow... loved this place.      1   yelp   \n",
              "1                                 crust is not good.      0   yelp   \n",
              "2          not tasty and the texture was just nasty.      0   yelp   \n",
              "3  stopped by during the late may bank holiday of...      1   yelp   \n",
              "4  the selection on the menu was great and so wer...      1   yelp   \n",
              "\n",
              "   word_count_function  word_count  char_count  avg_word  \n",
              "0                    4           4          24  5.250000  \n",
              "1                    4           4          18  3.750000  \n",
              "2                    8           8          41  4.250000  \n",
              "3                   15          15          87  4.866667  \n",
              "4                   12          12          59  4.000000  "
            ],
            "text/html": [
              "\n",
              "  <div id=\"df-659c1fcf-bb50-4fad-876f-7b1e09541bfa\" class=\"colab-df-container\">\n",
              "    <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>sentence</th>\n",
              "      <th>label</th>\n",
              "      <th>source</th>\n",
              "      <th>word_count_function</th>\n",
              "      <th>word_count</th>\n",
              "      <th>char_count</th>\n",
              "      <th>avg_word</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>wow... loved this place.</td>\n",
              "      <td>1</td>\n",
              "      <td>yelp</td>\n",
              "      <td>4</td>\n",
              "      <td>4</td>\n",
              "      <td>24</td>\n",
              "      <td>5.250000</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>crust is not good.</td>\n",
              "      <td>0</td>\n",
              "      <td>yelp</td>\n",
              "      <td>4</td>\n",
              "      <td>4</td>\n",
              "      <td>18</td>\n",
              "      <td>3.750000</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>not tasty and the texture was just nasty.</td>\n",
              "      <td>0</td>\n",
              "      <td>yelp</td>\n",
              "      <td>8</td>\n",
              "      <td>8</td>\n",
              "      <td>41</td>\n",
              "      <td>4.250000</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>stopped by during the late may bank holiday of...</td>\n",
              "      <td>1</td>\n",
              "      <td>yelp</td>\n",
              "      <td>15</td>\n",
              "      <td>15</td>\n",
              "      <td>87</td>\n",
              "      <td>4.866667</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>the selection on the menu was great and so wer...</td>\n",
              "      <td>1</td>\n",
              "      <td>yelp</td>\n",
              "      <td>12</td>\n",
              "      <td>12</td>\n",
              "      <td>59</td>\n",
              "      <td>4.000000</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>\n",
              "    <div class=\"colab-df-buttons\">\n",
              "\n",
              "  <div class=\"colab-df-container\">\n",
              "    <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-659c1fcf-bb50-4fad-876f-7b1e09541bfa')\"\n",
              "            title=\"Convert this dataframe to an interactive table.\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\" viewBox=\"0 -960 960 960\">\n",
              "    <path d=\"M120-120v-720h720v720H120Zm60-500h600v-160H180v160Zm220 220h160v-160H400v160Zm0 220h160v-160H400v160ZM180-400h160v-160H180v160Zm440 0h160v-160H620v160ZM180-180h160v-160H180v160Zm440 0h160v-160H620v160Z\"/>\n",
              "  </svg>\n",
              "    </button>\n",
              "\n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    .colab-df-buttons div {\n",
              "      margin-bottom: 4px;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "    <script>\n",
              "      const buttonEl =\n",
              "        document.querySelector('#df-659c1fcf-bb50-4fad-876f-7b1e09541bfa button.colab-df-convert');\n",
              "      buttonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "      async function convertToInteractive(key) {\n",
              "        const element = document.querySelector('#df-659c1fcf-bb50-4fad-876f-7b1e09541bfa');\n",
              "        const dataTable =\n",
              "          await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                    [key], {});\n",
              "        if (!dataTable) return;\n",
              "\n",
              "        const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "          '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "          + ' to learn more about interactive tables.';\n",
              "        element.innerHTML = '';\n",
              "        dataTable['output_type'] = 'display_data';\n",
              "        await google.colab.output.renderOutput(dataTable, element);\n",
              "        const docLink = document.createElement('div');\n",
              "        docLink.innerHTML = docLinkHtml;\n",
              "        element.appendChild(docLink);\n",
              "      }\n",
              "    </script>\n",
              "  </div>\n",
              "\n",
              "\n",
              "<div id=\"df-493d48e2-7eed-4b59-8bf4-fb0dae55f36f\">\n",
              "  <button class=\"colab-df-quickchart\" onclick=\"quickchart('df-493d48e2-7eed-4b59-8bf4-fb0dae55f36f')\"\n",
              "            title=\"Suggest charts\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "<svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "     width=\"24px\">\n",
              "    <g>\n",
              "        <path d=\"M19 3H5c-1.1 0-2 .9-2 2v14c0 1.1.9 2 2 2h14c1.1 0 2-.9 2-2V5c0-1.1-.9-2-2-2zM9 17H7v-7h2v7zm4 0h-2V7h2v10zm4 0h-2v-4h2v4z\"/>\n",
              "    </g>\n",
              "</svg>\n",
              "  </button>\n",
              "\n",
              "<style>\n",
              "  .colab-df-quickchart {\n",
              "      --bg-color: #E8F0FE;\n",
              "      --fill-color: #1967D2;\n",
              "      --hover-bg-color: #E2EBFA;\n",
              "      --hover-fill-color: #174EA6;\n",
              "      --disabled-fill-color: #AAA;\n",
              "      --disabled-bg-color: #DDD;\n",
              "  }\n",
              "\n",
              "  [theme=dark] .colab-df-quickchart {\n",
              "      --bg-color: #3B4455;\n",
              "      --fill-color: #D2E3FC;\n",
              "      --hover-bg-color: #434B5C;\n",
              "      --hover-fill-color: #FFFFFF;\n",
              "      --disabled-bg-color: #3B4455;\n",
              "      --disabled-fill-color: #666;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart {\n",
              "    background-color: var(--bg-color);\n",
              "    border: none;\n",
              "    border-radius: 50%;\n",
              "    cursor: pointer;\n",
              "    display: none;\n",
              "    fill: var(--fill-color);\n",
              "    height: 32px;\n",
              "    padding: 0;\n",
              "    width: 32px;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart:hover {\n",
              "    background-color: var(--hover-bg-color);\n",
              "    box-shadow: 0 1px 2px rgba(60, 64, 67, 0.3), 0 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "    fill: var(--button-hover-fill-color);\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart-complete:disabled,\n",
              "  .colab-df-quickchart-complete:disabled:hover {\n",
              "    background-color: var(--disabled-bg-color);\n",
              "    fill: var(--disabled-fill-color);\n",
              "    box-shadow: none;\n",
              "  }\n",
              "\n",
              "  .colab-df-spinner {\n",
              "    border: 2px solid var(--fill-color);\n",
              "    border-color: transparent;\n",
              "    border-bottom-color: var(--fill-color);\n",
              "    animation:\n",
              "      spin 1s steps(1) infinite;\n",
              "  }\n",
              "\n",
              "  @keyframes spin {\n",
              "    0% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "      border-left-color: var(--fill-color);\n",
              "    }\n",
              "    20% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    30% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    40% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    60% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    80% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "    90% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "  }\n",
              "</style>\n",
              "\n",
              "  <script>\n",
              "    async function quickchart(key) {\n",
              "      const quickchartButtonEl =\n",
              "        document.querySelector('#' + key + ' button');\n",
              "      quickchartButtonEl.disabled = true;  // To prevent multiple clicks.\n",
              "      quickchartButtonEl.classList.add('colab-df-spinner');\n",
              "      try {\n",
              "        const charts = await google.colab.kernel.invokeFunction(\n",
              "            'suggestCharts', [key], {});\n",
              "      } catch (error) {\n",
              "        console.error('Error during call to suggestCharts:', error);\n",
              "      }\n",
              "      quickchartButtonEl.classList.remove('colab-df-spinner');\n",
              "      quickchartButtonEl.classList.add('colab-df-quickchart-complete');\n",
              "    }\n",
              "    (() => {\n",
              "      let quickchartButtonEl =\n",
              "        document.querySelector('#df-493d48e2-7eed-4b59-8bf4-fb0dae55f36f button');\n",
              "      quickchartButtonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "    })();\n",
              "  </script>\n",
              "</div>\n",
              "\n",
              "    </div>\n",
              "  </div>\n"
            ],
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "dataframe",
              "variable_name": "df",
              "summary": "{\n  \"name\": \"df\",\n  \"rows\": 2748,\n  \"fields\": [\n    {\n      \"column\": \"sentence\",\n      \"properties\": {\n        \"dtype\": \"string\",\n        \"num_unique_values\": 2724,\n        \"samples\": [\n          \"i believe the screenwriter did a good job of tying up the loose ends.\",\n          \"what possesed me to get this junk, i have no idea...\",\n          \"disappointment.. i hate anything that goes in my ear.\"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"label\",\n      \"properties\": {\n        \"dtype\": \"number\",\n        \"std\": 0,\n        \"min\": 0,\n        \"max\": 1,\n        \"num_unique_values\": 2,\n        \"samples\": [\n          0,\n          1\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"source\",\n      \"properties\": {\n        \"dtype\": \"category\",\n        \"num_unique_values\": 3,\n        \"samples\": [\n          \"yelp\",\n          \"amazon\"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"word_count_function\",\n      \"properties\": {\n        \"dtype\": \"number\",\n        \"std\": 35,\n        \"min\": 1,\n        \"max\": 1393,\n        \"num_unique_values\": 57,\n        \"samples\": [\n          4,\n          6\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"word_count\",\n      \"properties\": {\n        \"dtype\": \"number\",\n        \"std\": 35,\n        \"min\": 1,\n        \"max\": 1393,\n        \"num_unique_values\": 57,\n        \"samples\": [\n          4,\n          6\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"char_count\",\n      \"properties\": {\n        \"dtype\": \"number\",\n        \"std\": 201,\n        \"min\": 6,\n        \"max\": 7944,\n        \"num_unique_values\": 214,\n        \"samples\": [\n          14,\n          237\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"avg_word\",\n      \"properties\": {\n        \"dtype\": \"number\",\n        \"std\": 1.0909993611067916,\n        \"min\": 2.0,\n        \"max\": 14.0,\n        \"num_unique_values\": 518,\n        \"samples\": [\n          3.8461538461538463,\n          4.294117647058823\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    }\n  ]\n}"
            }
          },
          "metadata": {},
          "execution_count": 17
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "f_xppk798hrz"
      },
      "source": [
        "Demonstrate of the remove digit function"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ntqhwksm8T9M",
        "outputId": "82ee1f3e-d81a-420a-b990-e821b376cae5",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "source": [
        "sample_text = 'Covid 19 is spreading fast'\n",
        "print(remove_digits(sample_text))"
      ],
      "execution_count": 18,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Covid is spreading fast\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "GlxTKunt8lyT"
      },
      "source": [
        "What does \"\".join() means?"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "f2kfwcVk8oyi",
        "outputId": "747937df-bab0-45f0-ca7a-6ffe8e249386",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "source": [
        "word_list = [\"Covid\", \"is\", \"spreading\", \"fast\"]\n",
        "sentence = \"     \".join(word_list)\n",
        "print(sentence)"
      ],
      "execution_count": 19,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Covid     is     spreading     fast\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "uwh1imSm-BC3"
      },
      "source": [
        "### Remove Stopwords\n",
        "\n",
        "[Stopwords](https://en.wikipedia.org/wiki/Stop_words) are deemed irrelevant for NLP purposes because they occur frequently in the language. Therefore, we will omit the stopwords as a pre-processing step. For this, we will use [NLTK](https://www.nltk.org/) library here.\n",
        "\n",
        "**NLTK, is a suite of libraries and programs for symbolic and statistical natural language processing (NLP) specifically for the English language written in  Python.**"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "u1J6fC9491k7",
        "outputId": "3a18a307-fd33-4e54-c057-c10b992ebb28",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "source": [
        "# Load NLTK library\n",
        "import nltk\n",
        "\n",
        "# Download the stopwords to the nltk library\n",
        "nltk.download('stopwords')\n",
        "\n",
        "# Load the stopwords\n",
        "from nltk.corpus import stopwords"
      ],
      "execution_count": 20,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "[nltk_data] Downloading package stopwords to /root/nltk_data...\n",
            "[nltk_data]   Unzipping corpora/stopwords.zip.\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ecLfjQgmgYpk"
      },
      "source": [
        "Have a look at the stopwords indexed in the NLTK library."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "BkpvQNV5-N2g",
        "outputId": "d3556d45-f8a5-4c22-9a9f-d297d0f38262",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "source": [
        "stop = stopwords.words('english')\n",
        "print(stop)"
      ],
      "execution_count": 21,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "['i', 'me', 'my', 'myself', 'we', 'our', 'ours', 'ourselves', 'you', \"you're\", \"you've\", \"you'll\", \"you'd\", 'your', 'yours', 'yourself', 'yourselves', 'he', 'him', 'his', 'himself', 'she', \"she's\", 'her', 'hers', 'herself', 'it', \"it's\", 'its', 'itself', 'they', 'them', 'their', 'theirs', 'themselves', 'what', 'which', 'who', 'whom', 'this', 'that', \"that'll\", 'these', 'those', 'am', 'is', 'are', 'was', 'were', 'be', 'been', 'being', 'have', 'has', 'had', 'having', 'do', 'does', 'did', 'doing', 'a', 'an', 'the', 'and', 'but', 'if', 'or', 'because', 'as', 'until', 'while', 'of', 'at', 'by', 'for', 'with', 'about', 'against', 'between', 'into', 'through', 'during', 'before', 'after', 'above', 'below', 'to', 'from', 'up', 'down', 'in', 'out', 'on', 'off', 'over', 'under', 'again', 'further', 'then', 'once', 'here', 'there', 'when', 'where', 'why', 'how', 'all', 'any', 'both', 'each', 'few', 'more', 'most', 'other', 'some', 'such', 'no', 'nor', 'not', 'only', 'own', 'same', 'so', 'than', 'too', 'very', 's', 't', 'can', 'will', 'just', 'don', \"don't\", 'should', \"should've\", 'now', 'd', 'll', 'm', 'o', 're', 've', 'y', 'ain', 'aren', \"aren't\", 'couldn', \"couldn't\", 'didn', \"didn't\", 'doesn', \"doesn't\", 'hadn', \"hadn't\", 'hasn', \"hasn't\", 'haven', \"haven't\", 'isn', \"isn't\", 'ma', 'mightn', \"mightn't\", 'mustn', \"mustn't\", 'needn', \"needn't\", 'shan', \"shan't\", 'shouldn', \"shouldn't\", 'wasn', \"wasn't\", 'weren', \"weren't\", 'won', \"won't\", 'wouldn', \"wouldn't\"]\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "H73Xr48-cHAS"
      },
      "source": [
        "Remove unwanted stop words from the NLTK stop word list."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "rPVzEQ_8O0UR"
      },
      "source": [
        "stop.remove('not')"
      ],
      "execution_count": 22,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "fO8HTTOXPYyy"
      },
      "source": [
        "all_words_i_want = ['had', \"has\"]\n",
        "for w in all_words_i_want:\n",
        "  stop.remove(w)"
      ],
      "execution_count": 23,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "zPugCcb3gzMj"
      },
      "source": [
        "Remove stopwords from the sentences."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "S_HwAnis-Ut1",
        "outputId": "05f95f14-4b5d-4e1c-bbdd-bfeda87bcced",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 206
        }
      },
      "source": [
        "df['sentence'] = df['sentence'].apply(lambda x: \" \".join(x for x in x.split() if x not in stop))\n",
        "df.head(5)"
      ],
      "execution_count": 24,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "                                            sentence  label source  \\\n",
              "0                                wow... loved place.      1   yelp   \n",
              "1                                    crust not good.      0   yelp   \n",
              "2                           not tasty texture nasty.      0   yelp   \n",
              "3  stopped late may bank holiday rick steve recom...      1   yelp   \n",
              "4                       selection menu great prices.      1   yelp   \n",
              "\n",
              "   word_count_function  word_count  char_count  avg_word  \n",
              "0                    4           4          24  5.250000  \n",
              "1                    4           4          18  3.750000  \n",
              "2                    8           8          41  4.250000  \n",
              "3                   15          15          87  4.866667  \n",
              "4                   12          12          59  4.000000  "
            ],
            "text/html": [
              "\n",
              "  <div id=\"df-52ee9938-bb8c-449e-9897-4cfe011e9675\" class=\"colab-df-container\">\n",
              "    <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>sentence</th>\n",
              "      <th>label</th>\n",
              "      <th>source</th>\n",
              "      <th>word_count_function</th>\n",
              "      <th>word_count</th>\n",
              "      <th>char_count</th>\n",
              "      <th>avg_word</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>wow... loved place.</td>\n",
              "      <td>1</td>\n",
              "      <td>yelp</td>\n",
              "      <td>4</td>\n",
              "      <td>4</td>\n",
              "      <td>24</td>\n",
              "      <td>5.250000</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>crust not good.</td>\n",
              "      <td>0</td>\n",
              "      <td>yelp</td>\n",
              "      <td>4</td>\n",
              "      <td>4</td>\n",
              "      <td>18</td>\n",
              "      <td>3.750000</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>not tasty texture nasty.</td>\n",
              "      <td>0</td>\n",
              "      <td>yelp</td>\n",
              "      <td>8</td>\n",
              "      <td>8</td>\n",
              "      <td>41</td>\n",
              "      <td>4.250000</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>stopped late may bank holiday rick steve recom...</td>\n",
              "      <td>1</td>\n",
              "      <td>yelp</td>\n",
              "      <td>15</td>\n",
              "      <td>15</td>\n",
              "      <td>87</td>\n",
              "      <td>4.866667</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>selection menu great prices.</td>\n",
              "      <td>1</td>\n",
              "      <td>yelp</td>\n",
              "      <td>12</td>\n",
              "      <td>12</td>\n",
              "      <td>59</td>\n",
              "      <td>4.000000</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>\n",
              "    <div class=\"colab-df-buttons\">\n",
              "\n",
              "  <div class=\"colab-df-container\">\n",
              "    <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-52ee9938-bb8c-449e-9897-4cfe011e9675')\"\n",
              "            title=\"Convert this dataframe to an interactive table.\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\" viewBox=\"0 -960 960 960\">\n",
              "    <path d=\"M120-120v-720h720v720H120Zm60-500h600v-160H180v160Zm220 220h160v-160H400v160Zm0 220h160v-160H400v160ZM180-400h160v-160H180v160Zm440 0h160v-160H620v160ZM180-180h160v-160H180v160Zm440 0h160v-160H620v160Z\"/>\n",
              "  </svg>\n",
              "    </button>\n",
              "\n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    .colab-df-buttons div {\n",
              "      margin-bottom: 4px;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "    <script>\n",
              "      const buttonEl =\n",
              "        document.querySelector('#df-52ee9938-bb8c-449e-9897-4cfe011e9675 button.colab-df-convert');\n",
              "      buttonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "      async function convertToInteractive(key) {\n",
              "        const element = document.querySelector('#df-52ee9938-bb8c-449e-9897-4cfe011e9675');\n",
              "        const dataTable =\n",
              "          await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                    [key], {});\n",
              "        if (!dataTable) return;\n",
              "\n",
              "        const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "          '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "          + ' to learn more about interactive tables.';\n",
              "        element.innerHTML = '';\n",
              "        dataTable['output_type'] = 'display_data';\n",
              "        await google.colab.output.renderOutput(dataTable, element);\n",
              "        const docLink = document.createElement('div');\n",
              "        docLink.innerHTML = docLinkHtml;\n",
              "        element.appendChild(docLink);\n",
              "      }\n",
              "    </script>\n",
              "  </div>\n",
              "\n",
              "\n",
              "<div id=\"df-6fadd118-1033-42c2-ae22-c6bebbcf33a8\">\n",
              "  <button class=\"colab-df-quickchart\" onclick=\"quickchart('df-6fadd118-1033-42c2-ae22-c6bebbcf33a8')\"\n",
              "            title=\"Suggest charts\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "<svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "     width=\"24px\">\n",
              "    <g>\n",
              "        <path d=\"M19 3H5c-1.1 0-2 .9-2 2v14c0 1.1.9 2 2 2h14c1.1 0 2-.9 2-2V5c0-1.1-.9-2-2-2zM9 17H7v-7h2v7zm4 0h-2V7h2v10zm4 0h-2v-4h2v4z\"/>\n",
              "    </g>\n",
              "</svg>\n",
              "  </button>\n",
              "\n",
              "<style>\n",
              "  .colab-df-quickchart {\n",
              "      --bg-color: #E8F0FE;\n",
              "      --fill-color: #1967D2;\n",
              "      --hover-bg-color: #E2EBFA;\n",
              "      --hover-fill-color: #174EA6;\n",
              "      --disabled-fill-color: #AAA;\n",
              "      --disabled-bg-color: #DDD;\n",
              "  }\n",
              "\n",
              "  [theme=dark] .colab-df-quickchart {\n",
              "      --bg-color: #3B4455;\n",
              "      --fill-color: #D2E3FC;\n",
              "      --hover-bg-color: #434B5C;\n",
              "      --hover-fill-color: #FFFFFF;\n",
              "      --disabled-bg-color: #3B4455;\n",
              "      --disabled-fill-color: #666;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart {\n",
              "    background-color: var(--bg-color);\n",
              "    border: none;\n",
              "    border-radius: 50%;\n",
              "    cursor: pointer;\n",
              "    display: none;\n",
              "    fill: var(--fill-color);\n",
              "    height: 32px;\n",
              "    padding: 0;\n",
              "    width: 32px;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart:hover {\n",
              "    background-color: var(--hover-bg-color);\n",
              "    box-shadow: 0 1px 2px rgba(60, 64, 67, 0.3), 0 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "    fill: var(--button-hover-fill-color);\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart-complete:disabled,\n",
              "  .colab-df-quickchart-complete:disabled:hover {\n",
              "    background-color: var(--disabled-bg-color);\n",
              "    fill: var(--disabled-fill-color);\n",
              "    box-shadow: none;\n",
              "  }\n",
              "\n",
              "  .colab-df-spinner {\n",
              "    border: 2px solid var(--fill-color);\n",
              "    border-color: transparent;\n",
              "    border-bottom-color: var(--fill-color);\n",
              "    animation:\n",
              "      spin 1s steps(1) infinite;\n",
              "  }\n",
              "\n",
              "  @keyframes spin {\n",
              "    0% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "      border-left-color: var(--fill-color);\n",
              "    }\n",
              "    20% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    30% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    40% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    60% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    80% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "    90% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "  }\n",
              "</style>\n",
              "\n",
              "  <script>\n",
              "    async function quickchart(key) {\n",
              "      const quickchartButtonEl =\n",
              "        document.querySelector('#' + key + ' button');\n",
              "      quickchartButtonEl.disabled = true;  // To prevent multiple clicks.\n",
              "      quickchartButtonEl.classList.add('colab-df-spinner');\n",
              "      try {\n",
              "        const charts = await google.colab.kernel.invokeFunction(\n",
              "            'suggestCharts', [key], {});\n",
              "      } catch (error) {\n",
              "        console.error('Error during call to suggestCharts:', error);\n",
              "      }\n",
              "      quickchartButtonEl.classList.remove('colab-df-spinner');\n",
              "      quickchartButtonEl.classList.add('colab-df-quickchart-complete');\n",
              "    }\n",
              "    (() => {\n",
              "      let quickchartButtonEl =\n",
              "        document.querySelector('#df-6fadd118-1033-42c2-ae22-c6bebbcf33a8 button');\n",
              "      quickchartButtonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "    })();\n",
              "  </script>\n",
              "</div>\n",
              "\n",
              "    </div>\n",
              "  </div>\n"
            ],
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "dataframe",
              "variable_name": "df",
              "summary": "{\n  \"name\": \"df\",\n  \"rows\": 2748,\n  \"fields\": [\n    {\n      \"column\": \"sentence\",\n      \"properties\": {\n        \"dtype\": \"string\",\n        \"num_unique_values\": 2710,\n        \"samples\": [\n          \"one wants surf small wave space movies (deep impact armageddon), one fails everywhere.\",\n          \"lame would best way describe it.\",\n          \"attached gas station, rarely good sign.\"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"label\",\n      \"properties\": {\n        \"dtype\": \"number\",\n        \"std\": 0,\n        \"min\": 0,\n        \"max\": 1,\n        \"num_unique_values\": 2,\n        \"samples\": [\n          0,\n          1\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"source\",\n      \"properties\": {\n        \"dtype\": \"category\",\n        \"num_unique_values\": 3,\n        \"samples\": [\n          \"yelp\",\n          \"amazon\"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"word_count_function\",\n      \"properties\": {\n        \"dtype\": \"number\",\n        \"std\": 35,\n        \"min\": 1,\n        \"max\": 1393,\n        \"num_unique_values\": 57,\n        \"samples\": [\n          4,\n          6\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"word_count\",\n      \"properties\": {\n        \"dtype\": \"number\",\n        \"std\": 35,\n        \"min\": 1,\n        \"max\": 1393,\n        \"num_unique_values\": 57,\n        \"samples\": [\n          4,\n          6\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"char_count\",\n      \"properties\": {\n        \"dtype\": \"number\",\n        \"std\": 201,\n        \"min\": 6,\n        \"max\": 7944,\n        \"num_unique_values\": 214,\n        \"samples\": [\n          14,\n          237\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"avg_word\",\n      \"properties\": {\n        \"dtype\": \"number\",\n        \"std\": 1.0909993611067916,\n        \"min\": 2.0,\n        \"max\": 14.0,\n        \"num_unique_values\": 518,\n        \"samples\": [\n          3.8461538461538463,\n          4.294117647058823\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    }\n  ]\n}"
            }
          },
          "metadata": {},
          "execution_count": 24
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "XUol6pfo_MEL"
      },
      "source": [
        "### Common and rare word analysis\n",
        "\n",
        "Aside from stopwords, some words appear rarely (only once or twice) in an entire body of text.\n",
        "Based on the analytics requirement, you can decide whether to keep or remove, and at what intensity/scale to remove."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "6iyA7bBDg_hB"
      },
      "source": [
        "In order to do this, first we have to construct a word frequency dictionary."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "XHCEJC3V-wG0"
      },
      "source": [
        "word_frequency = pd.Series(' '.join(df['sentence']).split()).value_counts()"
      ],
      "execution_count": 25,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "word_frequency"
      ],
      "metadata": {
        "id": "oi4pUKJp-UHV",
        "outputId": "1fb8fe16-cfe2-4dd9-8c03-c1672c8a67a7",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 458
        }
      },
      "execution_count": 26,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "not         302\n",
              "good        174\n",
              "great       157\n",
              "movie       133\n",
              "had         129\n",
              "           ... \n",
              "broke).-      1\n",
              "case)         1\n",
              "(with         1\n",
              "spinn         1\n",
              "one's         1\n",
              "Name: count, Length: 7168, dtype: int64"
            ],
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>count</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>not</th>\n",
              "      <td>302</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>good</th>\n",
              "      <td>174</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>great</th>\n",
              "      <td>157</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>movie</th>\n",
              "      <td>133</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>had</th>\n",
              "      <td>129</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>...</th>\n",
              "      <td>...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>broke).-</th>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>case)</th>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>(with</th>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>spinn</th>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>one's</th>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "<p>7168 rows × 1 columns</p>\n",
              "</div><br><label><b>dtype:</b> int64</label>"
            ]
          },
          "metadata": {},
          "execution_count": 26
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "3Tsu7TJOhGGK"
      },
      "source": [
        "List the top 10 common words."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "QP6ids9d_ldS",
        "outputId": "686836f9-e929-4f19-ee40-44a06f0267b5",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 397
        }
      },
      "source": [
        "# Top common words\n",
        "word_frequency[:10]  # get top 10"
      ],
      "execution_count": 27,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "not       302\n",
              "good      174\n",
              "great     157\n",
              "movie     133\n",
              "had       129\n",
              "one       124\n",
              "like      122\n",
              "phone     118\n",
              "film      107\n",
              "really     98\n",
              "Name: count, dtype: int64"
            ],
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>count</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>not</th>\n",
              "      <td>302</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>good</th>\n",
              "      <td>174</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>great</th>\n",
              "      <td>157</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>movie</th>\n",
              "      <td>133</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>had</th>\n",
              "      <td>129</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>one</th>\n",
              "      <td>124</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>like</th>\n",
              "      <td>122</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>phone</th>\n",
              "      <td>118</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>film</th>\n",
              "      <td>107</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>really</th>\n",
              "      <td>98</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div><br><label><b>dtype:</b> int64</label>"
            ]
          },
          "metadata": {},
          "execution_count": 27
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "rEIsBYCGhLlY"
      },
      "source": [
        "List the top 10 rare words."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "nvUQt5hRAM13",
        "outputId": "452f8364-1719-4141-e22b-332bdc05cd79",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 397
        }
      },
      "source": [
        "# least common words\n",
        "word_frequency[-10:]  # get top 10"
      ],
      "execution_count": 28,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "psyched     1\n",
              "iriver      1\n",
              "strap.      1\n",
              "magnetic    1\n",
              "fond        1\n",
              "broke).-    1\n",
              "case)       1\n",
              "(with       1\n",
              "spinn       1\n",
              "one's       1\n",
              "Name: count, dtype: int64"
            ],
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>count</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>psyched</th>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>iriver</th>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>strap.</th>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>magnetic</th>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>fond</th>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>broke).-</th>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>case)</th>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>(with</th>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>spinn</th>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>one's</th>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div><br><label><b>dtype:</b> int64</label>"
            ]
          },
          "metadata": {},
          "execution_count": 28
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "l7hd1zK8AXSl"
      },
      "source": [
        "### Spelling correction\n",
        "\n",
        "To correct misspelt words, we will use [textblob library](https://textblob.readthedocs.io/en/dev/) library. Keep in mind that corrections are always bound by the dictionary that you would use, and it may not account for context (their vs there).\n",
        "\n",
        "Due to the time complexity of spell-checking an entire corpus, in this exercise, we will use spell-check for just one example."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "iH_MOO58iity"
      },
      "source": [
        "from textblob import TextBlob"
      ],
      "execution_count": 29,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "jf9rFrJpARWH"
      },
      "source": [
        "# Do not run this line of code.\n",
        "# Following line of code will correct spellings of all the sentences in the dataset.\n",
        "# df['sentence'] = df['sentence'].apply(lambda x: str(TextBlob(x).correct()))   # This will take a long time. Thus, we will show an seperate example"
      ],
      "execution_count": 30,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "A6ERDidUivTx"
      },
      "source": [
        "Spelling correction example"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "J2U3yRK5QqkA",
        "outputId": "9eaee2d1-c6e2-42ce-c032-e0cd5c46891e",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "source": [
        "def correct_word(word):\n",
        "  return str(TextBlob(word).correct())\n",
        "\n",
        "print(correct_word('bisness'))"
      ],
      "execution_count": 31,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "business\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "k6D1dM55A_5l",
        "outputId": "c422e239-bfed-4eb3-db00-d029cc74de54",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "source": [
        "incorrect_text = 'bisness anlytis is an itant skil seit for any organizaton'\n",
        "\n",
        "func = lambda x: str(TextBlob(x).correct())\n",
        "print(incorrect_text)\n",
        "print(str(TextBlob(incorrect_text).correct()))"
      ],
      "execution_count": 32,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "bisness anlytis is an itant skil seit for any organizaton\n",
            "business analysis is an want skin set for any organization\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "5F_w641sCItw"
      },
      "source": [
        "### Stemming\n",
        "\n",
        "[Stemming](https://en.wikipedia.org/wiki/Stemming) is the removal of prefix, suffix etc, to derive the base form of a word. We will use the NLTK library."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "3MxoI8j4Bf93",
        "outputId": "de1b9bf5-9f41-4d64-8385-58ef65fabd18",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 223
        }
      },
      "source": [
        "from nltk.stem import PorterStemmer\n",
        "\n",
        "stemmer = PorterStemmer()\n",
        "\n",
        "def stemming_function(sent):\n",
        "  word_list = sent.split()\n",
        "  stemmed_word_list = [stemmer.stem(word) for word in word_list]\n",
        "  stemmed_sentence = \" \".join(stemmed_word_list)\n",
        "  return stemmed_sentence\n",
        "\n",
        "df['sentence_stemmed'] = df['sentence'].apply(stemming_function)\n",
        "\n",
        "df.head()"
      ],
      "execution_count": 33,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "                                            sentence  label source  \\\n",
              "0                                wow... loved place.      1   yelp   \n",
              "1                                    crust not good.      0   yelp   \n",
              "2                           not tasty texture nasty.      0   yelp   \n",
              "3  stopped late may bank holiday rick steve recom...      1   yelp   \n",
              "4                       selection menu great prices.      1   yelp   \n",
              "\n",
              "   word_count_function  word_count  char_count  avg_word  \\\n",
              "0                    4           4          24  5.250000   \n",
              "1                    4           4          18  3.750000   \n",
              "2                    8           8          41  4.250000   \n",
              "3                   15          15          87  4.866667   \n",
              "4                   12          12          59  4.000000   \n",
              "\n",
              "                                    sentence_stemmed  \n",
              "0                                 wow... love place.  \n",
              "1                                    crust not good.  \n",
              "2                            not tasti textur nasty.  \n",
              "3  stop late may bank holiday rick steve recommen...  \n",
              "4                          select menu great prices.  "
            ],
            "text/html": [
              "\n",
              "  <div id=\"df-8843abe3-fb63-47aa-bc23-2367659188a1\" class=\"colab-df-container\">\n",
              "    <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>sentence</th>\n",
              "      <th>label</th>\n",
              "      <th>source</th>\n",
              "      <th>word_count_function</th>\n",
              "      <th>word_count</th>\n",
              "      <th>char_count</th>\n",
              "      <th>avg_word</th>\n",
              "      <th>sentence_stemmed</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>wow... loved place.</td>\n",
              "      <td>1</td>\n",
              "      <td>yelp</td>\n",
              "      <td>4</td>\n",
              "      <td>4</td>\n",
              "      <td>24</td>\n",
              "      <td>5.250000</td>\n",
              "      <td>wow... love place.</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>crust not good.</td>\n",
              "      <td>0</td>\n",
              "      <td>yelp</td>\n",
              "      <td>4</td>\n",
              "      <td>4</td>\n",
              "      <td>18</td>\n",
              "      <td>3.750000</td>\n",
              "      <td>crust not good.</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>not tasty texture nasty.</td>\n",
              "      <td>0</td>\n",
              "      <td>yelp</td>\n",
              "      <td>8</td>\n",
              "      <td>8</td>\n",
              "      <td>41</td>\n",
              "      <td>4.250000</td>\n",
              "      <td>not tasti textur nasty.</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>stopped late may bank holiday rick steve recom...</td>\n",
              "      <td>1</td>\n",
              "      <td>yelp</td>\n",
              "      <td>15</td>\n",
              "      <td>15</td>\n",
              "      <td>87</td>\n",
              "      <td>4.866667</td>\n",
              "      <td>stop late may bank holiday rick steve recommen...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>selection menu great prices.</td>\n",
              "      <td>1</td>\n",
              "      <td>yelp</td>\n",
              "      <td>12</td>\n",
              "      <td>12</td>\n",
              "      <td>59</td>\n",
              "      <td>4.000000</td>\n",
              "      <td>select menu great prices.</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>\n",
              "    <div class=\"colab-df-buttons\">\n",
              "\n",
              "  <div class=\"colab-df-container\">\n",
              "    <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-8843abe3-fb63-47aa-bc23-2367659188a1')\"\n",
              "            title=\"Convert this dataframe to an interactive table.\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\" viewBox=\"0 -960 960 960\">\n",
              "    <path d=\"M120-120v-720h720v720H120Zm60-500h600v-160H180v160Zm220 220h160v-160H400v160Zm0 220h160v-160H400v160ZM180-400h160v-160H180v160Zm440 0h160v-160H620v160ZM180-180h160v-160H180v160Zm440 0h160v-160H620v160Z\"/>\n",
              "  </svg>\n",
              "    </button>\n",
              "\n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    .colab-df-buttons div {\n",
              "      margin-bottom: 4px;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "    <script>\n",
              "      const buttonEl =\n",
              "        document.querySelector('#df-8843abe3-fb63-47aa-bc23-2367659188a1 button.colab-df-convert');\n",
              "      buttonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "      async function convertToInteractive(key) {\n",
              "        const element = document.querySelector('#df-8843abe3-fb63-47aa-bc23-2367659188a1');\n",
              "        const dataTable =\n",
              "          await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                    [key], {});\n",
              "        if (!dataTable) return;\n",
              "\n",
              "        const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "          '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "          + ' to learn more about interactive tables.';\n",
              "        element.innerHTML = '';\n",
              "        dataTable['output_type'] = 'display_data';\n",
              "        await google.colab.output.renderOutput(dataTable, element);\n",
              "        const docLink = document.createElement('div');\n",
              "        docLink.innerHTML = docLinkHtml;\n",
              "        element.appendChild(docLink);\n",
              "      }\n",
              "    </script>\n",
              "  </div>\n",
              "\n",
              "\n",
              "<div id=\"df-6f5aa01b-5603-4d85-bb1a-19c614d8fe9f\">\n",
              "  <button class=\"colab-df-quickchart\" onclick=\"quickchart('df-6f5aa01b-5603-4d85-bb1a-19c614d8fe9f')\"\n",
              "            title=\"Suggest charts\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "<svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "     width=\"24px\">\n",
              "    <g>\n",
              "        <path d=\"M19 3H5c-1.1 0-2 .9-2 2v14c0 1.1.9 2 2 2h14c1.1 0 2-.9 2-2V5c0-1.1-.9-2-2-2zM9 17H7v-7h2v7zm4 0h-2V7h2v10zm4 0h-2v-4h2v4z\"/>\n",
              "    </g>\n",
              "</svg>\n",
              "  </button>\n",
              "\n",
              "<style>\n",
              "  .colab-df-quickchart {\n",
              "      --bg-color: #E8F0FE;\n",
              "      --fill-color: #1967D2;\n",
              "      --hover-bg-color: #E2EBFA;\n",
              "      --hover-fill-color: #174EA6;\n",
              "      --disabled-fill-color: #AAA;\n",
              "      --disabled-bg-color: #DDD;\n",
              "  }\n",
              "\n",
              "  [theme=dark] .colab-df-quickchart {\n",
              "      --bg-color: #3B4455;\n",
              "      --fill-color: #D2E3FC;\n",
              "      --hover-bg-color: #434B5C;\n",
              "      --hover-fill-color: #FFFFFF;\n",
              "      --disabled-bg-color: #3B4455;\n",
              "      --disabled-fill-color: #666;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart {\n",
              "    background-color: var(--bg-color);\n",
              "    border: none;\n",
              "    border-radius: 50%;\n",
              "    cursor: pointer;\n",
              "    display: none;\n",
              "    fill: var(--fill-color);\n",
              "    height: 32px;\n",
              "    padding: 0;\n",
              "    width: 32px;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart:hover {\n",
              "    background-color: var(--hover-bg-color);\n",
              "    box-shadow: 0 1px 2px rgba(60, 64, 67, 0.3), 0 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "    fill: var(--button-hover-fill-color);\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart-complete:disabled,\n",
              "  .colab-df-quickchart-complete:disabled:hover {\n",
              "    background-color: var(--disabled-bg-color);\n",
              "    fill: var(--disabled-fill-color);\n",
              "    box-shadow: none;\n",
              "  }\n",
              "\n",
              "  .colab-df-spinner {\n",
              "    border: 2px solid var(--fill-color);\n",
              "    border-color: transparent;\n",
              "    border-bottom-color: var(--fill-color);\n",
              "    animation:\n",
              "      spin 1s steps(1) infinite;\n",
              "  }\n",
              "\n",
              "  @keyframes spin {\n",
              "    0% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "      border-left-color: var(--fill-color);\n",
              "    }\n",
              "    20% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    30% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    40% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    60% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    80% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "    90% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "  }\n",
              "</style>\n",
              "\n",
              "  <script>\n",
              "    async function quickchart(key) {\n",
              "      const quickchartButtonEl =\n",
              "        document.querySelector('#' + key + ' button');\n",
              "      quickchartButtonEl.disabled = true;  // To prevent multiple clicks.\n",
              "      quickchartButtonEl.classList.add('colab-df-spinner');\n",
              "      try {\n",
              "        const charts = await google.colab.kernel.invokeFunction(\n",
              "            'suggestCharts', [key], {});\n",
              "      } catch (error) {\n",
              "        console.error('Error during call to suggestCharts:', error);\n",
              "      }\n",
              "      quickchartButtonEl.classList.remove('colab-df-spinner');\n",
              "      quickchartButtonEl.classList.add('colab-df-quickchart-complete');\n",
              "    }\n",
              "    (() => {\n",
              "      let quickchartButtonEl =\n",
              "        document.querySelector('#df-6f5aa01b-5603-4d85-bb1a-19c614d8fe9f button');\n",
              "      quickchartButtonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "    })();\n",
              "  </script>\n",
              "</div>\n",
              "\n",
              "    </div>\n",
              "  </div>\n"
            ],
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "dataframe",
              "variable_name": "df",
              "summary": "{\n  \"name\": \"df\",\n  \"rows\": 2748,\n  \"fields\": [\n    {\n      \"column\": \"sentence\",\n      \"properties\": {\n        \"dtype\": \"string\",\n        \"num_unique_values\": 2710,\n        \"samples\": [\n          \"one wants surf small wave space movies (deep impact armageddon), one fails everywhere.\",\n          \"lame would best way describe it.\",\n          \"attached gas station, rarely good sign.\"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"label\",\n      \"properties\": {\n        \"dtype\": \"number\",\n        \"std\": 0,\n        \"min\": 0,\n        \"max\": 1,\n        \"num_unique_values\": 2,\n        \"samples\": [\n          0,\n          1\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"source\",\n      \"properties\": {\n        \"dtype\": \"category\",\n        \"num_unique_values\": 3,\n        \"samples\": [\n          \"yelp\",\n          \"amazon\"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"word_count_function\",\n      \"properties\": {\n        \"dtype\": \"number\",\n        \"std\": 35,\n        \"min\": 1,\n        \"max\": 1393,\n        \"num_unique_values\": 57,\n        \"samples\": [\n          4,\n          6\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"word_count\",\n      \"properties\": {\n        \"dtype\": \"number\",\n        \"std\": 35,\n        \"min\": 1,\n        \"max\": 1393,\n        \"num_unique_values\": 57,\n        \"samples\": [\n          4,\n          6\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"char_count\",\n      \"properties\": {\n        \"dtype\": \"number\",\n        \"std\": 201,\n        \"min\": 6,\n        \"max\": 7944,\n        \"num_unique_values\": 214,\n        \"samples\": [\n          14,\n          237\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"avg_word\",\n      \"properties\": {\n        \"dtype\": \"number\",\n        \"std\": 1.0909993611067916,\n        \"min\": 2.0,\n        \"max\": 14.0,\n        \"num_unique_values\": 518,\n        \"samples\": [\n          3.8461538461538463,\n          4.294117647058823\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"sentence_stemmed\",\n      \"properties\": {\n        \"dtype\": \"string\",\n        \"num_unique_values\": 2705,\n        \"samples\": [\n          \")setup simpler.\",\n          \"i'd love go back.\"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    }\n  ]\n}"
            }
          },
          "metadata": {},
          "execution_count": 33
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "PuZ8AVKvDKQG"
      },
      "source": [
        "### Lemmatization\n",
        "\n",
        "[Lemmatization](https://nlp.stanford.edu/IR-book/html/htmledition/stemming-and-lemmatization-1.html), unlike Stemming, reduces the inflected words properly ensuring that the root word belongs to the language. In Lemmatization root word is called lemma. A lemma (plural lemmas or lemmata) is the canonical form, dictionary form, or citation form of a set of words.  \n",
        "  \n",
        "We will use  Wordnet for the lemmatization. Thus, we need to download Wordnet to the nltk library.\n",
        "\n",
        "WordNet is a lexical database for the English language. It groups English words into sets of synonyms called synsets, provides short definitions and usage\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "v8qJOjLwDhzB",
        "outputId": "17b27c88-1658-4a5f-8eb0-18e930d0b9ec",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "source": [
        "# Download wordnet\n",
        "nltk.download('wordnet')\n",
        "nltk.download('omw-1.4')"
      ],
      "execution_count": 34,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "[nltk_data] Downloading package wordnet to /root/nltk_data...\n",
            "[nltk_data] Downloading package omw-1.4 to /root/nltk_data...\n"
          ]
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "True"
            ]
          },
          "metadata": {},
          "execution_count": 34
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "qjCuPX1zjrWD"
      },
      "source": [
        "from nltk.stem import WordNetLemmatizer\n",
        "\n",
        "lemmtizer = WordNetLemmatizer()"
      ],
      "execution_count": 35,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "bRdYHVNQCiB9"
      },
      "source": [
        "def lemmatize_function(sent, pos='n'):\n",
        "  word_list = sent.split()\n",
        "  lemma_word_list = [lemmtizer.lemmatize(word, pos=pos) for word in word_list]\n",
        "  lemma_sentence = \" \".join(lemma_word_list)\n",
        "  return lemma_sentence\n",
        "\n",
        "df['sentence_lemmatized'] = df['sentence'].apply(lemmatize_function)"
      ],
      "execution_count": 36,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "lemmatize_function(\"I cannot believe dogs are walking\", pos='v')"
      ],
      "metadata": {
        "id": "CcdZ7DlFDUKU",
        "outputId": "fd6d0e89-b89b-4a88-c929-dbac2228df16",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 36
        }
      },
      "execution_count": 37,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "'I cannot believe dog be walk'"
            ],
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "string"
            }
          },
          "metadata": {},
          "execution_count": 37
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "TKCDAK0PkCeW"
      },
      "source": [
        "Display original pre-processed sentence, stemmed sentence and lemmatized sentence."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "XbfKn2OnkDgl",
        "outputId": "4c681be2-cd68-4517-bb97-89257d9ca668",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 363
        }
      },
      "source": [
        "df[['sentence', 'sentence_stemmed', 'sentence_lemmatized']].head(10)"
      ],
      "execution_count": 38,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "                                            sentence  \\\n",
              "0                                wow... loved place.   \n",
              "1                                    crust not good.   \n",
              "2                           not tasty texture nasty.   \n",
              "3  stopped late may bank holiday rick steve recom...   \n",
              "4                       selection menu great prices.   \n",
              "5                       getting angry want damn pho.   \n",
              "6                             honeslty taste fresh.)   \n",
              "7  potatoes like rubber could tell had made ahead...   \n",
              "8                                   fries great too.   \n",
              "9                                       great touch.   \n",
              "\n",
              "                                    sentence_stemmed  \\\n",
              "0                                 wow... love place.   \n",
              "1                                    crust not good.   \n",
              "2                            not tasti textur nasty.   \n",
              "3  stop late may bank holiday rick steve recommen...   \n",
              "4                          select menu great prices.   \n",
              "5                           get angri want damn pho.   \n",
              "6                              honeslti tast fresh.)   \n",
              "7  potato like rubber could tell had made ahead t...   \n",
              "8                                     fri great too.   \n",
              "9                                       great touch.   \n",
              "\n",
              "                                 sentence_lemmatized  \n",
              "0                                wow... loved place.  \n",
              "1                                    crust not good.  \n",
              "2                           not tasty texture nasty.  \n",
              "3  stopped late may bank holiday rick steve recom...  \n",
              "4                       selection menu great prices.  \n",
              "5                       getting angry want damn pho.  \n",
              "6                             honeslty taste fresh.)  \n",
              "7  potato like rubber could tell had made ahead t...  \n",
              "8                                     fry great too.  \n",
              "9                                       great touch.  "
            ],
            "text/html": [
              "\n",
              "  <div id=\"df-e1e453c7-d83e-4b19-9f53-52c2a55c6a27\" class=\"colab-df-container\">\n",
              "    <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>sentence</th>\n",
              "      <th>sentence_stemmed</th>\n",
              "      <th>sentence_lemmatized</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>wow... loved place.</td>\n",
              "      <td>wow... love place.</td>\n",
              "      <td>wow... loved place.</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>crust not good.</td>\n",
              "      <td>crust not good.</td>\n",
              "      <td>crust not good.</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>not tasty texture nasty.</td>\n",
              "      <td>not tasti textur nasty.</td>\n",
              "      <td>not tasty texture nasty.</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>stopped late may bank holiday rick steve recom...</td>\n",
              "      <td>stop late may bank holiday rick steve recommen...</td>\n",
              "      <td>stopped late may bank holiday rick steve recom...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>selection menu great prices.</td>\n",
              "      <td>select menu great prices.</td>\n",
              "      <td>selection menu great prices.</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>5</th>\n",
              "      <td>getting angry want damn pho.</td>\n",
              "      <td>get angri want damn pho.</td>\n",
              "      <td>getting angry want damn pho.</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>6</th>\n",
              "      <td>honeslty taste fresh.)</td>\n",
              "      <td>honeslti tast fresh.)</td>\n",
              "      <td>honeslty taste fresh.)</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>7</th>\n",
              "      <td>potatoes like rubber could tell had made ahead...</td>\n",
              "      <td>potato like rubber could tell had made ahead t...</td>\n",
              "      <td>potato like rubber could tell had made ahead t...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>8</th>\n",
              "      <td>fries great too.</td>\n",
              "      <td>fri great too.</td>\n",
              "      <td>fry great too.</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9</th>\n",
              "      <td>great touch.</td>\n",
              "      <td>great touch.</td>\n",
              "      <td>great touch.</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>\n",
              "    <div class=\"colab-df-buttons\">\n",
              "\n",
              "  <div class=\"colab-df-container\">\n",
              "    <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-e1e453c7-d83e-4b19-9f53-52c2a55c6a27')\"\n",
              "            title=\"Convert this dataframe to an interactive table.\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\" viewBox=\"0 -960 960 960\">\n",
              "    <path d=\"M120-120v-720h720v720H120Zm60-500h600v-160H180v160Zm220 220h160v-160H400v160Zm0 220h160v-160H400v160ZM180-400h160v-160H180v160Zm440 0h160v-160H620v160ZM180-180h160v-160H180v160Zm440 0h160v-160H620v160Z\"/>\n",
              "  </svg>\n",
              "    </button>\n",
              "\n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    .colab-df-buttons div {\n",
              "      margin-bottom: 4px;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "    <script>\n",
              "      const buttonEl =\n",
              "        document.querySelector('#df-e1e453c7-d83e-4b19-9f53-52c2a55c6a27 button.colab-df-convert');\n",
              "      buttonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "      async function convertToInteractive(key) {\n",
              "        const element = document.querySelector('#df-e1e453c7-d83e-4b19-9f53-52c2a55c6a27');\n",
              "        const dataTable =\n",
              "          await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                    [key], {});\n",
              "        if (!dataTable) return;\n",
              "\n",
              "        const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "          '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "          + ' to learn more about interactive tables.';\n",
              "        element.innerHTML = '';\n",
              "        dataTable['output_type'] = 'display_data';\n",
              "        await google.colab.output.renderOutput(dataTable, element);\n",
              "        const docLink = document.createElement('div');\n",
              "        docLink.innerHTML = docLinkHtml;\n",
              "        element.appendChild(docLink);\n",
              "      }\n",
              "    </script>\n",
              "  </div>\n",
              "\n",
              "\n",
              "<div id=\"df-559dfd28-f038-4184-aba7-3e2917b609f4\">\n",
              "  <button class=\"colab-df-quickchart\" onclick=\"quickchart('df-559dfd28-f038-4184-aba7-3e2917b609f4')\"\n",
              "            title=\"Suggest charts\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "<svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "     width=\"24px\">\n",
              "    <g>\n",
              "        <path d=\"M19 3H5c-1.1 0-2 .9-2 2v14c0 1.1.9 2 2 2h14c1.1 0 2-.9 2-2V5c0-1.1-.9-2-2-2zM9 17H7v-7h2v7zm4 0h-2V7h2v10zm4 0h-2v-4h2v4z\"/>\n",
              "    </g>\n",
              "</svg>\n",
              "  </button>\n",
              "\n",
              "<style>\n",
              "  .colab-df-quickchart {\n",
              "      --bg-color: #E8F0FE;\n",
              "      --fill-color: #1967D2;\n",
              "      --hover-bg-color: #E2EBFA;\n",
              "      --hover-fill-color: #174EA6;\n",
              "      --disabled-fill-color: #AAA;\n",
              "      --disabled-bg-color: #DDD;\n",
              "  }\n",
              "\n",
              "  [theme=dark] .colab-df-quickchart {\n",
              "      --bg-color: #3B4455;\n",
              "      --fill-color: #D2E3FC;\n",
              "      --hover-bg-color: #434B5C;\n",
              "      --hover-fill-color: #FFFFFF;\n",
              "      --disabled-bg-color: #3B4455;\n",
              "      --disabled-fill-color: #666;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart {\n",
              "    background-color: var(--bg-color);\n",
              "    border: none;\n",
              "    border-radius: 50%;\n",
              "    cursor: pointer;\n",
              "    display: none;\n",
              "    fill: var(--fill-color);\n",
              "    height: 32px;\n",
              "    padding: 0;\n",
              "    width: 32px;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart:hover {\n",
              "    background-color: var(--hover-bg-color);\n",
              "    box-shadow: 0 1px 2px rgba(60, 64, 67, 0.3), 0 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "    fill: var(--button-hover-fill-color);\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart-complete:disabled,\n",
              "  .colab-df-quickchart-complete:disabled:hover {\n",
              "    background-color: var(--disabled-bg-color);\n",
              "    fill: var(--disabled-fill-color);\n",
              "    box-shadow: none;\n",
              "  }\n",
              "\n",
              "  .colab-df-spinner {\n",
              "    border: 2px solid var(--fill-color);\n",
              "    border-color: transparent;\n",
              "    border-bottom-color: var(--fill-color);\n",
              "    animation:\n",
              "      spin 1s steps(1) infinite;\n",
              "  }\n",
              "\n",
              "  @keyframes spin {\n",
              "    0% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "      border-left-color: var(--fill-color);\n",
              "    }\n",
              "    20% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    30% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    40% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    60% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    80% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "    90% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "  }\n",
              "</style>\n",
              "\n",
              "  <script>\n",
              "    async function quickchart(key) {\n",
              "      const quickchartButtonEl =\n",
              "        document.querySelector('#' + key + ' button');\n",
              "      quickchartButtonEl.disabled = true;  // To prevent multiple clicks.\n",
              "      quickchartButtonEl.classList.add('colab-df-spinner');\n",
              "      try {\n",
              "        const charts = await google.colab.kernel.invokeFunction(\n",
              "            'suggestCharts', [key], {});\n",
              "      } catch (error) {\n",
              "        console.error('Error during call to suggestCharts:', error);\n",
              "      }\n",
              "      quickchartButtonEl.classList.remove('colab-df-spinner');\n",
              "      quickchartButtonEl.classList.add('colab-df-quickchart-complete');\n",
              "    }\n",
              "    (() => {\n",
              "      let quickchartButtonEl =\n",
              "        document.querySelector('#df-559dfd28-f038-4184-aba7-3e2917b609f4 button');\n",
              "      quickchartButtonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "    })();\n",
              "  </script>\n",
              "</div>\n",
              "\n",
              "    </div>\n",
              "  </div>\n"
            ],
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "dataframe",
              "summary": "{\n  \"name\": \"df[['sentence', 'sentence_stemmed', 'sentence_lemmatized']]\",\n  \"rows\": 10,\n  \"fields\": [\n    {\n      \"column\": \"sentence\",\n      \"properties\": {\n        \"dtype\": \"string\",\n        \"num_unique_values\": 10,\n        \"samples\": [\n          \"fries great too.\",\n          \"crust not good.\",\n          \"getting angry want damn pho.\"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"sentence_stemmed\",\n      \"properties\": {\n        \"dtype\": \"string\",\n        \"num_unique_values\": 10,\n        \"samples\": [\n          \"fri great too.\",\n          \"crust not good.\",\n          \"get angri want damn pho.\"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"sentence_lemmatized\",\n      \"properties\": {\n        \"dtype\": \"string\",\n        \"num_unique_values\": 10,\n        \"samples\": [\n          \"fry great too.\",\n          \"crust not good.\",\n          \"getting angry want damn pho.\"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    }\n  ]\n}"
            }
          },
          "metadata": {},
          "execution_count": 38
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "e6RTwgzzwsJg"
      },
      "source": [
        "Stemmed algorithm seems to be working better in this case when compared to the lemmatization. It is recommended to observe the results after pre-processing tasks to understand the performance of the third party libraries we are using in pre-processing."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ILB7exVHEY1y"
      },
      "source": [
        "## Text Feature Extraction\n",
        "\n",
        "In a numeric dataset (e.g., house price dataset, titanic survival dataset, dungaree dataset), we had numeric and categorical variables, which we transformed to numeric values for predictive analytics. Those numeric variables are called numeric features in the datasets. Similarly, for NLP, we need to derive features from text data in numerical format because machines can only understand numeric representations."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "7FlXXX-cfHZa"
      },
      "source": [
        "### N-Grams\n",
        "\n",
        "An [n-gram](https://en.wikipedia.org/wiki/N-gram) is a contiguous sequence of n items from a given sample of text or speech. They are basically a set of co-occuring words within a given window. When computing the n-grams, the shift is one-step forward (although you can move X words forward in more advanced scenarios). For example, for the sentence \"The cow jumps over the moon\". If N=2 (known as bigrams), then the ngrams would be:\n",
        "* the cow\n",
        "* cow jumps\n",
        "* jumps over\n",
        "* over the\n",
        "* the moon\n",
        "\n",
        "We will use NLTK ngrams and word_tokenizer libraries for n-gram feature extraction.\n",
        "\n",
        "Note: Need to download punkt resource for nltk for work tokenization"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "inAc6aR2lwEg",
        "outputId": "8c2da7aa-35f7-4ee3-980b-94fbbc5a90d2",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "source": [
        "from nltk.tokenize import word_tokenize\n",
        "from nltk.util import ngrams\n",
        "\n",
        "nltk.download('punkt')"
      ],
      "execution_count": 39,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "[nltk_data] Downloading package punkt to /root/nltk_data...\n",
            "[nltk_data]   Unzipping tokenizers/punkt.zip.\n"
          ]
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "True"
            ]
          },
          "metadata": {},
          "execution_count": 39
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "WJYeiIrClzp7"
      },
      "source": [
        "First we define the value for *n*, in n-gram representation."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "wb18qzswl-Bq"
      },
      "source": [
        "n = 3"
      ],
      "execution_count": 40,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "jX6iMftbmDPO"
      },
      "source": [
        "Following n_grams() method will take a sentence and construct a list of n-grams."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "LKPsz4T_l_ZU"
      },
      "source": [
        "def n_grams(text):\n",
        "  if len(word_tokenize(text)) < 3:\n",
        "    return []\n",
        "  n_grams = ngrams(word_tokenize(text), n)\n",
        "  return [' '.join(grams) for grams in n_grams]"
      ],
      "execution_count": 41,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "8zQdw-4pS8ws",
        "outputId": "ec019050-2e50-47e0-adf6-14dc02abbabd",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "source": [
        "txt = \"you want to exclude some stop word being getting ignored\"\n",
        "print(n_grams(txt))"
      ],
      "execution_count": 42,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "['you want to', 'want to exclude', 'to exclude some', 'exclude some stop', 'some stop word', 'stop word being', 'word being getting', 'being getting ignored']\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "n2EEAEptmPrn"
      },
      "source": [
        "Derive n-grams (n=3) for our dataset."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "qj2gxPH7DY84"
      },
      "source": [
        "df['3_grams'] = df['sentence'].apply(lambda x: n_grams(x))"
      ],
      "execution_count": 43,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "9XniThjwmV-r"
      },
      "source": [
        "Display original sentence and n-grams."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ZQUU-YcdmNTD",
        "outputId": "1fd12cc8-c32a-4784-e647-59eeb7a6759f",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 676
        }
      },
      "source": [
        "df[['sentence', '3_grams']].head(20)"
      ],
      "execution_count": 44,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "                                             sentence  \\\n",
              "0                                 wow... loved place.   \n",
              "1                                     crust not good.   \n",
              "2                            not tasty texture nasty.   \n",
              "3   stopped late may bank holiday rick steve recom...   \n",
              "4                        selection menu great prices.   \n",
              "5                        getting angry want damn pho.   \n",
              "6                              honeslty taste fresh.)   \n",
              "7   potatoes like rubber could tell had made ahead...   \n",
              "8                                    fries great too.   \n",
              "9                                        great touch.   \n",
              "10                                    service prompt.   \n",
              "11                                 would not go back.   \n",
              "12  cashier had care ever had say still ended wayy...   \n",
              "13  tried cape cod ravoli, chicken,with cranberry....   \n",
              "14                  disgusted pretty sure human hair.   \n",
              "15                  shocked signs indicate cash only.   \n",
              "16                                highly recommended.   \n",
              "17                      waitress little slow service.   \n",
              "18             place not worth time, let alone vegas.   \n",
              "19                                      not like all.   \n",
              "\n",
              "                                              3_grams  \n",
              "0     [wow ... loved, ... loved place, loved place .]  \n",
              "1                        [crust not good, not good .]  \n",
              "2   [not tasty texture, tasty texture nasty, textu...  \n",
              "3   [stopped late may, late may bank, may bank hol...  \n",
              "4   [selection menu great, menu great prices, grea...  \n",
              "5   [getting angry want, angry want damn, want dam...  \n",
              "6    [honeslty taste fresh, taste fresh ., fresh . )]  \n",
              "7   [potatoes like rubber, like rubber could, rubb...  \n",
              "8                      [fries great too, great too .]  \n",
              "9                                     [great touch .]  \n",
              "10                                 [service prompt .]  \n",
              "11             [would not go, not go back, go back .]  \n",
              "12  [cashier had care, had care ever, care ever ha...  \n",
              "13  [tried cape cod, cape cod ravoli, cod ravoli ,...  \n",
              "14  [disgusted pretty sure, pretty sure human, sur...  \n",
              "15  [shocked signs indicate, signs indicate cash, ...  \n",
              "16                             [highly recommended .]  \n",
              "17  [waitress little slow, little slow service, sl...  \n",
              "18  [place not worth, not worth time, worth time ,...  \n",
              "19                         [not like all, like all .]  "
            ],
            "text/html": [
              "\n",
              "  <div id=\"df-567168e0-b015-4bb1-aba0-5097ef4bb3b9\" class=\"colab-df-container\">\n",
              "    <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>sentence</th>\n",
              "      <th>3_grams</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>wow... loved place.</td>\n",
              "      <td>[wow ... loved, ... loved place, loved place .]</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>crust not good.</td>\n",
              "      <td>[crust not good, not good .]</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>not tasty texture nasty.</td>\n",
              "      <td>[not tasty texture, tasty texture nasty, textu...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>stopped late may bank holiday rick steve recom...</td>\n",
              "      <td>[stopped late may, late may bank, may bank hol...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>selection menu great prices.</td>\n",
              "      <td>[selection menu great, menu great prices, grea...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>5</th>\n",
              "      <td>getting angry want damn pho.</td>\n",
              "      <td>[getting angry want, angry want damn, want dam...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>6</th>\n",
              "      <td>honeslty taste fresh.)</td>\n",
              "      <td>[honeslty taste fresh, taste fresh ., fresh . )]</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>7</th>\n",
              "      <td>potatoes like rubber could tell had made ahead...</td>\n",
              "      <td>[potatoes like rubber, like rubber could, rubb...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>8</th>\n",
              "      <td>fries great too.</td>\n",
              "      <td>[fries great too, great too .]</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9</th>\n",
              "      <td>great touch.</td>\n",
              "      <td>[great touch .]</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>10</th>\n",
              "      <td>service prompt.</td>\n",
              "      <td>[service prompt .]</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>11</th>\n",
              "      <td>would not go back.</td>\n",
              "      <td>[would not go, not go back, go back .]</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>12</th>\n",
              "      <td>cashier had care ever had say still ended wayy...</td>\n",
              "      <td>[cashier had care, had care ever, care ever ha...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>13</th>\n",
              "      <td>tried cape cod ravoli, chicken,with cranberry....</td>\n",
              "      <td>[tried cape cod, cape cod ravoli, cod ravoli ,...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>14</th>\n",
              "      <td>disgusted pretty sure human hair.</td>\n",
              "      <td>[disgusted pretty sure, pretty sure human, sur...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>15</th>\n",
              "      <td>shocked signs indicate cash only.</td>\n",
              "      <td>[shocked signs indicate, signs indicate cash, ...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>16</th>\n",
              "      <td>highly recommended.</td>\n",
              "      <td>[highly recommended .]</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>17</th>\n",
              "      <td>waitress little slow service.</td>\n",
              "      <td>[waitress little slow, little slow service, sl...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>18</th>\n",
              "      <td>place not worth time, let alone vegas.</td>\n",
              "      <td>[place not worth, not worth time, worth time ,...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>19</th>\n",
              "      <td>not like all.</td>\n",
              "      <td>[not like all, like all .]</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>\n",
              "    <div class=\"colab-df-buttons\">\n",
              "\n",
              "  <div class=\"colab-df-container\">\n",
              "    <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-567168e0-b015-4bb1-aba0-5097ef4bb3b9')\"\n",
              "            title=\"Convert this dataframe to an interactive table.\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\" viewBox=\"0 -960 960 960\">\n",
              "    <path d=\"M120-120v-720h720v720H120Zm60-500h600v-160H180v160Zm220 220h160v-160H400v160Zm0 220h160v-160H400v160ZM180-400h160v-160H180v160Zm440 0h160v-160H620v160ZM180-180h160v-160H180v160Zm440 0h160v-160H620v160Z\"/>\n",
              "  </svg>\n",
              "    </button>\n",
              "\n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    .colab-df-buttons div {\n",
              "      margin-bottom: 4px;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "    <script>\n",
              "      const buttonEl =\n",
              "        document.querySelector('#df-567168e0-b015-4bb1-aba0-5097ef4bb3b9 button.colab-df-convert');\n",
              "      buttonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "      async function convertToInteractive(key) {\n",
              "        const element = document.querySelector('#df-567168e0-b015-4bb1-aba0-5097ef4bb3b9');\n",
              "        const dataTable =\n",
              "          await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                    [key], {});\n",
              "        if (!dataTable) return;\n",
              "\n",
              "        const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "          '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "          + ' to learn more about interactive tables.';\n",
              "        element.innerHTML = '';\n",
              "        dataTable['output_type'] = 'display_data';\n",
              "        await google.colab.output.renderOutput(dataTable, element);\n",
              "        const docLink = document.createElement('div');\n",
              "        docLink.innerHTML = docLinkHtml;\n",
              "        element.appendChild(docLink);\n",
              "      }\n",
              "    </script>\n",
              "  </div>\n",
              "\n",
              "\n",
              "<div id=\"df-324b0620-bf1e-42bf-9053-6ff18372a9f2\">\n",
              "  <button class=\"colab-df-quickchart\" onclick=\"quickchart('df-324b0620-bf1e-42bf-9053-6ff18372a9f2')\"\n",
              "            title=\"Suggest charts\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "<svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "     width=\"24px\">\n",
              "    <g>\n",
              "        <path d=\"M19 3H5c-1.1 0-2 .9-2 2v14c0 1.1.9 2 2 2h14c1.1 0 2-.9 2-2V5c0-1.1-.9-2-2-2zM9 17H7v-7h2v7zm4 0h-2V7h2v10zm4 0h-2v-4h2v4z\"/>\n",
              "    </g>\n",
              "</svg>\n",
              "  </button>\n",
              "\n",
              "<style>\n",
              "  .colab-df-quickchart {\n",
              "      --bg-color: #E8F0FE;\n",
              "      --fill-color: #1967D2;\n",
              "      --hover-bg-color: #E2EBFA;\n",
              "      --hover-fill-color: #174EA6;\n",
              "      --disabled-fill-color: #AAA;\n",
              "      --disabled-bg-color: #DDD;\n",
              "  }\n",
              "\n",
              "  [theme=dark] .colab-df-quickchart {\n",
              "      --bg-color: #3B4455;\n",
              "      --fill-color: #D2E3FC;\n",
              "      --hover-bg-color: #434B5C;\n",
              "      --hover-fill-color: #FFFFFF;\n",
              "      --disabled-bg-color: #3B4455;\n",
              "      --disabled-fill-color: #666;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart {\n",
              "    background-color: var(--bg-color);\n",
              "    border: none;\n",
              "    border-radius: 50%;\n",
              "    cursor: pointer;\n",
              "    display: none;\n",
              "    fill: var(--fill-color);\n",
              "    height: 32px;\n",
              "    padding: 0;\n",
              "    width: 32px;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart:hover {\n",
              "    background-color: var(--hover-bg-color);\n",
              "    box-shadow: 0 1px 2px rgba(60, 64, 67, 0.3), 0 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "    fill: var(--button-hover-fill-color);\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart-complete:disabled,\n",
              "  .colab-df-quickchart-complete:disabled:hover {\n",
              "    background-color: var(--disabled-bg-color);\n",
              "    fill: var(--disabled-fill-color);\n",
              "    box-shadow: none;\n",
              "  }\n",
              "\n",
              "  .colab-df-spinner {\n",
              "    border: 2px solid var(--fill-color);\n",
              "    border-color: transparent;\n",
              "    border-bottom-color: var(--fill-color);\n",
              "    animation:\n",
              "      spin 1s steps(1) infinite;\n",
              "  }\n",
              "\n",
              "  @keyframes spin {\n",
              "    0% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "      border-left-color: var(--fill-color);\n",
              "    }\n",
              "    20% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    30% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    40% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    60% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    80% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "    90% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "  }\n",
              "</style>\n",
              "\n",
              "  <script>\n",
              "    async function quickchart(key) {\n",
              "      const quickchartButtonEl =\n",
              "        document.querySelector('#' + key + ' button');\n",
              "      quickchartButtonEl.disabled = true;  // To prevent multiple clicks.\n",
              "      quickchartButtonEl.classList.add('colab-df-spinner');\n",
              "      try {\n",
              "        const charts = await google.colab.kernel.invokeFunction(\n",
              "            'suggestCharts', [key], {});\n",
              "      } catch (error) {\n",
              "        console.error('Error during call to suggestCharts:', error);\n",
              "      }\n",
              "      quickchartButtonEl.classList.remove('colab-df-spinner');\n",
              "      quickchartButtonEl.classList.add('colab-df-quickchart-complete');\n",
              "    }\n",
              "    (() => {\n",
              "      let quickchartButtonEl =\n",
              "        document.querySelector('#df-324b0620-bf1e-42bf-9053-6ff18372a9f2 button');\n",
              "      quickchartButtonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "    })();\n",
              "  </script>\n",
              "</div>\n",
              "\n",
              "    </div>\n",
              "  </div>\n"
            ],
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "dataframe",
              "summary": "{\n  \"name\": \"df[['sentence', '3_grams']]\",\n  \"rows\": 20,\n  \"fields\": [\n    {\n      \"column\": \"sentence\",\n      \"properties\": {\n        \"dtype\": \"string\",\n        \"num_unique_values\": 20,\n        \"samples\": [\n          \"wow... loved place.\",\n          \"waitress little slow service.\",\n          \"shocked signs indicate cash only.\"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"3_grams\",\n      \"properties\": {\n        \"dtype\": \"object\",\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    }\n  ]\n}"
            }
          },
          "metadata": {},
          "execution_count": 44
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "7RJBlyLjh0Zq"
      },
      "source": [
        "Based on above results (e.g., record 16) you can see that if there are only 2 words, the 3-grams would result no n-grams.  \n",
        "Thus, you may try to derive n-grams with *n=2*."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "xhlvf58tqq3C"
      },
      "source": [
        "### Bag of words\n",
        "\n",
        "[Bag of words](https://machinelearningmastery.com/gentle-introduction-bag-words-model/) is a simple text feature extraction mechanism.   \n",
        "A bag-of-words is a representation of text that describes the occurrence of words within a document. It involves two things:\n",
        "* A vocabulary of known words.  \n",
        "* A measure of the presence of known words.  \n",
        "\n",
        "We will use [CountVectorizer library](https://scikit-learn.org/stable/modules/generated/sklearn.feature_extraction.text.CountVectorizer.html) on sklearn for bag-of-words model creation."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "vDsvQka_oJGI"
      },
      "source": [
        "from sklearn.feature_extraction.text import CountVectorizer"
      ],
      "execution_count": 45,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "qBazktNpoPpH"
      },
      "source": [
        "You may refer to [CountVectorizer](https://scikit-learn.org/stable/modules/generated/sklearn.feature_extraction.text.CountVectorizer.html) API for detailed description about the parameters."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "HKuTRNfYmqV0"
      },
      "source": [
        "bow = CountVectorizer(max_features=1000, lowercase=True, ngram_range=(1,1), analyzer = \"word\")"
      ],
      "execution_count": 46,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "frDRiREJokx8"
      },
      "source": [
        "Transform lemmatized senteces into bag-of-words model."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "q1NwVXryrEj2"
      },
      "source": [
        "X_bow = bow.fit_transform(df['sentence_stemmed'])"
      ],
      "execution_count": 47,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "VLyqMA4CoxhA"
      },
      "source": [
        "The X_bow would result in a term-document matrix.  \n",
        "e.g., Output format:  (sentence_id, vocabulary_dictionary_id) count\n",
        "* sentence_id - sentence id in the dataframe\n",
        "* vocabulary_dictionary_id - id of the particular word in the bag of words model dictionary\n",
        "* count - count of words"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ZTedh3MhT6MT",
        "outputId": "104f001f-cb97-4f56-d3a7-934868911765",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "source": [
        "df['sentence_stemmed'].shape"
      ],
      "execution_count": 70,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "(2748,)"
            ]
          },
          "metadata": {},
          "execution_count": 70
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "imw2JQaYrM73",
        "outputId": "cabb35fd-ce60-4465-c5dd-56c51b3d9a1a",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "source": [
        "X_bow.shape"
      ],
      "execution_count": 72,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "(2748, 1000)"
            ]
          },
          "metadata": {},
          "execution_count": 72
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "d4Fxf1C1iPN3"
      },
      "source": [
        "### Term Frequency - Inverse Document Frequecy (TF-IDF)\n",
        "\n",
        "[Term frequency–inverse document frequency](https://www.kdnuggets.com/2018/08/wtf-tf-idf.html), is a numerical statistic that is intended to reflect how important a word is to a document in a collection. The tf–idf value increases proportionally to the number of times a word appears in the document and is offset by the number of documents in the corpus that contain the word, which helps to adjust for the fact that some words appear more frequently in general.\n",
        "\n",
        "We will use [feature extraction module](https://scikit-learn.org/stable/modules/generated/sklearn.feature_extraction.text.TfidfVectorizer.html) of the sklearn library for this."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "YFR61d5ehktG"
      },
      "source": [
        "from sklearn.feature_extraction.text import TfidfVectorizer\n",
        "\n",
        "vectorizer = TfidfVectorizer()"
      ],
      "execution_count": 50,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "UzNi0OmbqSs7"
      },
      "source": [
        "Construct TF-IDF using the lemmatized senteces."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "leDbV3eVjLUK"
      },
      "source": [
        "tf_idf = vectorizer.fit_transform(df['sentence_stemmed'])  # as the text data, we will use lemmatized sentences"
      ],
      "execution_count": 51,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "_kX79IoAqbdb"
      },
      "source": [
        "Display the list of all the words."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "XzBSDPTKktzH",
        "outputId": "6511b1c1-b6f0-4c37-e57c-594e7222a44e",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "source": [
        "print(vectorizer.get_feature_names_out())"
      ],
      "execution_count": 52,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "['00' '10' '100' ... 'zombi' 'zombie' 'zombiez']\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "_8oekEhSlt4r"
      },
      "source": [
        "Here you see there are quite many text that includes a number (digit).  \n",
        "In one of the pre-processing steps, we removed all the words/text that are only digits, but not combined.  \n",
        "You might want to remove these as well...  "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "LSgn4DPGqtEz"
      },
      "source": [
        "A comparison of TF-IDF values with respect to lemmatized sentences."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "_vUTSbDdk0wL",
        "outputId": "3ad5832d-2889-415c-ec17-f7662177ae53",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "source": [
        "print(df['sentence_lemmatized'].head())"
      ],
      "execution_count": 53,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "0                                  wow... loved place.\n",
            "1                                      crust not good.\n",
            "2                             not tasty texture nasty.\n",
            "3    stopped late may bank holiday rick steve recom...\n",
            "4                         selection menu great prices.\n",
            "Name: sentence_lemmatized, dtype: object\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "fb7xxL-LlIul",
        "outputId": "4b8dc50c-074f-4b18-b233-8b95a8558fe5",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "source": [
        "print(tf_idf[:5])"
      ],
      "execution_count": 54,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "  (0, 3098)\t0.4467515072948057\n",
            "  (0, 2493)\t0.4815253044339722\n",
            "  (0, 4685)\t0.7540202065724695\n",
            "  (1, 1842)\t0.3878330443592633\n",
            "  (1, 2830)\t0.35855314218756035\n",
            "  (1, 1018)\t0.8491320120749498\n",
            "  (2, 2764)\t0.5871056522043099\n",
            "  (2, 4132)\t0.5655082660842398\n",
            "  (2, 4095)\t0.5234957828317272\n",
            "  (2, 2830)\t0.24791030534761116\n",
            "  (3, 2242)\t0.19150592485821574\n",
            "  (3, 3356)\t0.2257634873436066\n",
            "  (3, 3924)\t0.3512597313662575\n",
            "  (3, 3463)\t0.3835821062091567\n",
            "  (3, 2026)\t0.3835821062091567\n",
            "  (3, 350)\t0.3835821062091567\n",
            "  (3, 2579)\t0.3000299793060093\n",
            "  (3, 2365)\t0.3408542508108477\n",
            "  (3, 3937)\t0.3134449769315596\n",
            "  (3, 2493)\t0.20765324628761367\n",
            "  (4, 3204)\t0.6029695547245832\n",
            "  (4, 1874)\t0.3020391000394054\n",
            "  (4, 2625)\t0.49730455114224764\n",
            "  (4, 3628)\t0.5457914267701826\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "3kE8A2SBmxdf"
      },
      "source": [
        "In the feature vector row (e.g., (0, 4843)), the first digit refers to the sentence row (i.e., first datarow).  \n",
        "The second digit is the index of alphebitically ordered word list."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "0eNVD0_wGYoP"
      },
      "source": [
        "### Sentiment Analysis\n",
        "\n",
        "Sentiment analysis is basically the process of determining the attitude or the emotion of the writer, i.e., whether it is positive or negative or neutral.\n",
        "\n",
        "We will use the Textblob library. The sentiment function of textblob returns the polarity of the sentence, i.e., a float value which lies in the range of [-1,1] where 1 means positive statement and -1 means a negative statement."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "p7Y1RR38HFs3"
      },
      "source": [
        "from textblob import TextBlob"
      ],
      "execution_count": 55,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "MPElhKYqU2-r",
        "outputId": "2a8b0af8-170f-413f-841b-6e11571f08fb",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "source": [
        "doc1 = \"wow\"\n",
        "TextBlob(doc1).sentiment.polarity"
      ],
      "execution_count": 56,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "0.1"
            ]
          },
          "metadata": {},
          "execution_count": 56
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "QTyI82q-rRZM"
      },
      "source": [
        "Derive sentiment of each sentence."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "MB_i-p0QG8WT"
      },
      "source": [
        "df['sentiment'] = df['sentence_lemmatized'].apply(lambda x: TextBlob(x).sentiment.polarity)"
      ],
      "execution_count": 57,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "QxndvXWrrbdK"
      },
      "source": [
        "Display original sentece with respect to its sentiment."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "EBZ-FeKdG8c2",
        "outputId": "5fc0f112-62bc-43b1-cfa6-6c48281d378f",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "source": [
        "print(df[['sentence_lemmatized', 'sentiment']][:40])"
      ],
      "execution_count": 58,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "                                  sentence_lemmatized  sentiment\n",
            "0                                 wow... loved place.   0.400000\n",
            "1                                     crust not good.  -0.350000\n",
            "2                            not tasty texture nasty.  -1.000000\n",
            "3   stopped late may bank holiday rick steve recom...   0.200000\n",
            "4                        selection menu great prices.   0.800000\n",
            "5                        getting angry want damn pho.  -0.500000\n",
            "6                              honeslty taste fresh.)   0.300000\n",
            "7   potato like rubber could tell had made ahead t...   0.000000\n",
            "8                                      fry great too.   0.800000\n",
            "9                                        great touch.   0.800000\n",
            "10                                    service prompt.   0.000000\n",
            "11                                 would not go back.   0.000000\n",
            "12  cashier had care ever had say still ended wayy...   0.000000\n",
            "13  tried cape cod ravoli, chicken,with cranberry....   0.000000\n",
            "14                  disgusted pretty sure human hair.  -0.062500\n",
            "15                   shocked sign indicate cash only.  -0.350000\n",
            "16                                highly recommended.   0.160000\n",
            "17                      waitress little slow service.  -0.243750\n",
            "18             place not worth time, let alone vegas.  -0.150000\n",
            "19                                      not like all.   0.000000\n",
            "20                                    burrittos blah!   0.000000\n",
            "21                                     food, amazing.   0.600000\n",
            "22                                 service also cute.   0.500000\n",
            "23             could care less... interior beautiful.   0.341667\n",
            "24                                         performed.   0.000000\n",
            "25  that's right....the red velvet cake.....ohhh s...   0.350000\n",
            "26                                             #name?   0.000000\n",
            "27  hole wall ha great mexican street tacos, frien...   0.391667\n",
            "28  took hour get food table restaurant food luke ...   0.300000\n",
            "29                              worst salmon sashimi.  -1.000000\n",
            "30   also combo like burger, fries, beer decent deal.   0.166667\n",
            "31                                   like final blow!   0.000000\n",
            "32            found place accident could not happier.   0.000000\n",
            "33  seems like good quick place grab bite familiar...   0.469444\n",
            "34                           overall, like place lot.   0.000000\n",
            "35          redeeming quality restaurant inexpensive.   0.500000\n",
            "36                         ample portion good prices.   0.700000\n",
            "37  poor service, waiter made feel like stupid eve...  -0.600000\n",
            "38                          first visit hiro delight!   0.312500\n",
            "39                                     service sucks.  -0.300000\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "acrCV8JVVjdx"
      },
      "source": [
        "## Text Classification\n",
        "\n",
        "We will explore few text classification approaches to classify the review data as either positive (1) or negative (0).  \n",
        "Here we will only use the amazon reviews (1000 reviews) for the workshop. (You may use yelp and imdb review data seperately evaluate the approaches.)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "FSsjUBYZrl-y"
      },
      "source": [
        "Previously, we conducted all the pre-processing steps to the entire 3 datasets (amazon, yelp and imdb). This for text classification we will filter only the reviews from amazon."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Sl8comQ3V0zJ"
      },
      "source": [
        "df_amazon = df.loc[df['source'] == 'amazon']"
      ],
      "execution_count": 59,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "qm2-8QfKb90u"
      },
      "source": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "dBIhvHkmWdku",
        "outputId": "3cb508b6-1318-448c-9dcf-3bff10ad32ec",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 417
        }
      },
      "source": [
        "df_amazon.head()"
      ],
      "execution_count": 60,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "                                               sentence  label  source  \\\n",
              "1000                   way plug us unless go converter.      0  amazon   \n",
              "1001                        good case, excellent value.      1  amazon   \n",
              "1002                                     great jawbone.      1  amazon   \n",
              "1003  tied charger conversations lasting minutes.maj...      0  amazon   \n",
              "1004                                         mic great.      1  amazon   \n",
              "\n",
              "      word_count_function  word_count  char_count  avg_word  \\\n",
              "1000                   21          21          82  2.952381   \n",
              "1001                    4           4          27  6.000000   \n",
              "1002                    4           4          22  4.750000   \n",
              "1003                   11          11          79  6.272727   \n",
              "1004                    4           4          17  3.500000   \n",
              "\n",
              "                                       sentence_stemmed  \\\n",
              "1000                   way plug us unless go converter.   \n",
              "1001                            good case, excel value.   \n",
              "1002                                     great jawbone.   \n",
              "1003  tie charger convers last minutes.major problems!!   \n",
              "1004                                         mic great.   \n",
              "\n",
              "                                    sentence_lemmatized  \\\n",
              "1000                    way plug u unless go converter.   \n",
              "1001                        good case, excellent value.   \n",
              "1002                                     great jawbone.   \n",
              "1003  tied charger conversation lasting minutes.majo...   \n",
              "1004                                         mic great.   \n",
              "\n",
              "                                                3_grams  sentiment  \n",
              "1000  [way plug us, plug us unless, us unless go, un...       0.00  \n",
              "1001  [good case ,, case , excellent, , excellent va...       0.85  \n",
              "1002                                  [great jawbone .]       0.80  \n",
              "1003  [tied charger conversations, charger conversat...       0.00  \n",
              "1004                                      [mic great .]       0.80  "
            ],
            "text/html": [
              "\n",
              "  <div id=\"df-788dd71c-4080-4fb7-9645-023c67183ebb\" class=\"colab-df-container\">\n",
              "    <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>sentence</th>\n",
              "      <th>label</th>\n",
              "      <th>source</th>\n",
              "      <th>word_count_function</th>\n",
              "      <th>word_count</th>\n",
              "      <th>char_count</th>\n",
              "      <th>avg_word</th>\n",
              "      <th>sentence_stemmed</th>\n",
              "      <th>sentence_lemmatized</th>\n",
              "      <th>3_grams</th>\n",
              "      <th>sentiment</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>1000</th>\n",
              "      <td>way plug us unless go converter.</td>\n",
              "      <td>0</td>\n",
              "      <td>amazon</td>\n",
              "      <td>21</td>\n",
              "      <td>21</td>\n",
              "      <td>82</td>\n",
              "      <td>2.952381</td>\n",
              "      <td>way plug us unless go converter.</td>\n",
              "      <td>way plug u unless go converter.</td>\n",
              "      <td>[way plug us, plug us unless, us unless go, un...</td>\n",
              "      <td>0.00</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1001</th>\n",
              "      <td>good case, excellent value.</td>\n",
              "      <td>1</td>\n",
              "      <td>amazon</td>\n",
              "      <td>4</td>\n",
              "      <td>4</td>\n",
              "      <td>27</td>\n",
              "      <td>6.000000</td>\n",
              "      <td>good case, excel value.</td>\n",
              "      <td>good case, excellent value.</td>\n",
              "      <td>[good case ,, case , excellent, , excellent va...</td>\n",
              "      <td>0.85</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1002</th>\n",
              "      <td>great jawbone.</td>\n",
              "      <td>1</td>\n",
              "      <td>amazon</td>\n",
              "      <td>4</td>\n",
              "      <td>4</td>\n",
              "      <td>22</td>\n",
              "      <td>4.750000</td>\n",
              "      <td>great jawbone.</td>\n",
              "      <td>great jawbone.</td>\n",
              "      <td>[great jawbone .]</td>\n",
              "      <td>0.80</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1003</th>\n",
              "      <td>tied charger conversations lasting minutes.maj...</td>\n",
              "      <td>0</td>\n",
              "      <td>amazon</td>\n",
              "      <td>11</td>\n",
              "      <td>11</td>\n",
              "      <td>79</td>\n",
              "      <td>6.272727</td>\n",
              "      <td>tie charger convers last minutes.major problems!!</td>\n",
              "      <td>tied charger conversation lasting minutes.majo...</td>\n",
              "      <td>[tied charger conversations, charger conversat...</td>\n",
              "      <td>0.00</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1004</th>\n",
              "      <td>mic great.</td>\n",
              "      <td>1</td>\n",
              "      <td>amazon</td>\n",
              "      <td>4</td>\n",
              "      <td>4</td>\n",
              "      <td>17</td>\n",
              "      <td>3.500000</td>\n",
              "      <td>mic great.</td>\n",
              "      <td>mic great.</td>\n",
              "      <td>[mic great .]</td>\n",
              "      <td>0.80</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>\n",
              "    <div class=\"colab-df-buttons\">\n",
              "\n",
              "  <div class=\"colab-df-container\">\n",
              "    <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-788dd71c-4080-4fb7-9645-023c67183ebb')\"\n",
              "            title=\"Convert this dataframe to an interactive table.\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\" viewBox=\"0 -960 960 960\">\n",
              "    <path d=\"M120-120v-720h720v720H120Zm60-500h600v-160H180v160Zm220 220h160v-160H400v160Zm0 220h160v-160H400v160ZM180-400h160v-160H180v160Zm440 0h160v-160H620v160ZM180-180h160v-160H180v160Zm440 0h160v-160H620v160Z\"/>\n",
              "  </svg>\n",
              "    </button>\n",
              "\n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    .colab-df-buttons div {\n",
              "      margin-bottom: 4px;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "    <script>\n",
              "      const buttonEl =\n",
              "        document.querySelector('#df-788dd71c-4080-4fb7-9645-023c67183ebb button.colab-df-convert');\n",
              "      buttonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "      async function convertToInteractive(key) {\n",
              "        const element = document.querySelector('#df-788dd71c-4080-4fb7-9645-023c67183ebb');\n",
              "        const dataTable =\n",
              "          await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                    [key], {});\n",
              "        if (!dataTable) return;\n",
              "\n",
              "        const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "          '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "          + ' to learn more about interactive tables.';\n",
              "        element.innerHTML = '';\n",
              "        dataTable['output_type'] = 'display_data';\n",
              "        await google.colab.output.renderOutput(dataTable, element);\n",
              "        const docLink = document.createElement('div');\n",
              "        docLink.innerHTML = docLinkHtml;\n",
              "        element.appendChild(docLink);\n",
              "      }\n",
              "    </script>\n",
              "  </div>\n",
              "\n",
              "\n",
              "<div id=\"df-cb5f9faf-96af-4cb8-85c3-4d0de8eb4170\">\n",
              "  <button class=\"colab-df-quickchart\" onclick=\"quickchart('df-cb5f9faf-96af-4cb8-85c3-4d0de8eb4170')\"\n",
              "            title=\"Suggest charts\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "<svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "     width=\"24px\">\n",
              "    <g>\n",
              "        <path d=\"M19 3H5c-1.1 0-2 .9-2 2v14c0 1.1.9 2 2 2h14c1.1 0 2-.9 2-2V5c0-1.1-.9-2-2-2zM9 17H7v-7h2v7zm4 0h-2V7h2v10zm4 0h-2v-4h2v4z\"/>\n",
              "    </g>\n",
              "</svg>\n",
              "  </button>\n",
              "\n",
              "<style>\n",
              "  .colab-df-quickchart {\n",
              "      --bg-color: #E8F0FE;\n",
              "      --fill-color: #1967D2;\n",
              "      --hover-bg-color: #E2EBFA;\n",
              "      --hover-fill-color: #174EA6;\n",
              "      --disabled-fill-color: #AAA;\n",
              "      --disabled-bg-color: #DDD;\n",
              "  }\n",
              "\n",
              "  [theme=dark] .colab-df-quickchart {\n",
              "      --bg-color: #3B4455;\n",
              "      --fill-color: #D2E3FC;\n",
              "      --hover-bg-color: #434B5C;\n",
              "      --hover-fill-color: #FFFFFF;\n",
              "      --disabled-bg-color: #3B4455;\n",
              "      --disabled-fill-color: #666;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart {\n",
              "    background-color: var(--bg-color);\n",
              "    border: none;\n",
              "    border-radius: 50%;\n",
              "    cursor: pointer;\n",
              "    display: none;\n",
              "    fill: var(--fill-color);\n",
              "    height: 32px;\n",
              "    padding: 0;\n",
              "    width: 32px;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart:hover {\n",
              "    background-color: var(--hover-bg-color);\n",
              "    box-shadow: 0 1px 2px rgba(60, 64, 67, 0.3), 0 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "    fill: var(--button-hover-fill-color);\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart-complete:disabled,\n",
              "  .colab-df-quickchart-complete:disabled:hover {\n",
              "    background-color: var(--disabled-bg-color);\n",
              "    fill: var(--disabled-fill-color);\n",
              "    box-shadow: none;\n",
              "  }\n",
              "\n",
              "  .colab-df-spinner {\n",
              "    border: 2px solid var(--fill-color);\n",
              "    border-color: transparent;\n",
              "    border-bottom-color: var(--fill-color);\n",
              "    animation:\n",
              "      spin 1s steps(1) infinite;\n",
              "  }\n",
              "\n",
              "  @keyframes spin {\n",
              "    0% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "      border-left-color: var(--fill-color);\n",
              "    }\n",
              "    20% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    30% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    40% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    60% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    80% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "    90% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "  }\n",
              "</style>\n",
              "\n",
              "  <script>\n",
              "    async function quickchart(key) {\n",
              "      const quickchartButtonEl =\n",
              "        document.querySelector('#' + key + ' button');\n",
              "      quickchartButtonEl.disabled = true;  // To prevent multiple clicks.\n",
              "      quickchartButtonEl.classList.add('colab-df-spinner');\n",
              "      try {\n",
              "        const charts = await google.colab.kernel.invokeFunction(\n",
              "            'suggestCharts', [key], {});\n",
              "      } catch (error) {\n",
              "        console.error('Error during call to suggestCharts:', error);\n",
              "      }\n",
              "      quickchartButtonEl.classList.remove('colab-df-spinner');\n",
              "      quickchartButtonEl.classList.add('colab-df-quickchart-complete');\n",
              "    }\n",
              "    (() => {\n",
              "      let quickchartButtonEl =\n",
              "        document.querySelector('#df-cb5f9faf-96af-4cb8-85c3-4d0de8eb4170 button');\n",
              "      quickchartButtonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "    })();\n",
              "  </script>\n",
              "</div>\n",
              "\n",
              "    </div>\n",
              "  </div>\n"
            ],
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "dataframe",
              "variable_name": "df_amazon",
              "summary": "{\n  \"name\": \"df_amazon\",\n  \"rows\": 1000,\n  \"fields\": [\n    {\n      \"column\": \"sentence\",\n      \"properties\": {\n        \"dtype\": \"string\",\n        \"num_unique_values\": 977,\n        \"samples\": [\n          \"color even prettier thought would be, graphics incredibly sharp.\",\n          \"case flimsy piece plastic has front side protection whatsoever.\",\n          \"superb phone great network.\"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"label\",\n      \"properties\": {\n        \"dtype\": \"number\",\n        \"std\": 0,\n        \"min\": 0,\n        \"max\": 1,\n        \"num_unique_values\": 2,\n        \"samples\": [\n          1,\n          0\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"source\",\n      \"properties\": {\n        \"dtype\": \"category\",\n        \"num_unique_values\": 1,\n        \"samples\": [\n          \"amazon\"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"word_count_function\",\n      \"properties\": {\n        \"dtype\": \"number\",\n        \"std\": 6,\n        \"min\": 1,\n        \"max\": 30,\n        \"num_unique_values\": 30,\n        \"samples\": [\n          1\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"word_count\",\n      \"properties\": {\n        \"dtype\": \"number\",\n        \"std\": 6,\n        \"min\": 1,\n        \"max\": 30,\n        \"num_unique_values\": 30,\n        \"samples\": [\n          1\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"char_count\",\n      \"properties\": {\n        \"dtype\": \"number\",\n        \"std\": 34,\n        \"min\": 11,\n        \"max\": 149,\n        \"num_unique_values\": 137,\n        \"samples\": [\n          67\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"avg_word\",\n      \"properties\": {\n        \"dtype\": \"number\",\n        \"std\": 1.278533398453215,\n        \"min\": 2.3333333333333335,\n        \"max\": 14.0,\n        \"num_unique_values\": 293,\n        \"samples\": [\n          3.142857142857143\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"sentence_stemmed\",\n      \"properties\": {\n        \"dtype\": \"string\",\n        \"num_unique_values\": 975,\n        \"samples\": [\n          \"think good investment.\"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"sentence_lemmatized\",\n      \"properties\": {\n        \"dtype\": \"string\",\n        \"num_unique_values\": 977,\n        \"samples\": [\n          \"color even prettier thought would be, graphic incredibly sharp.\"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"3_grams\",\n      \"properties\": {\n        \"dtype\": \"object\",\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"sentiment\",\n      \"properties\": {\n        \"dtype\": \"number\",\n        \"std\": 0.4512088165956607,\n        \"min\": -1.0,\n        \"max\": 1.0,\n        \"num_unique_values\": 222,\n        \"samples\": [\n          -0.32\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    }\n  ]\n}"
            }
          },
          "metadata": {},
          "execution_count": 60
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "F0JIXz4gr4A0"
      },
      "source": [
        "Split train/validation data\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "0hNVAhPGXmSi"
      },
      "source": [
        "from sklearn.model_selection import train_test_split\n",
        "\n",
        "sentences_train, sentences_test, y_train, y_test = train_test_split(df_amazon['sentence_lemmatized'], df_amazon['label'], test_size=0.3, random_state=2)"
      ],
      "execution_count": 61,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "19jv2X2kZV3B"
      },
      "source": [
        "### Logistic Regression"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "1kPqYktYsOxS"
      },
      "source": [
        "We will use the Bag of Words model as text features."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "EOLyQxTqWfWB"
      },
      "source": [
        "from sklearn.feature_extraction.text import CountVectorizer"
      ],
      "execution_count": 62,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "_aw3BCr0sYLQ"
      },
      "source": [
        "Construct the bag of words model."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "kgvIlfHhsYby",
        "outputId": "b72f15b7-2269-4594-e35f-e40307c1b79b",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 74
        }
      },
      "source": [
        "bow = CountVectorizer(min_df=0.0, lowercase=False)\n",
        "bow.fit(sentences_train)"
      ],
      "execution_count": 63,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "CountVectorizer(lowercase=False, min_df=0.0)"
            ],
            "text/html": [
              "<style>#sk-container-id-1 {color: black;}#sk-container-id-1 pre{padding: 0;}#sk-container-id-1 div.sk-toggleable {background-color: white;}#sk-container-id-1 label.sk-toggleable__label {cursor: pointer;display: block;width: 100%;margin-bottom: 0;padding: 0.3em;box-sizing: border-box;text-align: center;}#sk-container-id-1 label.sk-toggleable__label-arrow:before {content: \"▸\";float: left;margin-right: 0.25em;color: #696969;}#sk-container-id-1 label.sk-toggleable__label-arrow:hover:before {color: black;}#sk-container-id-1 div.sk-estimator:hover label.sk-toggleable__label-arrow:before {color: black;}#sk-container-id-1 div.sk-toggleable__content {max-height: 0;max-width: 0;overflow: hidden;text-align: left;background-color: #f0f8ff;}#sk-container-id-1 div.sk-toggleable__content pre {margin: 0.2em;color: black;border-radius: 0.25em;background-color: #f0f8ff;}#sk-container-id-1 input.sk-toggleable__control:checked~div.sk-toggleable__content {max-height: 200px;max-width: 100%;overflow: auto;}#sk-container-id-1 input.sk-toggleable__control:checked~label.sk-toggleable__label-arrow:before {content: \"▾\";}#sk-container-id-1 div.sk-estimator input.sk-toggleable__control:checked~label.sk-toggleable__label {background-color: #d4ebff;}#sk-container-id-1 div.sk-label input.sk-toggleable__control:checked~label.sk-toggleable__label {background-color: #d4ebff;}#sk-container-id-1 input.sk-hidden--visually {border: 0;clip: rect(1px 1px 1px 1px);clip: rect(1px, 1px, 1px, 1px);height: 1px;margin: -1px;overflow: hidden;padding: 0;position: absolute;width: 1px;}#sk-container-id-1 div.sk-estimator {font-family: monospace;background-color: #f0f8ff;border: 1px dotted black;border-radius: 0.25em;box-sizing: border-box;margin-bottom: 0.5em;}#sk-container-id-1 div.sk-estimator:hover {background-color: #d4ebff;}#sk-container-id-1 div.sk-parallel-item::after {content: \"\";width: 100%;border-bottom: 1px solid gray;flex-grow: 1;}#sk-container-id-1 div.sk-label:hover label.sk-toggleable__label {background-color: #d4ebff;}#sk-container-id-1 div.sk-serial::before {content: \"\";position: absolute;border-left: 1px solid gray;box-sizing: border-box;top: 0;bottom: 0;left: 50%;z-index: 0;}#sk-container-id-1 div.sk-serial {display: flex;flex-direction: column;align-items: center;background-color: white;padding-right: 0.2em;padding-left: 0.2em;position: relative;}#sk-container-id-1 div.sk-item {position: relative;z-index: 1;}#sk-container-id-1 div.sk-parallel {display: flex;align-items: stretch;justify-content: center;background-color: white;position: relative;}#sk-container-id-1 div.sk-item::before, #sk-container-id-1 div.sk-parallel-item::before {content: \"\";position: absolute;border-left: 1px solid gray;box-sizing: border-box;top: 0;bottom: 0;left: 50%;z-index: -1;}#sk-container-id-1 div.sk-parallel-item {display: flex;flex-direction: column;z-index: 1;position: relative;background-color: white;}#sk-container-id-1 div.sk-parallel-item:first-child::after {align-self: flex-end;width: 50%;}#sk-container-id-1 div.sk-parallel-item:last-child::after {align-self: flex-start;width: 50%;}#sk-container-id-1 div.sk-parallel-item:only-child::after {width: 0;}#sk-container-id-1 div.sk-dashed-wrapped {border: 1px dashed gray;margin: 0 0.4em 0.5em 0.4em;box-sizing: border-box;padding-bottom: 0.4em;background-color: white;}#sk-container-id-1 div.sk-label label {font-family: monospace;font-weight: bold;display: inline-block;line-height: 1.2em;}#sk-container-id-1 div.sk-label-container {text-align: center;}#sk-container-id-1 div.sk-container {/* jupyter's `normalize.less` sets `[hidden] { display: none; }` but bootstrap.min.css set `[hidden] { display: none !important; }` so we also need the `!important` here to be able to override the default hidden behavior on the sphinx rendered scikit-learn.org. See: https://github.com/scikit-learn/scikit-learn/issues/21755 */display: inline-block !important;position: relative;}#sk-container-id-1 div.sk-text-repr-fallback {display: none;}</style><div id=\"sk-container-id-1\" class=\"sk-top-container\"><div class=\"sk-text-repr-fallback\"><pre>CountVectorizer(lowercase=False, min_df=0.0)</pre><b>In a Jupyter environment, please rerun this cell to show the HTML representation or trust the notebook. <br />On GitHub, the HTML representation is unable to render, please try loading this page with nbviewer.org.</b></div><div class=\"sk-container\" hidden><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-1\" type=\"checkbox\" checked><label for=\"sk-estimator-id-1\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">CountVectorizer</label><div class=\"sk-toggleable__content\"><pre>CountVectorizer(lowercase=False, min_df=0.0)</pre></div></div></div></div></div>"
            ]
          },
          "metadata": {},
          "execution_count": 63
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "3mj1IC8CsgTW"
      },
      "source": [
        "Fit the train and test sentences to transform them to bag-of-word features."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "xNZIJTzdXCbE"
      },
      "source": [
        "X_train = bow.transform(sentences_train)\n",
        "X_test  = bow.transform(sentences_test)"
      ],
      "execution_count": 64,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "KJ3_5XCOspgD"
      },
      "source": [
        "Use a logistic regression model for classification."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "1Xqy_u4hYT2i",
        "outputId": "aeb21f53-f812-429a-cc0b-6410c6c725da",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 74
        }
      },
      "source": [
        "from sklearn.linear_model import LogisticRegression\n",
        "\n",
        "classifier = LogisticRegression()\n",
        "classifier.fit(X_train, y_train)"
      ],
      "execution_count": 65,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "LogisticRegression()"
            ],
            "text/html": [
              "<style>#sk-container-id-2 {color: black;}#sk-container-id-2 pre{padding: 0;}#sk-container-id-2 div.sk-toggleable {background-color: white;}#sk-container-id-2 label.sk-toggleable__label {cursor: pointer;display: block;width: 100%;margin-bottom: 0;padding: 0.3em;box-sizing: border-box;text-align: center;}#sk-container-id-2 label.sk-toggleable__label-arrow:before {content: \"▸\";float: left;margin-right: 0.25em;color: #696969;}#sk-container-id-2 label.sk-toggleable__label-arrow:hover:before {color: black;}#sk-container-id-2 div.sk-estimator:hover label.sk-toggleable__label-arrow:before {color: black;}#sk-container-id-2 div.sk-toggleable__content {max-height: 0;max-width: 0;overflow: hidden;text-align: left;background-color: #f0f8ff;}#sk-container-id-2 div.sk-toggleable__content pre {margin: 0.2em;color: black;border-radius: 0.25em;background-color: #f0f8ff;}#sk-container-id-2 input.sk-toggleable__control:checked~div.sk-toggleable__content {max-height: 200px;max-width: 100%;overflow: auto;}#sk-container-id-2 input.sk-toggleable__control:checked~label.sk-toggleable__label-arrow:before {content: \"▾\";}#sk-container-id-2 div.sk-estimator input.sk-toggleable__control:checked~label.sk-toggleable__label {background-color: #d4ebff;}#sk-container-id-2 div.sk-label input.sk-toggleable__control:checked~label.sk-toggleable__label {background-color: #d4ebff;}#sk-container-id-2 input.sk-hidden--visually {border: 0;clip: rect(1px 1px 1px 1px);clip: rect(1px, 1px, 1px, 1px);height: 1px;margin: -1px;overflow: hidden;padding: 0;position: absolute;width: 1px;}#sk-container-id-2 div.sk-estimator {font-family: monospace;background-color: #f0f8ff;border: 1px dotted black;border-radius: 0.25em;box-sizing: border-box;margin-bottom: 0.5em;}#sk-container-id-2 div.sk-estimator:hover {background-color: #d4ebff;}#sk-container-id-2 div.sk-parallel-item::after {content: \"\";width: 100%;border-bottom: 1px solid gray;flex-grow: 1;}#sk-container-id-2 div.sk-label:hover label.sk-toggleable__label {background-color: #d4ebff;}#sk-container-id-2 div.sk-serial::before {content: \"\";position: absolute;border-left: 1px solid gray;box-sizing: border-box;top: 0;bottom: 0;left: 50%;z-index: 0;}#sk-container-id-2 div.sk-serial {display: flex;flex-direction: column;align-items: center;background-color: white;padding-right: 0.2em;padding-left: 0.2em;position: relative;}#sk-container-id-2 div.sk-item {position: relative;z-index: 1;}#sk-container-id-2 div.sk-parallel {display: flex;align-items: stretch;justify-content: center;background-color: white;position: relative;}#sk-container-id-2 div.sk-item::before, #sk-container-id-2 div.sk-parallel-item::before {content: \"\";position: absolute;border-left: 1px solid gray;box-sizing: border-box;top: 0;bottom: 0;left: 50%;z-index: -1;}#sk-container-id-2 div.sk-parallel-item {display: flex;flex-direction: column;z-index: 1;position: relative;background-color: white;}#sk-container-id-2 div.sk-parallel-item:first-child::after {align-self: flex-end;width: 50%;}#sk-container-id-2 div.sk-parallel-item:last-child::after {align-self: flex-start;width: 50%;}#sk-container-id-2 div.sk-parallel-item:only-child::after {width: 0;}#sk-container-id-2 div.sk-dashed-wrapped {border: 1px dashed gray;margin: 0 0.4em 0.5em 0.4em;box-sizing: border-box;padding-bottom: 0.4em;background-color: white;}#sk-container-id-2 div.sk-label label {font-family: monospace;font-weight: bold;display: inline-block;line-height: 1.2em;}#sk-container-id-2 div.sk-label-container {text-align: center;}#sk-container-id-2 div.sk-container {/* jupyter's `normalize.less` sets `[hidden] { display: none; }` but bootstrap.min.css set `[hidden] { display: none !important; }` so we also need the `!important` here to be able to override the default hidden behavior on the sphinx rendered scikit-learn.org. See: https://github.com/scikit-learn/scikit-learn/issues/21755 */display: inline-block !important;position: relative;}#sk-container-id-2 div.sk-text-repr-fallback {display: none;}</style><div id=\"sk-container-id-2\" class=\"sk-top-container\"><div class=\"sk-text-repr-fallback\"><pre>LogisticRegression()</pre><b>In a Jupyter environment, please rerun this cell to show the HTML representation or trust the notebook. <br />On GitHub, the HTML representation is unable to render, please try loading this page with nbviewer.org.</b></div><div class=\"sk-container\" hidden><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-2\" type=\"checkbox\" checked><label for=\"sk-estimator-id-2\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">LogisticRegression</label><div class=\"sk-toggleable__content\"><pre>LogisticRegression()</pre></div></div></div></div></div>"
            ]
          },
          "metadata": {},
          "execution_count": 65
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "kvhA_f2FtDeU"
      },
      "source": [
        "Evaluate the model."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "PhqBSmAfYg6I",
        "outputId": "d48b883c-a69b-43f2-d8c1-d85178cab631",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "source": [
        "score_train = classifier.score(X_train, y_train)\n",
        "score_test = classifier.score(X_test, y_test)\n",
        "print('Train accuracy: {:.2f}%'.format(score_train*100))\n",
        "print('Test accuracy: {:.2f}%'.format(score_test*100))"
      ],
      "execution_count": 66,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Train accuracy: 98.86%\n",
            "Test accuracy: 80.33%\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "geKjAtjGtJ6g"
      },
      "source": [
        "What can you say about bias and variance of this model?  \n",
        "- Low bias and high variance"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Gq_TLlEUKQeC"
      },
      "source": [
        "Try few examples"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "riamqEZjGhrj",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "9374626b-1a64-4a6b-b394-d904d56dc1d5"
      },
      "source": [
        "testing = \"happy customer\"\n",
        "vector_representation = bow.transform([testing])\n",
        "prediction = classifier.predict(vector_representation)[0]\n",
        "\n",
        "if prediction == 1:\n",
        "   print(\"Positive Review\")\n",
        "else:\n",
        "   print(\"Negative Review\")"
      ],
      "execution_count": 67,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Positive Review\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "shvn4nHzhIRr"
      },
      "source": [
        "## Exercise\n",
        "\n",
        "You may conduct similar classification exercises for yelp and imdb datasets."
      ]
    }
  ]
}