Week_06_(1)_Scraping_Twitter_data.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "Week 06 (1) Scraping Twitter data.ipynb",
      "provenance": [],
      "collapsed_sections": [],
      "toc_visible": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "uFNwrV7QLrlh"
      },
      "source": [
        "# **Scraping data from Twitter using Python package**\n",
        "\n",
        "This tutorial will follow through the process of scraping tweets from Twitter using Twint python library.  \n",
        "\n",
        "Twint is an advanced Twitter scraping tool written in Python that allows for scraping Tweets from Twitter profiles without using Twitter's API. Twint utilizes Twitter's search operators to let you scrape Tweets from specific users, scrape Tweets relating to certain topics, hashtags & trends.\n",
        "\n",
        "Twint documentation can be found from [this URL](https://github.com/twintproject/twint)."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "StpJjsogVeBf"
      },
      "source": [
        "## Install and Load Libraries"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "RjqmeVAeRpOf"
      },
      "source": [
        "**twint** package is not pre-installed in the Colab environment. Therefore run the following command to install it.  \n",
        "If you are working on your own Python environment, you can install twint using Anaconda interactive user interface or running *pip install twint* from your command prompt."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "tilDnFIuLmuW",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "f11ece47-73d7-4041-ee46-5d2149e9e7ab"
      },
      "source": [
        "!pip install --user --upgrade -e git+https://github.com/twintproject/twint.git@origin/master#egg=twint\n",
        "# You need to go to Runtime option and select restart runtime to get the effect of this installation.\n",
        "!pip install nest-asyncio"
      ],
      "execution_count": 14,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Looking in indexes: https://pypi.org/simple, https://us-python.pkg.dev/colab-wheels/public/simple/\n",
            "Obtaining twint from git+https://github.com/twintproject/twint.git@origin/master#egg=twint\n",
            "  Updating ./src/twint clone (to revision origin/master)\n",
            "  Running command git fetch -q --tags\n",
            "\u001b[33m  WARNING: Did not find branch or tag 'origin/master', assuming revision or ref.\u001b[0m\n",
            "  Running command git reset --hard -q origin/master\n",
            "Requirement already satisfied: aiohttp in /usr/local/lib/python3.7/dist-packages (from twint) (3.8.1)\n",
            "Requirement already satisfied: aiodns in /root/.local/lib/python3.7/site-packages (from twint) (3.0.0)\n",
            "Requirement already satisfied: beautifulsoup4 in /usr/local/lib/python3.7/dist-packages (from twint) (4.6.3)\n",
            "Requirement already satisfied: cchardet in /root/.local/lib/python3.7/site-packages (from twint) (2.1.7)\n",
            "Requirement already satisfied: dataclasses in /root/.local/lib/python3.7/site-packages (from twint) (0.6)\n",
            "Requirement already satisfied: elasticsearch in /root/.local/lib/python3.7/site-packages (from twint) (8.3.3)\n",
            "Requirement already satisfied: pysocks in /usr/local/lib/python3.7/dist-packages (from twint) (1.7.1)\n",
            "Requirement already satisfied: pandas in /usr/local/lib/python3.7/dist-packages (from twint) (1.3.5)\n",
            "Requirement already satisfied: aiohttp_socks in /root/.local/lib/python3.7/site-packages (from twint) (0.7.1)\n",
            "Requirement already satisfied: schedule in /root/.local/lib/python3.7/site-packages (from twint) (1.1.0)\n",
            "Requirement already satisfied: geopy in /usr/local/lib/python3.7/dist-packages (from twint) (1.17.0)\n",
            "Requirement already satisfied: fake-useragent in /root/.local/lib/python3.7/site-packages (from twint) (0.1.11)\n",
            "Requirement already satisfied: googletransx in /root/.local/lib/python3.7/site-packages (from twint) (2.4.2)\n",
            "Requirement already satisfied: pycares>=4.0.0 in /root/.local/lib/python3.7/site-packages (from aiodns->twint) (4.2.2)\n",
            "Requirement already satisfied: cffi>=1.5.0 in /usr/local/lib/python3.7/dist-packages (from pycares>=4.0.0->aiodns->twint) (1.15.1)\n",
            "Requirement already satisfied: pycparser in /usr/local/lib/python3.7/dist-packages (from cffi>=1.5.0->pycares>=4.0.0->aiodns->twint) (2.21)\n",
            "Requirement already satisfied: attrs>=17.3.0 in /usr/local/lib/python3.7/dist-packages (from aiohttp->twint) (22.1.0)\n",
            "Requirement already satisfied: frozenlist>=1.1.1 in /usr/local/lib/python3.7/dist-packages (from aiohttp->twint) (1.3.1)\n",
            "Requirement already satisfied: typing-extensions>=3.7.4 in /usr/local/lib/python3.7/dist-packages (from aiohttp->twint) (4.1.1)\n",
            "Requirement already satisfied: aiosignal>=1.1.2 in /usr/local/lib/python3.7/dist-packages (from aiohttp->twint) (1.2.0)\n",
            "Requirement already satisfied: async-timeout<5.0,>=4.0.0a3 in /usr/local/lib/python3.7/dist-packages (from aiohttp->twint) (4.0.2)\n",
            "Requirement already satisfied: multidict<7.0,>=4.5 in /usr/local/lib/python3.7/dist-packages (from aiohttp->twint) (6.0.2)\n",
            "Requirement already satisfied: yarl<2.0,>=1.0 in /usr/local/lib/python3.7/dist-packages (from aiohttp->twint) (1.8.1)\n",
            "Requirement already satisfied: asynctest==0.13.0 in /usr/local/lib/python3.7/dist-packages (from aiohttp->twint) (0.13.0)\n",
            "Requirement already satisfied: charset-normalizer<3.0,>=2.0 in /usr/local/lib/python3.7/dist-packages (from aiohttp->twint) (2.1.0)\n",
            "Requirement already satisfied: idna>=2.0 in /usr/local/lib/python3.7/dist-packages (from yarl<2.0,>=1.0->aiohttp->twint) (2.10)\n",
            "Requirement already satisfied: python-socks[asyncio]<3.0.0,>=2.0.0 in /root/.local/lib/python3.7/site-packages (from aiohttp_socks->twint) (2.0.3)\n",
            "Requirement already satisfied: elastic-transport<9,>=8 in /root/.local/lib/python3.7/site-packages (from elasticsearch->twint) (8.1.2)\n",
            "Requirement already satisfied: certifi in /usr/local/lib/python3.7/dist-packages (from elastic-transport<9,>=8->elasticsearch->twint) (2022.6.15)\n",
            "Requirement already satisfied: urllib3<2,>=1.26.2 in /root/.local/lib/python3.7/site-packages (from elastic-transport<9,>=8->elasticsearch->twint) (1.26.11)\n",
            "Requirement already satisfied: geographiclib<2,>=1.49 in /usr/local/lib/python3.7/dist-packages (from geopy->twint) (1.52)\n",
            "Requirement already satisfied: requests in /root/.local/lib/python3.7/site-packages (from googletransx->twint) (2.28.1)\n",
            "Requirement already satisfied: python-dateutil>=2.7.3 in /usr/local/lib/python3.7/dist-packages (from pandas->twint) (2.8.2)\n",
            "Requirement already satisfied: pytz>=2017.3 in /usr/local/lib/python3.7/dist-packages (from pandas->twint) (2022.2.1)\n",
            "Requirement already satisfied: numpy>=1.17.3 in /usr/local/lib/python3.7/dist-packages (from pandas->twint) (1.21.6)\n",
            "Requirement already satisfied: six>=1.5 in /usr/local/lib/python3.7/dist-packages (from python-dateutil>=2.7.3->pandas->twint) (1.15.0)\n",
            "Installing collected packages: twint\n",
            "  Attempting uninstall: twint\n",
            "    Found existing installation: twint 2.1.21\n",
            "    Uninstalling twint-2.1.21:\n",
            "      Successfully uninstalled twint-2.1.21\n",
            "  Running setup.py develop for twint\n",
            "Successfully installed twint\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "vyjO-aGjRvUn"
      },
      "source": [
        "import twint\n",
        "import nest_asyncio\n",
        "nest_asyncio.apply()"
      ],
      "execution_count": 1,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "11eYw4aoWV6d"
      },
      "source": [
        "## Design the scaping parameters"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "oJJf66ZgMbCm"
      },
      "source": [
        "Next you need to design which parameters you need to scrape. The parameters to be customised can be found from Twint API located in [this URL](https://github.com/twintproject/twint/wiki/Configuration).  \n",
        "\n",
        "For this exercise, we will scrape tweets with specific keywords and within a pre-given time period."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "1UnCO28zXdce"
      },
      "source": [
        "keyword = 'LaTrobeBusiness'"
      ],
      "execution_count": 2,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "G1B4KTxlXdfN"
      },
      "source": [
        "since_date = \"2020-03-01 00:00:00\"  # Should be '%Y-%m-%d %H:%M:%S' time format"
      ],
      "execution_count": 3,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Pb8bC4g7XnI8"
      },
      "source": [
        "Set the parameters"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "4tX8qTPPXdiI"
      },
      "source": [
        "c = twint.Config()\n",
        "c.Search = keyword\n",
        "c.Since = since_date"
      ],
      "execution_count": 4,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "znpZxw73cFPr"
      },
      "source": [
        "Output parameters"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "6eg2OLIxcD0T"
      },
      "source": [
        "c.Lang = \"en\"\n",
        "c.Store_csv = True\n",
        "c.Output = keyword"
      ],
      "execution_count": 5,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "PJcve_ANbZvw"
      },
      "source": [
        "Format the output attributes.  \n",
        "Available parameters can be found from [this URL](https://github.com/twintproject/twint/wiki/Tweet-attributes)."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "G2AAxKx7XzAu"
      },
      "source": [
        "Here we will use exeption handling in order to warn us if there is an error. This is called python exception handling using try-catch block."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ryE6s7uGXq-o"
      },
      "source": [
        "try:\n",
        "  # Start search\n",
        "  twint.run.Search(c)\n",
        "except Exception as e:\n",
        "  print(e)\n",
        "  print('Error in {}'.format(keyword))"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "9pgnAEiNVPkT"
      },
      "source": [
        "The data will be saved in different folders based on the keywords in the **Files** tab.  \n",
        "To download, right click on tweets.csv file, and download."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "R29fdD45eXgA"
      },
      "source": [
        "### Keyword combinations\n",
        "\n",
        "You can search and scrape tweets using multiple keywords and combinations as documented in [this URL](https://github.com/twintproject/twint/issues/165#issuecomment-399747575)."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "J-t7kMz8e1OQ"
      },
      "source": [
        "keyword = 'latrobe covid19'\n",
        "since_date = \"2020-03-01 00:00:00\"  # Should be '%Y-%m-%d %H:%M:%S' time format\n",
        "c = twint.Config()\n",
        "c.Search = keyword\n",
        "c.Limit = 40  # (Increments of 20) e.g., If 23, it will scrape 40. If 12, it will scrape 20.\n",
        "c.Since = since_date\n",
        "c.Store_csv = True\n",
        "c.Output = keyword"
      ],
      "execution_count": 7,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "qI6_JL-CfEzF"
      },
      "source": [
        "try:\n",
        "  # Start search\n",
        "  twint.run.Search(c)\n",
        "except Exception as e:\n",
        "  print(e)\n",
        "  print('Error in {}'.format(keyword))"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Noa8UV3hghu0"
      },
      "source": [
        "### Multiple keywords\n",
        "\n",
        "We can loop to scape multiple keywords."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Gulilp79fE5d"
      },
      "source": [
        "Keyword_list = ['LBS', 'data analytics latrobe']"
      ],
      "execution_count": 9,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "WUp9HUflXAFh"
      },
      "source": [
        "since_date = \"2020-03-01 00:00:00\"  # Should be '%Y-%m-%d %H:%M:%S' time format"
      ],
      "execution_count": 10,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "sZjBvd-gNEja"
      },
      "source": [
        "for key in Keyword_list:\n",
        "\n",
        "  print('Searching Tweets for {}'.format(key))\n",
        "\n",
        "  c = twint.Config()\n",
        "  c.Search = key\n",
        "  c.Limit = 40\n",
        "  c.Since = since_date\n",
        "  c.Store_csv = True\n",
        "  c.Output = key\n",
        "\n",
        "  try:\n",
        "    # Start search\n",
        "    twint.run.Search(c)\n",
        "  except Exception as e:\n",
        "    print(e)\n",
        "    print('Error in {}'.format(key))\n",
        "  continue\n",
        "\n",
        "  print('\\n\\nCompleted {}\\n\\n'.format(key))"
      ],
      "execution_count": null,
      "outputs": []
    }
  ]
}